Skip to content

Storage → Migrating a live bucket to a newer format #196

Description

@revett

Problem

  • From 0.1.0 onwards a bucket is migrated forward, never abandoned (Storage → Reserve an encryption envelope in the object format #184), and nothing in the project says how that actually happens; the first change that needs it is encryption at 0.3.0
  • A format change is not one object. The manifest is a single conditional PUT, but re-encrypting or re-addressing every blob is thousands of writes with no compare and swap covering the set, so "this bucket is version N" is only true at the instant the manifest flips
  • Two devices on different plugin versions meet at the same bucket. The moment one migrates, every device still on the old build refuses to sync until its user updates, which on mobile can be days; a forced fleet wide update is the price of every format change unless something is done about it
  • Nobody has decided who migrates: the first device to notice, silently, or an explicit user action. Silent matches the "sync you never think about" bar right up until a phone rewrites an entire vault over cellular
  • A migration that rewrites blobs leaves the originals unreferenced, and geode never deletes a blob, so every format change permanently grows the bucket unless retention is part of the answer
  • An interrupted migration has to be resumable, and a half migrated bucket must never be indistinguishable from a corrupt one, which is exactly the state a closed laptop lid produces
  • Whatever this needs from the on disk format has to be known before 0.1.0, for the same reason Storage → Reserve an encryption envelope in the object format #184 existed: after that, adding a field to support migration means the migration needs a migration

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions