Versioning, Deployment Isolation & Upgrades

[VaguelySerious]

To make good on its promise for durability and reliability, any production workflow should be able to:

• Keep running on the code it was originally started at, even if new code is deployed
• Allow upgrading to a new version of the code safely, without losing state or data

Today, versioning reliability already works for Vercel deployments, but not for other worlds. This RFC and its implementation will allow any World to support it. Upgrading workflows is currently not supported for any world, until this RFC is implemented.

Interface changes

A Version loosely represents a (current or future) deployment of a workflow runtime bundle. Worlds may have very different implementations of versions internally, but
from a consumer perspective, the changes to the interface are minimal.

World Interface

Versions are mainly managed through Storage interactions. The storage interface will be extended to support CRUD operations for a new Version entity.

interface World {
  versions: {
    get(versionId: string): Promise<Version>;
    list(params?: VersionListParams): Promise<Version[]>;
    create({ versionId: string, manifest: Manifest, address?: string }): Promise<Version>;
    update(versionId: string, updates: VersionUpdate): Promise<Version>;
    // Returns the version for the environment that the code is running in.
    // If we're not running in a workflow environment, throws a MissingWorkflowContextError
    getCurrent(): Promise<Version>;
    // There is no delete calluse update(versionId, {address: null}) instead
  };
}

type Version = {
  // The structure of the version ID is up to the World or the environment.
  // IDs are generated through `versions.create` and returned through
  // `versions.getCurrent` or `versions.list`.
  // For implementing a World, we recommend a prefixed ULID, like "vers_01ARZ3NDEKTSV4RRFFQ69G5FAV"
  versionId: string;
  // Target address that the bundle is deployed to (optional). Could be a URL,
  // or an ID, e.g. an ARN, or any other identifying string.
  // Presence of the string will usually indicate that the workflow .well-known
  // API for this version of code is live, and accessible using this address.
  // The structure of the address and whether the address indicates liveness
  // depends on the World implementation.
  address?: string;
  // A copy of the JSON build manifest that was output by the build step.
  manifest: Manifest;
  createdAt: Date;
  // Worlds may choose to add additional data to the Version entity,
  // for internal use. This has been separated out to avoid conflicts with future
  // extensions to the Version type.
  metadata?: Record<string, unknown>;
};

type Manifest = {
  manifestVersion: 1;          // Schema version for forward compatibility
  steps?: { /* ... */ };       // Same as current build output
  workflows?: { /* ... */ };   // Same as current build output
  // A future RFC will define additions to the Manifest that will be useful
  // for programmatic use and visualization of workflows.
};

// Also, we'll rename `run.deploymentId` to `run.versionId` for consistency
type WorkflowRun {
  // ...
  versionId: string;
}

// and allow filtering by versionId when listing runs
type RunListParams {
  // ...
  versionId: string;
}

Queue Changes

Detecting the current version has moved to the storage layer:

Current: world.queue.getDeploymentId() → string

New: world.versions.getCurrent() → Version

Enqueuing messages already takes a deploymentId currently, which will be renamed to versionId, but will otherwise be the same.

Overall, the queue interface remains unchanged, but with the addition of versioning,
a World can now choose to perform additional checks for queueing. A World may choose to:

• Use the storage layer to check whether a version the request is routed to is live
• Spin up an old version of the bundle to handle an incoming request
• Selectively route to different versions of workflows (e.g. for A/B testing) after checking version compatibility (note that checking compatibility will be available as a Runtime helper function, see "Manifests & Compatibility" section)

Runtime Changes

The main new addition to the runtime is the ability to upgrade a running workflow to a new version of the code.

Upgrade API

type UpgradeStrategy = "cancel-and-restart" | "cancel-and-resume";

// From `workflow/api`
class Run {
	//...
	
	// Created a new Run and returns the Run instance for it
	// Throws UpgradeNotCompatibleError if the upgrade strategy determined the new version is incompatible (old run will continue running)
	// Throws UpgradeError if the upgrade fails for other reasons (old run will continue running)
	// Throws CriticalUpgradeError if the upgrade fails and could not guarantee the existing run was unaffected, it might have been paused and needs manual attention. This should not happen unless the world has implementation issues.
	async upgrade(
	  targetVersionId: string,
	  options?: { strategy: UpgradeStrategy }
	) => Promise<Run>
}

// Use like
import { getRun } from 'workflow/api';

const run = getRun(runId);
run.upgrade(newVersionId, strategy)

cancel-and-restart:

1. Fetch new and old Version
2. Validate compatibility between input argument types taken from Manifest
3. If incompatible, fail the upgrade.
4. If compatible, pause old run
5. Start new run with same inputs
6. Cancel old run (non-reversible)
7. A failure in (5) or (6) will catch and resume the old run

cancel-and-resume:

1. Fetch new and old Version
2. Validate compatibility between Manifests
3. If incompatible, fail the upgrade. Might later return how much of the event log was compatible in number of events.
4. Instantiate a dry-run of the new run version with the serialized state of the old run to check if the run is resumable.
5. ... remaining steps same as "cancel-and-restart" ...

cancel-and-partial-resume:

1. Same as cancel-and-resume but will not fail the upgrade if the event log is only partially resumable, instead will upgrade and resume from the last compatible state. For completely incompatible events logs, will behave the same as cancel-and-restart.

Helpers

Functions used to provide the upgrade functionality will be exposed as helpers on the runtime:

• getManifest() to return the JSON Manifest that was included in the bundle code that is currently running.
• assertInputCompatibility() to validate that the input of a run is compatible with a new version of the code, asserts viability for "cancel-and-restart"
• assertEventLogCompatibility() to validate that the event log of a run is compatible with a new version of the code, asserts viability for "cancel-and-resume"

Manifest & Compatibility

Compatibility helpers will be based on comparing the Manifests of the old and new versions. See Manifest type in on the World Interface. The exact shape of the Manifest is not part of this RFC, and will be expanded upon separately.

A first iteration implementation of this RFC will not support compatibility checking, and will assume that all versions are compatible. upgrade will only support the "cancel-and-restart" strategy initially, will upgrade without checking compatibility, and the new run will simply fail if the input is incompatible, leaving the old run in pause state, with user code able to resume the old run if needed after an upgrade failure.

CLI Changes

New CLI commands will be added to expose the new World and Runtime API changes:

• workflow version create to create a new version
• workflow version update to update a version
• workflow version delete to delete a version
• workflow version list to list versions
• workflow version get to get a version
• workflow upgrade to upgrade the version of an existing run

And some existing commands will change to emit additional version information:

• workflow build will call world.versions.getCurrent() internally, or generate a new version ID if one is not provided, to save to the created manifest and bundle.
  • This is done so that a bundle will have direct access to its own version ID.

Deployment Changes

The bigger change that comes with versioning is the responsibilities of the World implementation. To understand these changes, let's take a look at the user experience of using a World.

CI Experience

This is an example CI script that a user might use to deploy their application:

# 1. Regular build step to create the bundle
npx workflow build
# → Calls world.versions.getCurrent() or generates new version ID
# → Outputs manifest.json with version metadata

# 2. Deploy the bundle to compute (implementation and world specific)
# A user might be deploying this to anything from a serverless platform to a
# bare metal instance with a custom homegrown routing solution.
# This process should result in the bundle being deployed and accessible on the web.

# 3. Create version, uploading the manifest and setting the address to the URL where the deploy lives
npx workflow version create ./build/manifest.json --address "https://..."
# → Calls world.versions.create({ manifest, address })
# → This should let the Queue implementation know / allow routing requests to the new version.

# 4. If any steps here modify the deployed version, they can call. This is usually not required, but depends on the user's setup and the World used
npx workflow version update <versionId> [...params]

# 5. If there any failures, the CI can clean up the version:
npx workflow version delete <versionId>

Opinionated vs. Flexible Worlds

In the above example, step (3) could be part of the World implementation, or down
to the user. This depends on the World. Imagine, for example, an "AWS World".

This world could either:

• Handle Queueing and Storage with SNS and DynamoDB
• Take your call to versions.update(id, { address }) and simply store it in DynamoDB
• When a request comes in to start a workflow with a certain version ID, it will route that request to the address provided, using it as a URL.
• Leave the actual deployment, routing, management, and provisioning of the bundle to the user.

Or, it could take over more of the user's responsibilities:

• Handle Queueing and Storage with SNS and DynamoDB
• When you call versions.create({ manifest }), it will:
  • require a bundle path to be set at call time, e.g. as a WORKFLOW_BUNDLE_PATH env variable
  • internally use its AWS config to provision a new Lambda function
  • upload the bundle to the Lambda function
  • wait for the Lambda function to be live
  • create a new API Gateway endpoint for it
  • confirm it's accessible
  • call versions.update( id, { address }) to store the address in DynamoDB, mainly to mark it as "live".
• Maintain a proxy to always route to the latest version's address for publicly accessible endpoints and serving your client and server code (e.g. if using Next.js, Remix, etc.)
• An incoming request to start a workflow will not need a lookup to the Version table, as it will use the version ID to directly construct and route the correct target address.

This example is independent of whether the World is based on a major cloud provider's serverless implementation, Supabase, Jazz, Netlify, self-hosted Kubernetes, etc.

These are two ends up a spectrum, here called "flexible" (user-defined) and "opinionated" (world-defined). At the very least, a world will need to support a Queue and Storage implementation, though these need not be the same implementation. A community npm package might implement a queue, and another a storage layer, and these can still be combined to form a minimal world.

It's especially important to note that versioning is optional. If you simply always deploy your latest instance of code, it will cause old workflows runs to stand still. However, they may still be upgraded manually by the user. A "flexible" world with the a minimal user-given compute layer is viable, though not recommended.

In the best case, a user may be able to download, e.g. an awesome-supabase-world package, provide a Supabase project ID and credentials, and do nothing but call workflow version create in Github CI to create a full (and possibly client/server) deployment with minimal configuration.

Minimal single-VM World Implementation Example

With a minimal production-ready world like the @workflow/world-postgres, which implements Storage and Queueing off of a single PostgresSQL instance, a user could create a production deploy like so:

• Set up a single VM with a static IP address
• When deploying a bundle
  • Upload the bundle to any folder on the VM
  • Start the nodejs instance in the background
  • This will run a pull-based queue worker against postgres
  • If CI is running on "main" branch, also terminate the instance running on Port 80 and replace with the new instance (or use a more sophisticated for instance swapping)
• Optionally, if we wanted to programmatically terminate old versions, e.g. older than 10 versions ago, we could save the port that the version is running under in the Version object and then run a job to query old versions and terminate process by port number.

Open Questions

1. The naming of “version” is interchangeable with “deployment”. Should this be renamed to Deployment/ deploymentId/ world.deployments/ run.deploymentId instead? It would make more sense semantically, but be less clearly tied to the idea of versioning.
2. Should the versionId be baked into the bundle as it currently is? The upside is that most worlds can implement versions.getCurrent() simply by calling getManifest().versionId from the runtime helpers, instead of requiring world implementations to set or expect environment variables. The downside is that we can't swap this ID out as easily as an environment variable, e.g. if we wanted to re-use a bundle across deploys of different environments. However, this is still possible if the World chooses to use environment variables for this purpose.
3. Should version.update() exist at all? It is mainly used to update a address, implicitly marking a Version as live, if it was initially created as “not live”. Given that the overhead is small to maintain this and we might want to allow arbitrary properties to the set on the Version object (for world internal management), it would make sense to keep the update call for forward compatibility. On the flip side, the conceptual simplicity of immutable Version objects could be nice too.
4. Should the World interface include a property to indicate if a Version is live? Coding this into Storage is unreliable, which is why the current address field is optional and doesn't come with a guarantee, but it would be nice if a call to e.g. versions.create({ setLive: true }) would only return if the bundle was deployed successfully and is accessible. A World may still choose to do this, but the question is whether it should be a required contract across Worlds. Similarly, there's no way to list "alive" versions, though again, the World may choose to model this based on the address field.
5. Should we add versions to .well-known route names, e.g. .well-known/workflow/v1/flow/vers_01ARZ3NDEKTSV4RRFFQ69G5FAV, also aliased as .well-known/workflow/v1/flow? This would allow combining bundles to allow a single nodejs instance to server multiple versions. Some Worlds might make use of this, though it is unlikely to be a good option.

[Wundero]

How would this work for users who are looking to change out only parts of the World implementation? I want to use workflows in a NextJS app deployed to Vercel, but due to various constraints I need to ensure data is stored in AWS (RDS/SQS). I have currently implemented a world which handles the existing interfaces for storage and queues, but am unsure of how I would have to implement these changes such that it can pull the correct info (e.g. deployment ID and build manifest) and route runs to the correct code.

Separately, in response to the open questions:

1. I personally think versions makes more sense in the long-term, especially if support for non-deployment-related versions is added for some reason. My opinion on this is not super strong though, I feel that both options are fairly clear for what is meant.
2. I think baking the version ID into the manifest makes sense, maybe called bundleVersion, and allowing each World implementation to choose whether to rely on the bundle version and/or environment variables to decide what ID a version has. More options for world implementers is good here IMO.
3. I think keeping version.update() makes sense, especially if a property is added to the Version iface to indicate liveness. I don't think immutable version objects is enough of a plus-side as compared to having the ability to control liveness and addresses, plus any future enhancements.
4. I think it makes sense to have a field to indicate liveness in the interface, though I am not sure I am the biggest fan of making the create command block until the world is deployed, especially in more complex CI workflows, as it feels like it railroads you a bit into the workflow deployment being implemented in the world. It could maybe be a good idea to have a separate function, e.g. version.waitForLive(), which handles the blocking separately.
   • Separately, what value would listing "alive" versions alone have that makes it worth mandating over just having the list function for all versions?
5. I think adding the versions to well-known routes, provided routes without versions exist and work properly, is a nice to have in some cases that doesn't really seem to have downsides.

[VaguelySerious]

I want to use workflows in a NextJS app deployed to Vercel with [the world using] AWS (RDS/SQS). I have [...] a world which handles the existing interfaces for storage and queues, [how do I extend the world] to pull the correct info (e.g. deployment ID and build manifest)

It's important to mention this spec should be backwards compatible, and if your World is currently working fine, it should continue to work fine after this spec is deployed.

If you want to upgrade your World according to this spec to allow for listing/upgrading versions and improved observability support, you can do the following:
Assuming your World runs storage/queues off of AWS, and compute off of Vercel, you can store the Version entity in RDS just like other storage items. Presumably, your queue already routes to Vercel correctly based on the Vercel deployment ID (if not, how are you doing it?). This means your world would only need to implement versions.getCurrent() to return VERCEL_DEPLOYMENT_ID, and pass that ID when calling versions.create()

[VaguelySerious]

4. I think it makes sense to have a field to indicate liveness in the interface, though I am not sure I am the biggest fan of making the create command block until the world is deployed, especially in more complex CI workflows

I agree, and whether a command is blocking or not is up to the world. create and update calls can be non-blocking, and your CI could easily include a polling loop that calls version.get(myVersionCurrentlyBeingDeployed) every ~5 seconds to wait for liveness, if that's a relevant part of the deploy process, before or after calling update.

What value would listing "alive" versions alone have

Assuming that you might have thousands of versions at some point, only a subset of which is live, being able to filter by liveness is just a query optimization on the storage layer, allowing you to possibly skip a pagination loop just to get the most relevant versions. This is no different from being able to filter on other properties when doing a CRUD list call, and your world may support list filtering by arbitrary properties stored in the version metadata.

[Wundero]

It's important to mention this spec should be backwards compatible, and if your World is currently working fine, it should continue to work fine after this spec is deployed.

If you want to upgrade your World according to this spec to allow for listing/upgrading versions and improved observability support, you can do the following:
Assuming your World runs storage/queues off of AWS, and compute off of Vercel, you can store the Version entity in RDS just like other storage items. Presumably, your queue already routes to Vercel correctly based on the Vercel deployment ID (if not, how are you doing it?). This means your world would only need to implement versions.getCurrent() to return VERCEL_DEPLOYMENT_ID, and pass that ID when calling versions.create()

This makes sense, thanks!

Assuming that you might have thousands of versions at some point, only a subset of which is live, being able to filter by liveness is just a query optimization on the storage layer, allowing you to possibly skip a pagination loop just to get the most relevant versions. This is no different from being able to filter on other properties when doing a CRUD list call, and your world may support list filtering by arbitrary properties stored in the version metadata.

Gotcha. I think it makes sense then, and I personally would like it to just be a parameter on the list function (in VersionListParams) just to keep the API surface a bit cleaner.

[eersnington]

What are the exact semantics when upgrading a workflow that is executing "mid-step", or waiting for webhook? that also brings me to question is there a defined set of "safe pause points" where an upgrade is allowed? I'm a bit confused on how step 4 of "cancel-and-restart" works

[VaguelySerious]

On "mid-step" pausing/upgrading

There's a distinction here between a step and awaiting hook or sleep calls. Hooks/sleeps are not really steps, in that they don't actively run, instead they do nothing until an event is fired (either a hook gets called, or a timer fires).
This means that upgrading a run during a sleep or await hook call will work, in that the old run will continue doing nothing, and the new run will wake up / resume when the sleep/hook events are done instead. For hooks/webhooks, the old webhook will be retired on the upgrade call, meaning that an in-flight hook request for the old run will trigger resume the new run if it arrives after the upgrade call goes through.

Upgrading a run in the middle of a user-defined step, however, MIGHT run the step twice. The initial implementation will likely run the step twice: the step will run in the old run/version, its output will be discarded, and the step will also run in the new run/version, and the output of that gets used in the continuation of the run.

Later on, we'll make this more robust by migrating the event log in a way that allow the new version of the run to see that the old run is already running the step, and instead redirect the output of the old-version step to the new-version step (if compatible), so that we don't run steps twice. However, there might still be a race-condition: upgrading a run can (very rarely) happen during a step execution change, in a way that makes the step complete both in the old run (to be discarded) and the new run. This is non-trivial to work around, though technically possible. We might not focus on this immediately, as we assume steps follow at-least-once-semantics, and the user impact of having a small amount of duplicate step runs is likely minimal, see below.

---

On idempotency / at-least-once

A "step" in a workflow is generally assumed to use the at-least-once semantic. Steps can fail and get re-tried, which means any side-effects inside steps are preferably idempotent, or don't affect the rest of the workflow resources except through their return value. Since this is the general pattern used for steps, it should similarly not be an issue if a workflow being upgraded ends up running a specific step twice.

To clarify, steps don't need to be idempotent. Instead, consider a side-effectful step that allocates an external resource, e.g. a VM or GPU. You could write the step as either of:

1. id => { "use step"; allocateResource(id); return id; }
2. () => { "use step"; allocateResource("my-global-id"); return void; }
3. () => { "use step"; const id = allocateResource(id); return id; }

if allocateResource is itself not idempotent and fails if the ID is already allocated, then all three examples might run into an error loop. This is generally discouraged. If allocateResource is itself idempotent and won't try creating existing IDs, or if the step function handles this basic issue, then all three examples will work fine, which is why the at-least-once-semantic is in place.

Not required, but for further optimization, examples (1) and (2) are better than (3), since (3) will create two resources, only one of which will be used by the workflow. Hence, it's best-practice that a step acts only on resources with deterministic identifiers, even if the code itself does not need to be deterministic, if only to prevent excess resource consumption.

[eersnington]

Ahh thank you for clarifying this! So conceptually, the core guarantee is that a step will run at least once, and users keep the following mental model while designing their workflows

• steps are at-least-once, not exactly-once
• retries can come from failures, replays, and now also from upgrades that happen mid-step

But I can give you a few problematic examples that can cause unintended side-effects which a developer can easily miss.

• If I have a sendWelcomeEmail(userId) step and I upgrade while it is running, I should assume that email might be sent twice unless I make the step or the mail layer idempotent
• If I have Stripe.Session.create(stripeCustomerId, amount), a mid-step upgrade can lead to the same charge being attempted twice unless I use idempotency keys on the Stripe side
• If I have Sandbox.create(vmId) and vmId is a random value generated inside the step, a mid-step upgrade can create two separate VMs. If vmId is deterministic and I handle “already exists” gracefully, then the extra execution is mostly wasted work, not a correctness issue

So from a dev's point of view, upgrades are effectively another source of "this step might run again" even if the first run succeeded. That feels totally fine as long as the docs and mental model are explicit about it: if you are going to call run.upgrade() on live runs, your step patterns need to be the same ones you would use for safe retries

[eersnington]

Also to clarify to anyone reading, these examples are intentionally bad ways of writing steps, and no workflow engine can magically fix them for you. Temporal, Conductor, AWS and Azure and all work on the same at-least-once principle, and they all expect the developer to make side effects safe step functions.

The point isn't that the WDK (or any workflow engine as matter of fact) should fix it, but that upgrades now join failures and replays as another trigger for re-execution of steps, so having that spelled out in the mental model helps developers avoid these kinds of mistakes. I hope that is clear 🫡

[VaguelySerious]

• steps are at-least-once, not exactly-once
• retries can come from failures, replays, and now also from upgrades that happen mid-step

That's a great way to summarize it.

problematic examples

Right - developers need to take some prudence when writing code that is costly (billing a customer, allocating an expensive resource), and that prudence is helpful in general for minimizing issues. For most of your examples, you can either do this by creating your action idempotency keys inside steps deterministically (e.g. userId + transactionId or similar), or by passing the ID as an input through the workflow:

function myWorkflow() {
  // This ID could also be the runID from `getWorkflowMetadata()`, or an ID tied to your object/user
  const id = createUuid();
  await myStep(id);
}

function myStep(idempotencyId: string) {
  // Charge a customer, or send an email, or allocate a VM
  mySensitiveAction(idempotencyId: string);
}

[eersnington]

during an upgrade, how does rehydrating the args of a workflow function work? the RFC says "cancel-and-restart" uses the "same inputs", but doesn't define how input shape compatibility is checked. for eg., what happens if the workflow function signature changes between versions, such as adding a new parameter or reordering existing ones?

[VaguelySerious]

This is a simpler answer: input shape compatibility is checked through the typescript type signature of both input/output. As long as the types are accurate. Will update the doc to say as much.

[eersnington]

Not sure if I missed something but how does a World know that sunsetting a version is safe and won't strand runs that still depend on it (hooks, and long sleeps)?

Your single VM example suggests that a user can sunset a workflow version manually (maybe it is inherently a --force operation but that's not clear either)

... if we wanted to programmatically terminate old versions, e.g. older than 10 versions ago...

Will there be an API through the WDK, CLI, or the observability web app to confirm that no active runs are still tied to that versionId?

Also to follow up on this:

• will the runtime/storage expose something like runs.list({ versionId }) or runs.count({ versionId })?
• if not, is it expected that Worlds must implement their own bookkeeping of active runs per version, or will this be left as a best-effort pattern?

[VaguelySerious]

Will there be an API through the WDK, CLI, or the observability web app to confirm that no active runs are still tied to that versionId?

As you said in the follow-up, we'll have runs.list({ versionId: <id>, status: 'running' }). It's just a filter param.

how does a World know that sunsetting a version is safe

Given the above, the world can query active runs for a version and make arbitrary choices based on that. For Worlds that handle compute, calls to version.create or version.update can do this internally. For Worlds with user-managed compute, like the single-VM example, the user can manage the same by calling the world list call directly and making decisions based on that