From 317bfd755efc7e977b8adca7e61599a88deba56a Mon Sep 17 00:00:00 2001 From: Luke Knepper Date: Mon, 29 Jun 2026 15:51:10 -0700 Subject: [PATCH 01/12] Clarifying workflow ID consistency and HA --- .../high-availability/failovers/index.mdx | 25 ++++++++++++++++++- .../workflow-execution/workflowid-runid.mdx | 10 ++++++++ 2 files changed, 34 insertions(+), 1 deletion(-) diff --git a/docs/cloud/high-availability/failovers/index.mdx b/docs/cloud/high-availability/failovers/index.mdx index 50b0018409..929d227636 100644 --- a/docs/cloud/high-availability/failovers/index.mdx +++ b/docs/cloud/high-availability/failovers/index.mdx @@ -146,13 +146,36 @@ remains active. Namespaces with replicas rely on asynchronous event replication. Updates made to the primary may not immediately be reflected in the replica due to , particularly during failovers. In the event of a -non-graceful failover, replication lag may cause a temporary setback in Workflow progress. +non-graceful failover, replication lag has two distinct effects, which are important to tell apart: + +- **A temporary setback in Workflow progress.** Operations that had already replicated remain durable in the replica. + Operations still in the replication backlog are reconciled when the original region recovers, so progress is set back + temporarily but not lost. +- **Operations that had not yet replicated at the moment of failover.** A Workflow start, Signal, Update, or other + operation that the active region accepted but had not yet replicated falls within the + [recovery point](/cloud/rpo-rto). If the original region recovers, these operations are reconciled into the active + Namespace and virtually nothing is lost. If the original region is _permanently_ lost — for example, an unrecoverable + cell outage — operations within the recovery point window may be lost entirely. Temporal Cloud targets a recovery + point of under one minute for these outages. + +A graceful failover avoids both effects: Temporal Cloud drains the replication backlog to zero and pauses traffic before +switching regions, so the replica holds every acknowledged operation and the Namespace achieves a recovery point of +zero. Namespaces that are not replicated can be configured to provide _at-most-once_ semantics for Activity execution when a retry policy's [maximum attempts](https://docs.temporal.io/retry-policies#maximum-attempts) is set to 0. High Availability Namespaces provide _at-least-once_ semantics for execution of Activities. Completed Activities _may_ be re-dispatched in a newly active Namespace, leading to repeated executions. +The same durability boundary applies to Workflow starts and Signals: a `StartWorkflowExecution` or +`SignalWorkflowExecution` call that returns success is durably committed in the active region, but an operation that had +not yet replicated could be absent from the newly active region after a forced failover with permanent regional loss, as +described above. A successful start is therefore not guaranteed to survive a permanent loss of the original region if it +fell within the recovery point window. + +The [Workflow Id uniqueness guarantee](/workflow-execution/workflowid-runid) — at most one Open Workflow Execution per +Workflow Id — is always enforced within the active Namespace, and conflict resolution preserves it across a failover. + When a Workflow Execution is updated in a newly active replica following a failover, events from the previously active Namespace that arrive after the failover cannot be directly applied. At this point, Temporal Cloud has forked the Event History. diff --git a/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx b/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx index 23bf90699e..8bf7e6f914 100644 --- a/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx +++ b/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx @@ -27,6 +27,16 @@ In some cases, such as when running the same Workflow at recurring intervals usi In this case, all runs will have the same Workflow ID. However, each run will have a unique system-generated [Run ID](#run-id). +:::note Workflow Id uniqueness and High Availability + +This uniqueness constraint is enforced within the active [Namespace](/namespaces). +On a [High Availability](/cloud/high-availability) Namespace, replication is asynchronous, and a failover preserves the constraint through Temporal Cloud's [conflict resolution](/cloud/high-availability/failovers#conflict-resolution) process. +A Workflow start that the active region accepted but had not yet replicated falls within the [recovery point](/cloud/rpo-rto): if the original region is permanently lost, that start can be absent from the newly active region. +In other words, a failover never causes the same Workflow Id to run twice concurrently, but a non-graceful failover with permanent regional loss can drop a start that had not yet replicated. +See [conflict resolution](/cloud/high-availability/failovers#conflict-resolution) for details. + +::: + ## What is a Run Id? {/* #run-id */} A Run Id is a globally unique, platform-level identifier for a [Workflow Execution](/workflow-execution). From 34f95b567b445589061df68bc0df9d0cf0d0910a Mon Sep 17 00:00:00 2001 From: Luke Knepper Date: Wed, 1 Jul 2026 11:58:58 -0700 Subject: [PATCH 02/12] Apply review feedback: caution admonition for RPO data loss Address reviewer comments on the Conflict resolution section: - Remove "virtually nothing is lost" wording - Reference the Conflict Resolution process instead - Move permanent-loss detail into a :::caution admonition Co-Authored-By: Claude Opus 4.8 --- .../high-availability/failovers/index.mdx | 57 ++++++++++++------- 1 file changed, 37 insertions(+), 20 deletions(-) diff --git a/docs/cloud/high-availability/failovers/index.mdx b/docs/cloud/high-availability/failovers/index.mdx index 929d227636..366ee26479 100644 --- a/docs/cloud/high-availability/failovers/index.mdx +++ b/docs/cloud/high-availability/failovers/index.mdx @@ -31,9 +31,10 @@ failover, Temporal automatically fails back to the original region once it is he ## Automatic failover -Temporal Cloud offers managed outage detection and failover to all Namespaces that use High Availability. -These automatic failovers keep your Namespace available without manual intervention. Temporal aims to both detect the outage and complete a failover in minutes from when the outage began, -according to the stated [Recovery Time Objective (RTO)](/cloud/rpo-rto). +Temporal Cloud offers managed outage detection and failover to all Namespaces that use High Availability. These +automatic failovers keep your Namespace available without manual intervention. Temporal aims to both detect the outage +and complete a failover in minutes from when the outage began, according to the stated +[Recovery Time Objective (RTO)](/cloud/rpo-rto). After an automatic failover, the Namespace will have a replica in its original region. Once the original region is healthy again, Temporal Cloud automatically performs a [failback](/cloud/high-availability/failovers/manage#failbacks), @@ -64,8 +65,8 @@ regional outage. :::info -The following list gives a general idea of the conditions that trigger an automatic failover. This is not an -exhaustive list, and it may change over time. +The following list gives a general idea of the conditions that trigger an automatic failover. This is not an exhaustive +list, and it may change over time. ::: @@ -80,22 +81,23 @@ exhaustive list, and it may change over time. You can also [manually trigger a failover](/cloud/high-availability/failovers/manage#trigger-failover) based on your own monitoring or for failover testing. -Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover -is warranted are: +Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover is +warranted are: - **Testing failover or migrating to a new region.** A manual failover is the standard way to exercise your failover process with your Clients and Workers, or to move a Namespace to a different region. - **An outage that affects only your systems.** If an outage is contained to your application, Workers, or other infrastructure, and Temporal Cloud is not affected, Temporal will not initiate a failover on your behalf. Detect the outage with your own monitoring and trigger a failover yourself. -- **Failing over more aggressively during a regional outage.** Even with automatic failovers enabled, you can - trigger a failover yourself if you detect a regional outage before Temporal does. Whichever failover happens first - takes effect, and the later one is a no-op. A user-triggered failover does not conflict with Temporal's automatic - failover. +- **Failing over more aggressively during a regional outage.** Even with automatic failovers enabled, you can trigger a + failover yourself if you detect a regional outage before Temporal does. Whichever failover happens first takes effect, + and the later one is a no-op. A user-triggered failover does not conflict with Temporal's automatic failover. :::note Same-region Replication -Manual failovers apply only to Multi-region and Multi-cloud Replication. A [Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over automatically between cells and cannot be failed over manually or have its automatic failovers disabled. +Manual failovers apply only to Multi-region and Multi-cloud Replication. A +[Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over automatically between +cells and cannot be failed over manually or have its automatic failovers disabled. ::: @@ -127,9 +129,15 @@ The failover process is the same whether it is triggered automatically by Tempor ## Post-failover events {/* #post-failover-events */} -After any failover, whether triggered by you or by Temporal, an event appears in both the [Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. The audit log entry uses the `"operation": "FailoverNamespace"` event. Temporal Cloud [notifies you via email](/cloud/notifications#admin-notifications) whenever a failover occurs. +After any failover, whether triggered by you or by Temporal, an event appears in both the +[Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. The +audit log entry uses the `"operation": "FailoverNamespace"` event. Temporal Cloud +[notifies you via email](/cloud/notifications#admin-notifications) whenever a failover occurs. -After an automatic failover, Temporal automatically fails back to the original region once the region is healthy, unless you [opt out](/cloud/high-availability/failovers/manage#after-an-automatic-failover). After a user-triggered failover, the Namespace stays in the replica region until a user triggers another failover. See [failback options](/cloud/high-availability/failovers/manage#failbacks) for details. +After an automatic failover, Temporal automatically fails back to the original region once the region is healthy, unless +you [opt out](/cloud/high-availability/failovers/manage#after-an-automatic-failover). After a user-triggered failover, +the Namespace stays in the replica region until a user triggers another failover. See +[failback options](/cloud/high-availability/failovers/manage#failbacks) for details. ## Split-brain scenario @@ -146,17 +154,24 @@ remains active. Namespaces with replicas rely on asynchronous event replication. Updates made to the primary may not immediately be reflected in the replica due to , particularly during failovers. In the event of a -non-graceful failover, replication lag has two distinct effects, which are important to tell apart: +non-graceful failover, replication lag has two distinct effects: - **A temporary setback in Workflow progress.** Operations that had already replicated remain durable in the replica. Operations still in the replication backlog are reconciled when the original region recovers, so progress is set back temporarily but not lost. - **Operations that had not yet replicated at the moment of failover.** A Workflow start, Signal, Update, or other operation that the active region accepted but had not yet replicated falls within the - [recovery point](/cloud/rpo-rto). If the original region recovers, these operations are reconciled into the active - Namespace and virtually nothing is lost. If the original region is _permanently_ lost — for example, an unrecoverable - cell outage — operations within the recovery point window may be lost entirely. Temporal Cloud targets a recovery - point of under one minute for these outages. + [recovery point](/cloud/rpo-rto). If the original region recovers, those operations are reconciled into the active + Namespace according to the Conflict Resolution process. If the original region is _permanently_ lost — for example, an + unrecoverable cell outage — operations within the recovery point window may be lost entirely. Temporal Cloud targets a + recovery point of under one minute for these outages. + +:::caution + +Conflict Resolution can only recover data from a functioning Temporal Server. If the active server completely fails and +does not recover, Workflow API calls that fall within the 1 minute RPO may be permanently lost. + +::: A graceful failover avoids both effects: Temporal Cloud drains the replication backlog to zero and pauses traffic before switching regions, so the replica holds every acknowledged operation and the Namespace achieves a recovery point of @@ -180,4 +195,6 @@ When a Workflow Execution is updated in a newly active replica following a failo Namespace that arrive after the failover cannot be directly applied. At this point, Temporal Cloud has forked the Event History. -After failover, Temporal Cloud creates a new branch history for execution and begins its process. The Temporal Service ensures that Event Histories remain valid and are replayable by SDKs post-failover or after conflict resolution. +After failover, Temporal Cloud creates a new branch history for execution and begins its + process. The Temporal Service ensures that Event Histories remain valid and +are replayable by SDKs post-failover or after conflict resolution. From 4aa3276f5d42b8ed9f79d40a1ab366c372f69ea4 Mon Sep 17 00:00:00 2001 From: Luke Knepper Date: Wed, 1 Jul 2026 12:04:32 -0700 Subject: [PATCH 03/12] Simplify note on Workflow Id uniqueness Removed redundant explanation about Workflow Id uniqueness during failover scenarios. --- .../workflow/workflow-execution/workflowid-runid.mdx | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx b/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx index 8bf7e6f914..050366a4fc 100644 --- a/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx +++ b/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx @@ -31,9 +31,6 @@ However, each run will have a unique system-generated [Run ID](#run-id). This uniqueness constraint is enforced within the active [Namespace](/namespaces). On a [High Availability](/cloud/high-availability) Namespace, replication is asynchronous, and a failover preserves the constraint through Temporal Cloud's [conflict resolution](/cloud/high-availability/failovers#conflict-resolution) process. -A Workflow start that the active region accepted but had not yet replicated falls within the [recovery point](/cloud/rpo-rto): if the original region is permanently lost, that start can be absent from the newly active region. -In other words, a failover never causes the same Workflow Id to run twice concurrently, but a non-graceful failover with permanent regional loss can drop a start that had not yet replicated. -See [conflict resolution](/cloud/high-availability/failovers#conflict-resolution) for details. ::: From 6ff7dfe73269602445e2b5cf4c496a86060261e7 Mon Sep 17 00:00:00 2001 From: Luke Knepper Date: Wed, 1 Jul 2026 12:06:18 -0700 Subject: [PATCH 04/12] Refine manual failover explanation in documentation Clarify the definition of manual failover and its scenarios. --- docs/cloud/high-availability/failovers/index.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/cloud/high-availability/failovers/index.mdx b/docs/cloud/high-availability/failovers/index.mdx index 366ee26479..c37cdc0e4c 100644 --- a/docs/cloud/high-availability/failovers/index.mdx +++ b/docs/cloud/high-availability/failovers/index.mdx @@ -81,8 +81,8 @@ list, and it may change over time. You can also [manually trigger a failover](/cloud/high-availability/failovers/manage#trigger-failover) based on your own monitoring or for failover testing. -Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover is -warranted are: +Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover (i.e., +a failover triggered by a user) is warranted are: - **Testing failover or migrating to a new region.** A manual failover is the standard way to exercise your failover process with your Clients and Workers, or to move a Namespace to a different region. @@ -91,7 +91,7 @@ warranted are: outage with your own monitoring and trigger a failover yourself. - **Failing over more aggressively during a regional outage.** Even with automatic failovers enabled, you can trigger a failover yourself if you detect a regional outage before Temporal does. Whichever failover happens first takes effect, - and the later one is a no-op. A user-triggered failover does not conflict with Temporal's automatic failover. + and the later one is a no-op. A manual failover does not conflict with Temporal's automatic failover. :::note Same-region Replication From 1ca47e4cda6948d75c4204f0a72d1ebc5b8513a9 Mon Sep 17 00:00:00 2001 From: Luke Knepper Date: Wed, 1 Jul 2026 12:08:10 -0700 Subject: [PATCH 05/12] Update caution note on data loss during failovers Clarify the conditions under which data may be permanently lost during a server failure. --- docs/cloud/high-availability/failovers/index.mdx | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/cloud/high-availability/failovers/index.mdx b/docs/cloud/high-availability/failovers/index.mdx index c37cdc0e4c..8f86388ca4 100644 --- a/docs/cloud/high-availability/failovers/index.mdx +++ b/docs/cloud/high-availability/failovers/index.mdx @@ -169,7 +169,8 @@ non-graceful failover, replication lag has two distinct effects: :::caution Conflict Resolution can only recover data from a functioning Temporal Server. If the active server completely fails and -does not recover, Workflow API calls that fall within the 1 minute RPO may be permanently lost. +does not recover, Workflow API calls that fall within the 1 minute RPO may be permanently lost. Such a case would require +the permanent loss of multiple cloud Availability Zones and has never happened in the history of Temporal Cloud. ::: From 111be20c470a58860a7ec0ee97acf765b08beaaf Mon Sep 17 00:00:00 2001 From: Luke Knepper Date: Wed, 1 Jul 2026 12:09:35 -0700 Subject: [PATCH 06/12] Update failover documentation for clarity Clarified the conditions under which operations may be lost during failover and emphasized the importance of a functioning Temporal Server for Conflict Resolution. --- docs/cloud/high-availability/failovers/index.mdx | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/docs/cloud/high-availability/failovers/index.mdx b/docs/cloud/high-availability/failovers/index.mdx index 8f86388ca4..618b9ce662 100644 --- a/docs/cloud/high-availability/failovers/index.mdx +++ b/docs/cloud/high-availability/failovers/index.mdx @@ -162,18 +162,11 @@ non-graceful failover, replication lag has two distinct effects: - **Operations that had not yet replicated at the moment of failover.** A Workflow start, Signal, Update, or other operation that the active region accepted but had not yet replicated falls within the [recovery point](/cloud/rpo-rto). If the original region recovers, those operations are reconciled into the active - Namespace according to the Conflict Resolution process. If the original region is _permanently_ lost — for example, an - unrecoverable cell outage — operations within the recovery point window may be lost entirely. Temporal Cloud targets a - recovery point of under one minute for these outages. - -:::caution - -Conflict Resolution can only recover data from a functioning Temporal Server. If the active server completely fails and -does not recover, Workflow API calls that fall within the 1 minute RPO may be permanently lost. Such a case would require + Namespace according to the Conflict Resolution process. + - Note that Conflict Resolution can only recover data from a functioning Temporal Server. If the active server never +recovers, the Workflow API calls that fall within the 1 minute RPO may be permanently lost. Such a case would require the permanent loss of multiple cloud Availability Zones and has never happened in the history of Temporal Cloud. -::: - A graceful failover avoids both effects: Temporal Cloud drains the replication backlog to zero and pauses traffic before switching regions, so the replica holds every acknowledged operation and the Namespace achieves a recovery point of zero. From 81fe9a86cded5e244a58b21502276286174c3db4 Mon Sep 17 00:00:00 2001 From: Luke Knepper Date: Wed, 1 Jul 2026 12:11:23 -0700 Subject: [PATCH 07/12] Update failover section for Workflow execution semantics Clarify the durability of Workflow starts and Signals during failovers. --- docs/cloud/high-availability/failovers/index.mdx | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/cloud/high-availability/failovers/index.mdx b/docs/cloud/high-availability/failovers/index.mdx index 618b9ce662..da705998e9 100644 --- a/docs/cloud/high-availability/failovers/index.mdx +++ b/docs/cloud/high-availability/failovers/index.mdx @@ -177,10 +177,8 @@ Availability Namespaces provide _at-least-once_ semantics for execution of Activ re-dispatched in a newly active Namespace, leading to repeated executions. The same durability boundary applies to Workflow starts and Signals: a `StartWorkflowExecution` or -`SignalWorkflowExecution` call that returns success is durably committed in the active region, but an operation that had -not yet replicated could be absent from the newly active region after a forced failover with permanent regional loss, as -described above. A successful start is therefore not guaranteed to survive a permanent loss of the original region if it -fell within the recovery point window. +`SignalWorkflowExecution` call that returns success is durably committed in the active region, and replicated asynchronously +to the replica. The [Workflow Id uniqueness guarantee](/workflow-execution/workflowid-runid) — at most one Open Workflow Execution per Workflow Id — is always enforced within the active Namespace, and conflict resolution preserves it across a failover. From 25259e049c663786e7068fa72d50153b91470847 Mon Sep 17 00:00:00 2001 From: Luke Knepper Date: Wed, 1 Jul 2026 12:14:09 -0700 Subject: [PATCH 08/12] Update conflict resolution section for clarity Clarify effects of non-graceful failover on Workflow progress and replication. --- docs/cloud/high-availability/failovers/index.mdx | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/docs/cloud/high-availability/failovers/index.mdx b/docs/cloud/high-availability/failovers/index.mdx index da705998e9..2386f813e3 100644 --- a/docs/cloud/high-availability/failovers/index.mdx +++ b/docs/cloud/high-availability/failovers/index.mdx @@ -154,20 +154,16 @@ remains active. Namespaces with replicas rely on asynchronous event replication. Updates made to the primary may not immediately be reflected in the replica due to , particularly during failovers. In the event of a -non-graceful failover, replication lag has two distinct effects: - -- **A temporary setback in Workflow progress.** Operations that had already replicated remain durable in the replica. - Operations still in the replication backlog are reconciled when the original region recovers, so progress is set back - temporarily but not lost. -- **Operations that had not yet replicated at the moment of failover.** A Workflow start, Signal, Update, or other - operation that the active region accepted but had not yet replicated falls within the - [recovery point](/cloud/rpo-rto). If the original region recovers, those operations are reconciled into the active - Namespace according to the Conflict Resolution process. +non-graceful failover, replication lag causes temporary setback in Workflow progress. At the moment of non-graceful failover: + +- Operations that had already replicated remain durable in the replica. +- Operations that had not yet replicated (i.e., that are still in the replication backlog) are reconciled when the region recovers. + according to the Conflict Resolution process. - Note that Conflict Resolution can only recover data from a functioning Temporal Server. If the active server never recovers, the Workflow API calls that fall within the 1 minute RPO may be permanently lost. Such a case would require the permanent loss of multiple cloud Availability Zones and has never happened in the history of Temporal Cloud. -A graceful failover avoids both effects: Temporal Cloud drains the replication backlog to zero and pauses traffic before +In a graceful failover, Temporal Cloud drains the replication backlog to zero and pauses traffic before switching regions, so the replica holds every acknowledged operation and the Namespace achieves a recovery point of zero. From 383afbcaef0744ab2fc3950b5c399bbcaf29628e Mon Sep 17 00:00:00 2001 From: Luke Knepper Date: Wed, 1 Jul 2026 12:27:00 -0700 Subject: [PATCH 09/12] Add section explaining how Workflow Id uniqueness survives forced failover Expand the bare assertion that conflict resolution preserves Workflow Id uniqueness into a titled subsection that walks the mechanism: steady-state enforcement, divergence during a forced/split-brain failover, and the fork-not-merge reconciliation that leaves a single Open Execution per Workflow Id. Notes that uniqueness is distinct from durability. Co-Authored-By: Claude Opus 4.8 --- .../high-availability/failovers/index.mdx | 52 ++++++++++++------- 1 file changed, 32 insertions(+), 20 deletions(-) diff --git a/docs/cloud/high-availability/failovers/index.mdx b/docs/cloud/high-availability/failovers/index.mdx index 2386f813e3..d23359df7e 100644 --- a/docs/cloud/high-availability/failovers/index.mdx +++ b/docs/cloud/high-availability/failovers/index.mdx @@ -81,7 +81,7 @@ list, and it may change over time. You can also [manually trigger a failover](/cloud/high-availability/failovers/manage#trigger-failover) based on your own monitoring or for failover testing. -Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover (i.e., +Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover (i.e., a failover triggered by a user) is warranted are: - **Testing failover or migrating to a new region.** A manual failover is the standard way to exercise your failover @@ -154,18 +154,18 @@ remains active. Namespaces with replicas rely on asynchronous event replication. Updates made to the primary may not immediately be reflected in the replica due to , particularly during failovers. In the event of a -non-graceful failover, replication lag causes temporary setback in Workflow progress. At the moment of non-graceful failover: +non-graceful failover, replication lag causes temporary setback in Workflow progress. At the moment of non-graceful +failover: - Operations that had already replicated remain durable in the replica. -- Operations that had not yet replicated (i.e., that are still in the replication backlog) are reconciled when the region recovers. - according to the Conflict Resolution process. - - Note that Conflict Resolution can only recover data from a functioning Temporal Server. If the active server never -recovers, the Workflow API calls that fall within the 1 minute RPO may be permanently lost. Such a case would require -the permanent loss of multiple cloud Availability Zones and has never happened in the history of Temporal Cloud. +- Operations that had not yet replicated (i.e., that are still in the replication backlog) are reconciled when the + region recovers. according to the Conflict Resolution process. - Note that Conflict Resolution can only recover data + from a functioning Temporal Server. If the active server never recovers, the Workflow API calls that fall within the 1 + minute RPO may be permanently lost. Such a case would require the permanent loss of multiple cloud Availability Zones + and has never happened in the history of Temporal Cloud. -In a graceful failover, Temporal Cloud drains the replication backlog to zero and pauses traffic before -switching regions, so the replica holds every acknowledged operation and the Namespace achieves a recovery point of -zero. +In a graceful failover, Temporal Cloud drains the replication backlog to zero and pauses traffic before switching +regions, so the replica holds every acknowledged operation and the Namespace achieves a recovery point of zero. Namespaces that are not replicated can be configured to provide _at-most-once_ semantics for Activity execution when a retry policy's [maximum attempts](https://docs.temporal.io/retry-policies#maximum-attempts) is set to 0. High @@ -173,16 +173,28 @@ Availability Namespaces provide _at-least-once_ semantics for execution of Activ re-dispatched in a newly active Namespace, leading to repeated executions. The same durability boundary applies to Workflow starts and Signals: a `StartWorkflowExecution` or -`SignalWorkflowExecution` call that returns success is durably committed in the active region, and replicated asynchronously -to the replica. +`SignalWorkflowExecution` call that returns success is durably committed in the active region, and replicated +asynchronously to the replica. + +### How Workflow Id uniqueness is preserved after a forced failover The [Workflow Id uniqueness guarantee](/workflow-execution/workflowid-runid) — at most one Open Workflow Execution per Workflow Id — is always enforced within the active Namespace, and conflict resolution preserves it across a failover. - -When a Workflow Execution is updated in a newly active replica following a failover, events from the previously active -Namespace that arrive after the failover cannot be directly applied. At this point, Temporal Cloud has forked the Event -History. - -After failover, Temporal Cloud creates a new branch history for execution and begins its - process. The Temporal Service ensures that Event Histories remain valid and -are replayable by SDKs post-failover or after conflict resolution. +The guarantee limits how many Executions are _Open_ at the same time, not how many +[Run Ids](/workflow-execution/workflowid-runid#run-id) a Workflow Id accumulates over its lifetime. That distinction is +what lets conflict resolution reconcile a divergence without ever running the same Workflow Id twice concurrently. + +1. **Steady state.** The active region enforces uniqueness on every write and asynchronously replicates the Event + History to the replica. Because only the active region accepts writes, the guarantee holds trivially. +2. **Failover with divergence.** In a forced failover — or a [split-brain scenario](#split-brain-scenario) where the + replica is promoted during a network partition — both regions can independently append events to the same Workflow + Execution. When the regions reconnect, their Event Histories have diverged for that Workflow Id. +3. **Fork instead of merge.** Temporal Cloud does not interleave the divergent histories. Events from the previously + active Namespace that arrive after the failover cannot be directly applied, so Temporal Cloud forks the Event History + and creates a new branch history, each branch identified by its own Run Id. Its + process keeps one branch as the Open Execution and supersedes the other, + leaving exactly one Open Workflow Execution per Workflow Id. The Temporal Service ensures the resulting Event + Histories remain valid and replayable by SDKs. + +This guarantee is separate from durability: a forced failover never runs the same Workflow Id twice, but a start that +had not yet replicated can still be lost if the original region is permanently lost, as described above. From f1785c15066c9497dabb1b05e8b3c34bd1e56e90 Mon Sep 17 00:00:00 2001 From: Luke Knepper Date: Wed, 1 Jul 2026 12:28:10 -0700 Subject: [PATCH 10/12] Trim redundant clause from steady-state uniqueness note Co-Authored-By: Claude Opus 4.8 --- docs/cloud/high-availability/failovers/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cloud/high-availability/failovers/index.mdx b/docs/cloud/high-availability/failovers/index.mdx index d23359df7e..5e489814b2 100644 --- a/docs/cloud/high-availability/failovers/index.mdx +++ b/docs/cloud/high-availability/failovers/index.mdx @@ -185,7 +185,7 @@ The guarantee limits how many Executions are _Open_ at the same time, not how ma what lets conflict resolution reconcile a divergence without ever running the same Workflow Id twice concurrently. 1. **Steady state.** The active region enforces uniqueness on every write and asynchronously replicates the Event - History to the replica. Because only the active region accepts writes, the guarantee holds trivially. + History to the replica. 2. **Failover with divergence.** In a forced failover — or a [split-brain scenario](#split-brain-scenario) where the replica is promoted during a network partition — both regions can independently append events to the same Workflow Execution. When the regions reconnect, their Event Histories have diverged for that Workflow Id. From d84dd88a25c00979a1fa511bdc934b96b1a96f4c Mon Sep 17 00:00:00 2001 From: Luke Knepper Date: Wed, 1 Jul 2026 12:29:12 -0700 Subject: [PATCH 11/12] Simplify divergence condition in uniqueness note Co-Authored-By: Claude Opus 4.8 --- docs/cloud/high-availability/failovers/index.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/cloud/high-availability/failovers/index.mdx b/docs/cloud/high-availability/failovers/index.mdx index 5e489814b2..8ee195019f 100644 --- a/docs/cloud/high-availability/failovers/index.mdx +++ b/docs/cloud/high-availability/failovers/index.mdx @@ -186,9 +186,9 @@ what lets conflict resolution reconcile a divergence without ever running the sa 1. **Steady state.** The active region enforces uniqueness on every write and asynchronously replicates the Event History to the replica. -2. **Failover with divergence.** In a forced failover — or a [split-brain scenario](#split-brain-scenario) where the - replica is promoted during a network partition — both regions can independently append events to the same Workflow - Execution. When the regions reconnect, their Event Histories have diverged for that Workflow Id. +2. **Failover with divergence.** In a forced failover when replication lag is present, both regions can independently + append events to the same Workflow Execution. When the regions reconnect, their Event Histories have diverged for + that Workflow Id. 3. **Fork instead of merge.** Temporal Cloud does not interleave the divergent histories. Events from the previously active Namespace that arrive after the failover cannot be directly applied, so Temporal Cloud forks the Event History and creates a new branch history, each branch identified by its own Run Id. Its From f58c3022814911cd78f601501d3354ea7e9bab29 Mon Sep 17 00:00:00 2001 From: Luke Knepper Date: Wed, 1 Jul 2026 12:31:03 -0700 Subject: [PATCH 12/12] Remove durability-vs-uniqueness caveat sentence Co-Authored-By: Claude Opus 4.8 --- docs/cloud/high-availability/failovers/index.mdx | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/cloud/high-availability/failovers/index.mdx b/docs/cloud/high-availability/failovers/index.mdx index 8ee195019f..694e5b4a83 100644 --- a/docs/cloud/high-availability/failovers/index.mdx +++ b/docs/cloud/high-availability/failovers/index.mdx @@ -195,6 +195,3 @@ what lets conflict resolution reconcile a divergence without ever running the sa process keeps one branch as the Open Execution and supersedes the other, leaving exactly one Open Workflow Execution per Workflow Id. The Temporal Service ensures the resulting Event Histories remain valid and replayable by SDKs. - -This guarantee is separate from durability: a forced failover never runs the same Workflow Id twice, but a start that -had not yet replicated can still be lost if the original region is permanently lost, as described above.