diff --git a/docs/cloud/high-availability/failovers/index.mdx b/docs/cloud/high-availability/failovers/index.mdx index 50b0018409..694e5b4a83 100644 --- a/docs/cloud/high-availability/failovers/index.mdx +++ b/docs/cloud/high-availability/failovers/index.mdx @@ -31,9 +31,10 @@ failover, Temporal automatically fails back to the original region once it is he ## Automatic failover -Temporal Cloud offers managed outage detection and failover to all Namespaces that use High Availability. -These automatic failovers keep your Namespace available without manual intervention. Temporal aims to both detect the outage and complete a failover in minutes from when the outage began, -according to the stated [Recovery Time Objective (RTO)](/cloud/rpo-rto). +Temporal Cloud offers managed outage detection and failover to all Namespaces that use High Availability. These +automatic failovers keep your Namespace available without manual intervention. Temporal aims to both detect the outage +and complete a failover in minutes from when the outage began, according to the stated +[Recovery Time Objective (RTO)](/cloud/rpo-rto). After an automatic failover, the Namespace will have a replica in its original region. Once the original region is healthy again, Temporal Cloud automatically performs a [failback](/cloud/high-availability/failovers/manage#failbacks), @@ -64,8 +65,8 @@ regional outage. :::info -The following list gives a general idea of the conditions that trigger an automatic failover. This is not an -exhaustive list, and it may change over time. +The following list gives a general idea of the conditions that trigger an automatic failover. This is not an exhaustive +list, and it may change over time. ::: @@ -80,22 +81,23 @@ exhaustive list, and it may change over time. You can also [manually trigger a failover](/cloud/high-availability/failovers/manage#trigger-failover) based on your own monitoring or for failover testing. -Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover -is warranted are: +Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover (i.e., +a failover triggered by a user) is warranted are: - **Testing failover or migrating to a new region.** A manual failover is the standard way to exercise your failover process with your Clients and Workers, or to move a Namespace to a different region. - **An outage that affects only your systems.** If an outage is contained to your application, Workers, or other infrastructure, and Temporal Cloud is not affected, Temporal will not initiate a failover on your behalf. Detect the outage with your own monitoring and trigger a failover yourself. -- **Failing over more aggressively during a regional outage.** Even with automatic failovers enabled, you can - trigger a failover yourself if you detect a regional outage before Temporal does. Whichever failover happens first - takes effect, and the later one is a no-op. A user-triggered failover does not conflict with Temporal's automatic - failover. +- **Failing over more aggressively during a regional outage.** Even with automatic failovers enabled, you can trigger a + failover yourself if you detect a regional outage before Temporal does. Whichever failover happens first takes effect, + and the later one is a no-op. A manual failover does not conflict with Temporal's automatic failover. :::note Same-region Replication -Manual failovers apply only to Multi-region and Multi-cloud Replication. A [Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over automatically between cells and cannot be failed over manually or have its automatic failovers disabled. +Manual failovers apply only to Multi-region and Multi-cloud Replication. A +[Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over automatically between +cells and cannot be failed over manually or have its automatic failovers disabled. ::: @@ -127,9 +129,15 @@ The failover process is the same whether it is triggered automatically by Tempor ## Post-failover events {/* #post-failover-events */} -After any failover, whether triggered by you or by Temporal, an event appears in both the [Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. The audit log entry uses the `"operation": "FailoverNamespace"` event. Temporal Cloud [notifies you via email](/cloud/notifications#admin-notifications) whenever a failover occurs. +After any failover, whether triggered by you or by Temporal, an event appears in both the +[Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. The +audit log entry uses the `"operation": "FailoverNamespace"` event. Temporal Cloud +[notifies you via email](/cloud/notifications#admin-notifications) whenever a failover occurs. -After an automatic failover, Temporal automatically fails back to the original region once the region is healthy, unless you [opt out](/cloud/high-availability/failovers/manage#after-an-automatic-failover). After a user-triggered failover, the Namespace stays in the replica region until a user triggers another failover. See [failback options](/cloud/high-availability/failovers/manage#failbacks) for details. +After an automatic failover, Temporal automatically fails back to the original region once the region is healthy, unless +you [opt out](/cloud/high-availability/failovers/manage#after-an-automatic-failover). After a user-triggered failover, +the Namespace stays in the replica region until a user triggers another failover. See +[failback options](/cloud/high-availability/failovers/manage#failbacks) for details. ## Split-brain scenario @@ -146,15 +154,44 @@ remains active. Namespaces with replicas rely on asynchronous event replication. Updates made to the primary may not immediately be reflected in the replica due to , particularly during failovers. In the event of a -non-graceful failover, replication lag may cause a temporary setback in Workflow progress. +non-graceful failover, replication lag causes temporary setback in Workflow progress. At the moment of non-graceful +failover: + +- Operations that had already replicated remain durable in the replica. +- Operations that had not yet replicated (i.e., that are still in the replication backlog) are reconciled when the + region recovers. according to the Conflict Resolution process. - Note that Conflict Resolution can only recover data + from a functioning Temporal Server. If the active server never recovers, the Workflow API calls that fall within the 1 + minute RPO may be permanently lost. Such a case would require the permanent loss of multiple cloud Availability Zones + and has never happened in the history of Temporal Cloud. + +In a graceful failover, Temporal Cloud drains the replication backlog to zero and pauses traffic before switching +regions, so the replica holds every acknowledged operation and the Namespace achieves a recovery point of zero. Namespaces that are not replicated can be configured to provide _at-most-once_ semantics for Activity execution when a retry policy's [maximum attempts](https://docs.temporal.io/retry-policies#maximum-attempts) is set to 0. High Availability Namespaces provide _at-least-once_ semantics for execution of Activities. Completed Activities _may_ be re-dispatched in a newly active Namespace, leading to repeated executions. -When a Workflow Execution is updated in a newly active replica following a failover, events from the previously active -Namespace that arrive after the failover cannot be directly applied. At this point, Temporal Cloud has forked the Event -History. - -After failover, Temporal Cloud creates a new branch history for execution and begins its process. The Temporal Service ensures that Event Histories remain valid and are replayable by SDKs post-failover or after conflict resolution. +The same durability boundary applies to Workflow starts and Signals: a `StartWorkflowExecution` or +`SignalWorkflowExecution` call that returns success is durably committed in the active region, and replicated +asynchronously to the replica. + +### How Workflow Id uniqueness is preserved after a forced failover + +The [Workflow Id uniqueness guarantee](/workflow-execution/workflowid-runid) — at most one Open Workflow Execution per +Workflow Id — is always enforced within the active Namespace, and conflict resolution preserves it across a failover. +The guarantee limits how many Executions are _Open_ at the same time, not how many +[Run Ids](/workflow-execution/workflowid-runid#run-id) a Workflow Id accumulates over its lifetime. That distinction is +what lets conflict resolution reconcile a divergence without ever running the same Workflow Id twice concurrently. + +1. **Steady state.** The active region enforces uniqueness on every write and asynchronously replicates the Event + History to the replica. +2. **Failover with divergence.** In a forced failover when replication lag is present, both regions can independently + append events to the same Workflow Execution. When the regions reconnect, their Event Histories have diverged for + that Workflow Id. +3. **Fork instead of merge.** Temporal Cloud does not interleave the divergent histories. Events from the previously + active Namespace that arrive after the failover cannot be directly applied, so Temporal Cloud forks the Event History + and creates a new branch history, each branch identified by its own Run Id. Its + process keeps one branch as the Open Execution and supersedes the other, + leaving exactly one Open Workflow Execution per Workflow Id. The Temporal Service ensures the resulting Event + Histories remain valid and replayable by SDKs. diff --git a/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx b/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx index 23bf90699e..050366a4fc 100644 --- a/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx +++ b/docs/encyclopedia/workflow/workflow-execution/workflowid-runid.mdx @@ -27,6 +27,13 @@ In some cases, such as when running the same Workflow at recurring intervals usi In this case, all runs will have the same Workflow ID. However, each run will have a unique system-generated [Run ID](#run-id). +:::note Workflow Id uniqueness and High Availability + +This uniqueness constraint is enforced within the active [Namespace](/namespaces). +On a [High Availability](/cloud/high-availability) Namespace, replication is asynchronous, and a failover preserves the constraint through Temporal Cloud's [conflict resolution](/cloud/high-availability/failovers#conflict-resolution) process. + +::: + ## What is a Run Id? {/* #run-id */} A Run Id is a globally unique, platform-level identifier for a [Workflow Execution](/workflow-execution).