Clarifying workflow ID consistency and HA #4794
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -31,9 +31,10 @@ failover, Temporal automatically fails back to the original region once it is he | |
|
|
||
| ## Automatic failover | ||
|
|
||
| Temporal Cloud offers managed outage detection and failover to all Namespaces that use High Availability. These | ||
| automatic failovers keep your Namespace available without manual intervention. Temporal aims to both detect the outage | ||
| and complete a failover in minutes from when the outage began, according to the stated | ||
| [Recovery Time Objective (RTO)](/cloud/rpo-rto). | ||
|
|
||
| After an automatic failover, the Namespace will have a replica in its original region. Once the original region is | ||
| healthy again, Temporal Cloud automatically performs a [failback](/cloud/high-availability/failovers/manage#failbacks), | ||
|
|
@@ -64,8 +65,8 @@ regional outage. | |
|
|
||
| :::info | ||
|
|
||
| The following list gives a general idea of the conditions that trigger an automatic failover. This is not an exhaustive | ||
| list, and it may change over time. | ||
|
|
||
| ::: | ||
|
|
||
|
|
@@ -80,22 +81,23 @@ exhaustive list, and it may change over time. | |
| You can also [manually trigger a failover](/cloud/high-availability/failovers/manage#trigger-failover) based on your own | ||
| monitoring or for failover testing. | ||
|
|
||
| Most Namespaces with High Availability are well-served by automatic failovers. The cases where a manual failover (i.e., | ||
| a failover triggered by a user) is warranted are: | ||
|
|
||
| - **Testing failover or migrating to a new region.** A manual failover is the standard way to exercise your failover | ||
| process with your Clients and Workers, or to move a Namespace to a different region. | ||
| - **An outage that affects only your systems.** If an outage is contained to your application, Workers, or other | ||
| infrastructure, and Temporal Cloud is not affected, Temporal will not initiate a failover on your behalf. Detect the | ||
| outage with your own monitoring and trigger a failover yourself. | ||
| - **Failing over more aggressively during a regional outage.** Even with automatic failovers enabled, you can trigger a | ||
| failover yourself if you detect a regional outage before Temporal does. Whichever failover happens first takes effect, | ||
| and the later one is a no-op. A manual failover does not conflict with Temporal's automatic failover. | ||
|
|
||
| :::note Same-region Replication | ||
|
|
||
| Manual failovers apply only to Multi-region and Multi-cloud Replication. A | ||
| [Same-region Replication](/cloud/high-availability#same-region-replication) Namespace fails over automatically between | ||
| cells and cannot be failed over manually or have its automatic failovers disabled. | ||
|
|
||
| ::: | ||
|
|
||
|
|
@@ -127,9 +129,15 @@ The failover process is the same whether it is triggered automatically by Tempor | |
|
|
||
| ## Post-failover events {/* #post-failover-events */} | ||
|
|
||
| After any failover, whether triggered by you or by Temporal, an event appears in both the | ||
| [Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. The | ||
| audit log entry uses the `"operation": "FailoverNamespace"` event. Temporal Cloud | ||
| [notifies you via email](/cloud/notifications#admin-notifications) whenever a failover occurs. | ||
|
|
||
| After an automatic failover, Temporal automatically fails back to the original region once the region is healthy, unless | ||
| you [opt out](/cloud/high-availability/failovers/manage#after-an-automatic-failover). After a user-triggered failover, | ||
| the Namespace stays in the replica region until a user triggers another failover. See | ||
| [failback options](/cloud/high-availability/failovers/manage#failbacks) for details. | ||
|
|
||
| ## Split-brain scenario | ||
|
|
||
|
|
@@ -146,15 +154,44 @@ remains active. | |
|
|
||
| Namespaces with replicas rely on asynchronous event replication. Updates made to the primary may not immediately be | ||
| reflected in the replica due to <ToolTipTerm term="replication lag" />, particularly during failovers. In the event of a | ||
| non-graceful failover, replication lag causes temporary setback in Workflow progress. At the moment of non-graceful | ||
| failover: | ||
|
|
||
| - Operations that had already replicated remain durable in the replica. | ||
| - Operations that had not yet replicated (i.e., that are still in the replication backlog) are reconciled when the | ||
| region recovers. according to the Conflict Resolution process. - Note that Conflict Resolution can only recover data | ||
|
Contributor
There was a problem hiding this comment. This is a period |
||
| from a functioning Temporal Server. If the active server never recovers, the Workflow API calls that fall within the 1 | ||
| minute RPO may be permanently lost. Such a case would require the permanent loss of multiple cloud Availability Zones | ||
| and has never happened in the history of Temporal Cloud. | ||
|
|
||
| In a graceful failover, Temporal Cloud drains the replication backlog to zero and pauses traffic before switching | ||
| regions, so the replica holds every acknowledged operation and the Namespace achieves a recovery point of zero. | ||
|
|
||
| Namespaces that are not replicated can be configured to provide _at-most-once_ semantics for Activity execution when a | ||
| retry policy's [maximum attempts](https://docs.temporal.io/retry-policies#maximum-attempts) is set to 0. High | ||
| Availability Namespaces provide _at-least-once_ semantics for execution of Activities. Completed Activities _may_ be | ||
| re-dispatched in a newly active Namespace, leading to repeated executions. | ||
|
|
||
| The same durability boundary applies to Workflow starts and Signals: a `StartWorkflowExecution` or | ||
| `SignalWorkflowExecution` call that returns success is durably committed in the active region, and replicated | ||
|
Contributor
There was a problem hiding this comment. also SignalWithStartWorkflowExecution? |
||
| asynchronously to the replica. | ||
|
|
||
| ### How Workflow Id uniqueness is preserved after a forced failover | ||
|
Contributor
Author
There was a problem hiding this comment. @temporal-nick can you check for correctness? |
||
|
|
||
| The [Workflow Id uniqueness guarantee](/workflow-execution/workflowid-runid) — at most one Open Workflow Execution per | ||
| Workflow Id — is always enforced within the active Namespace, and conflict resolution preserves it across a failover. | ||
| The guarantee limits how many Executions are _Open_ at the same time, not how many | ||
|
Contributor
There was a problem hiding this comment. There are two policies on workflow ID:
I think here you try to attach the conflict policy. It is a little confusing as the ID reuse policy manages the workflow ID based on previous run. |
||
| [Run Ids](/workflow-execution/workflowid-runid#run-id) a Workflow Id accumulates over its lifetime. That distinction is | ||
| what lets conflict resolution reconcile a divergence without ever running the same Workflow Id twice concurrently. | ||
|
|
||
| 1. **Steady state.** The active region enforces uniqueness on every write and asynchronously replicates the Event | ||
| History to the replica. | ||
| 2. **Failover with divergence.** In a forced failover when replication lag is present, both regions can independently | ||
| append events to the same Workflow Execution. When the regions reconnect, their Event Histories have diverged for | ||
| that Workflow Id. | ||
| 3. **Fork instead of merge.** Temporal Cloud does not interleave the divergent histories. Events from the previously | ||
| active Namespace that arrive after the failover cannot be directly applied, so Temporal Cloud forks the Event History | ||
| and creates a new branch history, each branch identified by its own Run Id. Its | ||
| <ToolTipTerm term="conflict resolution" /> process keeps one branch as the Open Execution and supersedes the other, | ||
|
Contributor
There was a problem hiding this comment. One "execution", not one branch. The execution from the source-side will become a "zombie" workflow execution that is eventually terminated on the source once the target replicates information about a competing workflow ID |
||
| leaving exactly one Open Workflow Execution per Workflow Id. The Temporal Service ensures the resulting Event | ||
| Histories remain valid and replayable by SDKs. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@brianmacdonald-temporal you had some comments here. Could you review my rewrite?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd still suggest putting the text starting with "Note that" in a Caution box. I think we want to be clear on the circumstances in which data cannot be fully recovered. Other than that, the text itself looks good.