Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 99 additions & 0 deletions assets/temporal-cloud-serverless-worker-agentcore-role.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# CloudFormation template for creating an IAM role that Temporal Cloud can assume to invoke AgentCore runtimes.
AWSTemplateFormatVersion: "2010-09-09"
Description: Creates an IAM role that Temporal Cloud can assume to invoke Amazon Bedrock AgentCore runtimes for Serverless Workers.

Parameters:
AssumeRoleExternalId:
Type: String
Description: The External ID provided by Temporal Cloud
AllowedPattern: "[a-zA-Z0-9_+=,.@-]*"
MinLength: 5
MaxLength: 45

AgentRuntimeARNs:
Type: CommaDelimitedList
Description: >-
Comma-separated list of AgentCore runtime ARNs to invoke (e.g.,
arn:aws:bedrock-agentcore:us-west-2:123456789012:runtime/temporal_worker-XXXXXXXXXX).
Append a wildcard (.../runtime/temporal_worker-XXXXXXXXXX*) to cover the
runtime's endpoints as well, which GetAgentRuntimeEndpoint is scoped to.

RoleName:
Type: String
Default: "Temporal-Cloud-Serverless-Worker"

Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "Temporal Cloud Configuration"
Parameters:
- AssumeRoleExternalId
- Label:
default: "AgentCore Configuration"
Parameters:
- AgentRuntimeARNs
- RoleName
ParameterLabels:
AssumeRoleExternalId:
default: "External ID (provided by Temporal Cloud)"
AgentRuntimeARNs:
default: "AgentCore Runtime ARNs (comma-separated list)"
RoleName:
default: "IAM Role Name"

Resources:
TemporalCloudServerlessWorker:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub "${RoleName}-${AWS::StackName}"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
AWS:
[
arn:aws:iam::902542641901:role/wci-lambda-invoke,
arn:aws:iam::160190466495:role/wci-lambda-invoke,
arn:aws:iam::819232936619:role/wci-lambda-invoke,
arn:aws:iam::829909441867:role/wci-lambda-invoke,
arn:aws:iam::354116250941:role/wci-lambda-invoke,
]
Action: sts:AssumeRole
Condition:
StringEquals:
"sts:ExternalId": [!Ref AssumeRoleExternalId]
Description: "The role Temporal Cloud uses to invoke AgentCore runtimes for Serverless Workers"
MaxSessionDuration: 3600 # 1 hour

TemporalCloudAgentCoreInvokePermissions:
Type: AWS::IAM::Policy
DependsOn: TemporalCloudServerlessWorker
Properties:
PolicyName: "Temporal-Cloud-AgentCore-Invoke-Permissions"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- bedrock-agentcore:InvokeAgentRuntime
- bedrock-agentcore:GetAgentRuntimeEndpoint
Resource: !Ref AgentRuntimeARNs
Roles:
- !Sub "${RoleName}-${AWS::StackName}"

Outputs:
RoleARN:
Description: The ARN of the IAM role created for Temporal Cloud
Value: !GetAtt TemporalCloudServerlessWorker.Arn
Export:
Name: !Sub "${AWS::StackName}-RoleARN"

RoleName:
Description: The name of the IAM role
Value: !Ref RoleName

AgentRuntimeARNs:
Description: The AgentCore runtime ARNs that can be invoked
Value: !Join [", ", !Ref AgentRuntimeARNs]
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# CloudFormation template for creating an IAM role that a self-hosted Temporal Service can assume to invoke AgentCore runtimes.
AWSTemplateFormatVersion: '2010-09-09'
Description:
Creates an IAM role that a self-hosted Temporal Service can assume to invoke Amazon Bedrock AgentCore runtimes.

Parameters:
TemporalIamRoleArn:
Type: String
Description: The ARN of the IAM role or user that the Temporal Service runs as.

AssumeRoleExternalId:
Type: String
Description: A unique identifier to prevent confused deputy attacks.
AllowedPattern: '[a-zA-Z0-9_+=,.@-]*'
MinLength: 5
MaxLength: 45

AgentRuntimeARNs:
Type: CommaDelimitedList
Description: >-
Comma-separated list of AgentCore Runtime ARNs that Temporal may access. Append a wildcard to each Runtime ARN
to include its endpoints.

RoleName:
Type: String
Default: 'Temporal-AgentCore-Worker'

Resources:
TemporalAgentCoreWorker:
Type: AWS::IAM::Role
Properties:
RoleName: !Ref RoleName
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
AWS: [!Ref TemporalIamRoleArn]
Action: sts:AssumeRole
Condition:
StringEquals:
'sts:ExternalId': [!Ref AssumeRoleExternalId]
Description: The role the Temporal Service uses to invoke AgentCore runtimes for Serverless Workers
MaxSessionDuration: 3600

TemporalAgentCoreInvokePermissions:
Type: AWS::IAM::Policy
Properties:
PolicyName: 'Temporal-AgentCore-Invoke-Permissions'
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- bedrock-agentcore:InvokeAgentRuntime
- bedrock-agentcore:GetAgentRuntimeEndpoint
Resource: !Ref AgentRuntimeARNs
Roles:
- !Ref TemporalAgentCoreWorker

Outputs:
RoleARN:
Description: The ARN of the IAM role created for the Temporal Service
Value: !GetAtt TemporalAgentCoreWorker.Arn

AgentRuntimeARNs:
Description: The AgentCore Runtime ARNs that Temporal may access
Value: !Join [', ', !Ref AgentRuntimeARNs]
59 changes: 59 additions & 0 deletions references/aws-agentcore/constraints.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# AgentCore — Execution-model constraints

These are the consequences of AgentCore Runtime's execution model. **The WCI invokes a named AgentCore Runtime endpoint when it needs capacity. The Runtime starts a standard long-running Worker inside a Runtime session, and that Worker polls until it drains or AgentCore ends its compute.**

## Pre-release status

AgentCore Runtime support is in **Pre-release**, and its APIs may change in backwards-incompatible ways. A Temporal Cloud Namespace needs Pre-release access, which the user requests through a [support ticket](https://docs.temporal.io/evaluate/cloud/support#support-ticket) or their account team. A self-hosted Temporal Service v1.32.0 or later does not need an access request. See `self-hosted.md`.

The Cloud Namespace must be **hosted on AWS**.

## Worker code is a standard Worker inside a Runtime handler

AgentCore Workers are standard long-lived Temporal Workers. There is no serverless Worker package. The Runtime entry point must implement the AgentCore Runtime HTTP contract by serving `/invocations` and `/ping`. For a Serverless Worker, `/invocations` starts the Worker as background work and acknowledges the request without waiting for the Worker to stop. The per-language entry point is in the selected `sdk-<language>.md` reference.

## Compute lifetime

- **Up to 8 hours per compute.** With the serverless microVM compute type, a Runtime session can run for up to 8 hours. The `maxLifetime` setting accepts 60–28800 seconds for microVM runtimes and defaults to 28800 seconds. Its timer starts when the microVM is created and cannot be reset.
- **No fixed invocation deadline.** An AgentCore Worker has no invocation deadline to monitor. It polls until it drains or AgentCore ends its compute.
- **The session is not durable storage.** AgentCore can resume a session on new compute after the previous compute ends, and a later Task can run on another Worker. Keep the state a Workflow needs in the Workflow or another durable store.
- **Process-local reuse is an optimization only.** The Worker can reuse initialization work, in-memory caches and temporary files while its Runtime compute remains available. Do not make Workflow progress depend on them.
- **Workflow duration is unbounded.** A Workflow continues across as many Worker processes as needed. Durable Timers and waits for approvals or external events do not keep compute running.

## Two independent stop controls

Two sets of controls determine when the Worker stops. Both must be configured deliberately.

| Control | Owner | What it measures |
|---|---|---|
| Worker idle and graceful-shutdown policy | The Runtime handler you write | When the Worker has been idle. It then stops polling and waits for in-flight Activities. |
| `idleRuntimeSessionTimeout` | AgentCore lifecycle settings | Time since the last **AgentCore Runtime invocation** to the session. Default 900 seconds. Polling Temporal does **not** reset it. |
| `maxLifetime` | AgentCore lifecycle settings | Time since the compute was created. Default and microVM maximum: 28800 seconds (8 hours). |

The idle session timeout is not a Worker idle timer, and it does not replace one. Implement a Worker shutdown policy in the handler: when the idle condition is met, stop polling and drain in-flight Activities before the handler releases the session. Choose the idle period and drain timeout for the workload, and keep both inside `maxLifetime`.

`idleRuntimeSessionTimeout` must be less than or equal to `maxLifetime`, and both must be at least 60 seconds. A request that breaks these rules fails with `ValidationException`. When AgentCore ends compute, termination can take up to 15 seconds.

In the Python sample, the Worker drains after `AGENTCORE_DEBOUNCE_SECONDS` without an Activity starting or finishing. The default is `60`. The sample's drain timeout is 120 seconds, and its `agentcore.json` keeps the AgentCore defaults (`idleRuntimeSessionTimeout: 900`, `maxLifetime: 28800`). These are configured values, not platform limits.

## What bounds an Activity: AgentCore ending the compute

AgentCore can end the compute before an Activity completes. Configure Activity timeouts. For long-running Activities, also configure [Activity Heartbeats](https://docs.temporal.io/encyclopedia/detecting-activity-failures#activity-heartbeat) so a retry can recover the work. A Worker crash follows standard retry semantics: the Activity Timeout fires and Temporal retries the Activity on another Worker.

## Autoscaling

Autoscaling is event-driven (`no-sync`). The WCI invokes individual Runtime sessions when it needs more capacity. It does not manage a target-sized pool. For the shared WCI lifecycle and inspection commands, see [Worker Controller Instance (WCI)](../wci.md).

When the provider's capacity limit is reached, the WCI cannot add Workers. Tasks wait in the backlog without data loss.

## One named endpoint per Worker Deployment Version

The compute configuration names a Runtime **endpoint** ARN, not a Runtime ARN. Each Worker Deployment Version needs its own named endpoint pinned to one immutable Runtime version. Never configure the `DEFAULT` endpoint, because it moves to the latest Runtime version on every update. → `versioning.md`.

## Region and high availability

The AWS account must be in an [AgentCore-supported Region](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agentcore-regions.html). Compute provider configuration is scoped to one region. On failover of a replicated Namespace, the WCI keeps invoking Workers in the original region until the compute provider is repointed manually.

## AgentCore services are not configured by the compute provider

Configuring AgentCore Runtime as a compute provider does not set up AgentCore Identity, Gateway, Policy, Memory or Observability. Call those services from Activities, and keep credentials out of Workflow state. Writing the agent itself (Workflow and Activity boundaries, Strands, model calls) is out of scope for this skill. Link to [Build a durable agent on Amazon Bedrock AgentCore](https://docs.temporal.io/guides/durable-agent-on-agentcore) instead.
72 changes: 72 additions & 0 deletions references/aws-agentcore/diagnostics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# AgentCore — Diagnostics and troubleshooting

Temporal has no AgentCore troubleshooting page. This guide uses only the gotchas documented in the sources above and the shared [Worker Controller Instance (WCI)](../wci.md) inspection flow. If a symptom is not covered here, gather the evidence below and report it to the user. Do not attach an undocumented explanation to it.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you remove the explanation of how this guide was authored (“uses only the gotchas documented in the sources above”) and state the durable operating rule directly? For example: “Use the WCI inspection flow below. For symptoms not covered here, gather the listed evidence and escalate without speculating about the cause.”


## Invocation flow (when working correctly)

1. Creating the Worker Deployment Version starts its WCI, which invokes the named endpoint and waits for the Worker to register.
2. The Runtime's `/invocations` handler starts the Worker in the background and acknowledges.
3. The Worker polls the Task Queue as `<deployment-name>/<build-id>`, and the version binds the Task Queue.
4. When Tasks arrive and no Worker is polling, the WCI invokes the endpoint again.
5. The Worker drains when its idle policy fires, or stops when AgentCore ends the compute.

## Start here: did the expected Worker register?

```bash
temporal worker deployment describe-version \
--namespace <NAMESPACE> --deployment-name <DEPLOYMENT_NAME> --build-id <BUILD_ID> \
--report-task-queue-stats -o json
```

The decisive signal is that the expected Task Queue appears in `taskQueuesInfos` with the types the Worker polls. That only happens after a Worker that identifies as this deployment name and Build ID has polled. A successful Validate Connection or a Running WCI supports the diagnosis but does not replace this check. Wait for this state to appear rather than for a fixed time. Re-run the command while you wait. If the Task Queue is still not bound after several minutes, stop waiting and use the table below. A bound Task Queue does not show that the version is current or that Tasks complete.

| Result | Next check |
|---|---|
| Task Queue bound | Registration succeeded. Confirm the version is current, then go to [Tasks not completing](#worker-runs-but-tasks-do-not-complete). |
| Not bound, and the WCI history shows invocation failures | [The Runtime is not invoked](#the-runtime-is-not-invoked) |
| Not bound, invocation succeeded | [Invoked, but no Worker registers](#invoked-but-no-worker-registers) |

## The Runtime is not invoked

1. **Validate Connection (Cloud).** In the version's **Actions → Validate Connection**, a failure means Temporal cannot assume the invocation role, get the named endpoint, or invoke the Runtime. Check the following:
- `--aws-agentcore-endpoint-arn` is the **endpoint** ARN (`…/runtime-endpoint/<name>`), not the Runtime ARN.
- The role's `AgentRuntimeARNs` is the Runtime ARN **with a trailing `*`**. Without the wildcard, `GetAgentRuntimeEndpoint` on the endpoint is not covered.
- The External ID is identical in the stack and in the version.
- Self-hosted: the server's AWS identity is the stack's `TemporalIamRoleArn` and has `sts:AssumeRole` on the role. The server can also reach the AgentCore control-plane and data-plane APIs. → `self-hosted.md`.
2. **Stack never finished.** If the role stack failed, check the role-name length. On Cloud, `<ROLE_NAME>-<STACK_NAME>` must be 64 characters or fewer. → `iam.md`.
3. **Endpoint not ready.** Check the endpoint's state with `agentcore status --type runtime-endpoint --json`. `CREATE_FAILED` and `UPDATE_FAILED` mean a failure caused by permissions, the artifact, or another configuration problem. The sources document no cause for an endpoint that stays `CREATING` or `UPDATING`. If it is still in one of those states after several minutes, report the state and the `agentcore deploy` output to the user. Do not redeploy to get it to move.
4. **Version not current.** If registration succeeded but new Workflows never reach the Worker, set the version current (`setup.md` Step 6).
5. **Read the WCI history** for the failing Activity result. Use the inspection commands and interpretation rules in [`../wci.md`](../wci.md#inspect-the-wci).

## Invoked, but no Worker registers

Read the Runtime logs first:

```bash
aws logs tail /aws/bedrock-agentcore/runtimes/<RUNTIME_ID>-<ENDPOINT_NAME> --since 1h --region <AWS_REGION>
```

- **Name mismatch.** The Runtime's `TEMPORAL_DEPLOYMENT_NAME` / `TEMPORAL_BUILD_ID` differ from the version's. Temporal starts compute that registers as some other version. Fix the environment, which creates a new Runtime version. Then follow `versioning.md`.
- **Missing versioning settings.** The sample Worker refuses to start without `TEMPORAL_DEPLOYMENT_NAME` and `TEMPORAL_BUILD_ID`.
- **Cannot reach Temporal.** With a VPC network mode, the VPC needs outbound access to the Temporal Service. For self-hosted, see `self-hosted.md` step 1.
- **Endpoint runs other code.** The version points at `DEFAULT`, or at a named endpoint that was moved to another Runtime version. → `versioning.md`.
- **Deploy failed with `uv install failed ... with exit code null`.** `uv` is missing on the operator's machine.

## Worker runs but Tasks do not complete

- **Non-determinism errors after a deploy.** Code changed underneath a live Worker Deployment Version, because the version uses `DEFAULT` or a repointed endpoint. Pinned Workflows are affected too. Restore the old endpoint-to-version mapping, as described in `versioning.md`.
- **Old code still running after a deploy.** Existing sessions keep the code that was deployed when their microVM was created, until they terminate.
- **Activities interrupted.** AgentCore ended the compute (`maxLifetime`) before the Activity completed, or the idle policy drained too early. Configure Activity timeouts and Heartbeats so retries recover the work. → `constraints.md`.

## Sessions stay alive after work is done

In the sample, `/ping` reports `HealthyBusy` while an async task is registered, and AgentCore keeps the session alive while it does. If the handler never calls `complete_async_task` when the Worker exits, the session stays `HealthyBusy` until `maxLifetime`. The idle session timeout does not help either, because it resets only on AgentCore Runtime invocations, not on Temporal polling. The fix is a Worker idle policy that drains the Worker and releases the async task in all cases, including errors. The language-specific code is in the selected `sdk-<language>.md`.

## Lifecycle configuration rejected

`ValidationException` on a Runtime create or update means one of these: a lifecycle value is below 60 seconds or above 28800 seconds (microVMs), or `idleRuntimeSessionTimeout` is greater than `maxLifetime`.

## Rules

- Inspect the specific Runtime, endpoint, stack and version named in the inventory. Do not enumerate AgentCore resources across Regions to reconstruct state.
- Do not change IAM, redeploy, or repoint an endpoint until the evidence above identifies the cause.
Loading
Loading