diff --git a/docs/acts-specification.md b/docs/acts-specification.md new file mode 100644 index 000000000..2269e3ff3 --- /dev/null +++ b/docs/acts-specification.md @@ -0,0 +1,2149 @@ +# A2A Conformance Test Specification (ACTS) + +**Version:** 0.1.0 (Draft) +**Status:** Proposal +**Target A2A Version:** 1.0 + +## 1. Introduction + +### 1.1. Purpose + +The A2A Conformance Test Specification (ACTS) defines a language-neutral, declarative format for specifying conformance tests for implementations of the [Agent2Agent (A2A) Protocol](https://a2a-protocol.org). An ACTS file describes **what** to test and **what to expect**, not **how** to execute the test. Any programming language or framework can implement a test runner that consumes ACTS files. + +The goals of this format are: + +- **Unify** fragmented conformance testing efforts across the A2A ecosystem. +- **Decouple** test definitions from test infrastructure so that SDK teams can write runners in their own language while testing against one canonical set of tests. +- **Ensure interoperability** by verifying that every conforming SDK produces and consumes wire-level messages that any other conforming SDK can understand. + +### 1.2. Scope + +ACTS covers **conformance testing** of a single A2A implementation — verifying that a server (or client) correctly implements the protocol as defined in the A2A specification. + +ACTS does **not** cover: + +- **Interoperability testing** (connecting SDK A's client to SDK B's server). However, ACTS is designed so that if two implementations each pass the full conformance suite, they will interoperate. +- **Performance or load testing.** +- **Application-level behavior** beyond what the protocol requires. + +### 1.3. Terminology + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119). + +- **SUT** — System Under Test. The A2A server implementation being tested. +- **Runner** — A program that reads ACTS files, executes the described tests against a SUT, and reports results. +- **Test** — A single named scenario that verifies one or more protocol requirements. +- **Step** — An individual action within a test (e.g., send a message, check the response). +- **Suite** — A named grouping of related tests. + +### 1.4. File Format + +ACTS files are expressed in [YAML 1.2](https://yaml.org/spec/1.2.2/). Runners SHOULD also accept equivalent JSON input. All examples in this specification use YAML. + +### 1.5. Notational Conventions + +Data structures in this specification are defined using [CDDL (Concise Data Definition Language)](https://www.rfc-editor.org/rfc/rfc8610) as extended by [RFC 9165](https://www.rfc-editor.org/rfc/rfc9165). CDDL rules describe the logical structure; the serialization is YAML (or JSON). + +In CDDL definitions: + +- `text` corresponds to a YAML string. +- `int` corresponds to a YAML integer. +- `bool` corresponds to a YAML boolean. +- `any` corresponds to any YAML value. +- `? key` denotes an optional field. +- `* key` denotes zero or more entries. +- `+ key` denotes one or more entries. + +--- + +## 2. Document Structure + +An ACTS document is a YAML file whose root is a map conforming to the `acts-document` rule. + +```cddl +acts-document = { + acts_version: text, ; Version of the ACTS format (e.g., "1.0") + spec_version: text, ; A2A spec version these tests target (e.g., "1.0") + ? spec_ref: text, ; URL to the A2A specification + ? metadata: metadata, + ? variables: { * text => text }, ; Default variables available to all tests + ? include: [+ text], ; List of ACTS files to include (for master files) + ? suites: [+ suite] ; At least one of include or suites MUST be present +} + +metadata = { + ? title: text, + ? description: text, + ? authors: [+ text], + ? license: text +} +``` + +An ACTS document MUST contain at least one of `suites` or `include`. A master file MAY use only `include` to reference other ACTS files without defining its own suites. + +### Example + +```yaml +acts_version: "1.0" +spec_version: "1.0" +spec_ref: "https://github.com/a2aproject/A2A/blob/main/docs/specification.md" + +metadata: + title: "A2A v1.0 Official Conformance Tests" + description: "Canonical test suite for A2A protocol v1.0 implementations" + +variables: + baseUrl: "{{env.A2A_BASE_URL}}" + +suites: + - id: discovery + name: "Agent Card Discovery" + tests: [...] +``` + +--- + +## 3. Suites and Tests + +### 3.1. Suite + +A suite is a named group of related tests. + +```cddl +suite = { + id: text, ; Unique identifier (kebab-case) + name: text, ; Human-readable name + ? description: text, + ? tags: [+ text], ; Tags inherited by all tests in the suite + tests: [+ test] +} +``` + +### 3.2. Test + +A test is a single conformance scenario. It declares which protocol requirements it verifies, its conformance level, and an ordered list of steps. + +```cddl +test = { + id: text, ; Unique requirement ID (e.g., "CORE-SEND-001") + name: text, ; Human-readable name + ? description: text, ; What this test verifies and why + ? spec_ref: text, ; Section reference in the A2A spec + level: conformance-level, + ? tags: [+ text], + ? transport: [+ transport-binding], ; Omit to apply to all bindings + ? preconditions: preconditions, + ? requires_behaviors: [+ text], ; SUT behavior prefixes this test depends on (see §11) + ? origin: text, ; URL to issue/PR where this test originated + ? runner_requirements: [+ runner-requirement], + steps: [+ step], + ? assertions: [+ named-assertion] +} + +conformance-level = "must" / "should" / "may" + +transport-binding = "jsonrpc" / "grpc" / "rest" + +runner-requirement = + "webhook_endpoint" / ; Runner must set up a webhook to receive push notifications + "concurrent_streams" / ; Runner must open multiple simultaneous streams + "stream_disconnect" / ; Runner must disconnect and reconnect mid-stream + "auth_credentials" / ; Runner must provide valid/invalid auth credentials + "header_inspection" ; Runner must inspect HTTP response headers +``` + +**Conformance levels** map to RFC 2119 keywords: + +| Level | Meaning | +|-------|---------| +| `must` | Absolute requirement. Failure means the implementation is non-conformant. | +| `should` | Recommended behavior. Failure means the implementation is conformant but may have reduced interoperability. | +| `may` | Optional behavior. Passing provides additional interoperability. | + +Tests MAY declare `runner_requirements` when execution depends on runner-managed capabilities that are not visible from the SUT alone, such as webhook provisioning, concurrent streams, reconnection behavior, injected credentials, or HTTP header inspection. Runners that cannot satisfy a declared requirement SHOULD skip the test with a clear reason. + +Tests MAY also define a top-level `assertions` array for validations that run after all `steps` complete. This is useful for cross-step checks without adding a separate trailing `assertion-step`. + +### 3.3. Preconditions + +Preconditions describe what the SUT must advertise or support for the test to be applicable. If preconditions are not met, the runner MUST mark the test as **skipped**, not failed. + +```cddl +preconditions = { + ? capabilities: { * text => any }, ; Required AgentCard capability fields + ? skills: [+ { id: text }], ; Required skill IDs in the AgentCard + ? transport: [+ transport-binding], ; Required transport binding + ? extensions: [+ text], ; Required extension URIs + ? description: text ; Human-readable precondition summary +} +``` + +### Example + +```yaml +- id: STREAM-ORDER-001 + name: "Streaming events never regress state" + description: > + When an agent streams status updates, the task state MUST + follow the state machine and never move backward + (e.g., COMPLETED must not be followed by WORKING). + spec_ref: "specification.md#section-3.5.2" + level: must + tags: [streaming, ordering] + preconditions: + capabilities: + streaming: true + steps: [...] +``` + +--- + +## 4. Steps + +Steps are the building blocks of a test. They execute sequentially within a test. Each step performs one action and optionally asserts expectations on the result. + +**Test isolation:** Tests within a suite (or across suites) MUST NOT share state. Each test starts with a clean context — no tasks, captures, or side effects carry over from previous tests. Runners MAY execute tests in any order or in parallel. + +```cddl +step = server-step / client-step / assertion-step / raw-step + +server-step = { + id: text, ; Unique within the test + ? description: text, + operation: abstract-operation, ; Abstract A2A operation name + params: request-params, ; Parameters for the operation + ? expect: expect-block, ; Expected response assertions + ? expect_error: expect-error, ; Expected error (mutually exclusive with expect) + ? expect_stream: expect-stream, ; Streaming assertions (for streaming operations) + ? capture: { + text => text }, ; Variable capture: varName => response.path + ? assertions: [+ named-assertion], ; Post-step assertions on the step result + ? repeat: repeat-config, ; Polling/retry configuration + ? delay_ms: int ; Delay before executing this step +} + +client-step = { + id: text, + ? description: text, + client_response: client-response, ; Canonical wire payload for the client to parse + expect_parsed: { * text => assertion }, ; Assertions on the parsed result + ? assertions: [+ named-assertion] +} + +assertion-step = { + id: text, + ? description: text, + assertion: inline-assertion ; Assert against a prior step's response +} + +client-response = { + operation: abstract-operation, ; Abstract operation represented by the payload + wire_payload: any ; Canonical transport payload for the client to parse +} +``` + +### 4.1. Abstract Operations + +Tests reference abstract A2A operations rather than wire-level method names. The runner maps each abstract operation to the appropriate wire format for the transport binding under test. + +```cddl +abstract-operation = + "send_message" / + "send_streaming_message" / + "get_task" / + "list_tasks" / + "cancel_task" / + "subscribe_to_task" / + "get_agent_card" / + "get_extended_agent_card" / + "create_push_config" / + "get_push_config" / + "list_push_configs" / + "delete_push_config" +``` + +The runner MUST translate each abstract operation to the correct wire representation: + +| Abstract Operation | JSON-RPC Method | gRPC RPC | REST | +|---|---|---|---| +| `send_message` | `SendMessage` | `SendMessage` | `POST /message:send` | +| `send_streaming_message` | `SendStreamingMessage` | `SendStreamingMessage` | `POST /message:stream` | +| `get_task` | `GetTask` | `GetTask` | `GET /tasks/{id}` | +| `list_tasks` | `ListTasks` | `ListTasks` | `GET /tasks` | +| `cancel_task` | `CancelTask` | `CancelTask` | `POST /tasks/{id}:cancel` | +| `subscribe_to_task` | `SubscribeToTask` | `SubscribeToTask` | `GET /tasks/{id}:subscribe` | +| `get_agent_card` | *(HTTP GET)* | *(HTTP GET)* | `GET /.well-known/agent-card.json` | +| `create_push_config` | `CreatePushNotificationConfig` | `CreatePushNotificationConfig` | `POST /tasks/{id}/pushNotifications` | +| `get_push_config` | `GetPushNotificationConfig` | `GetPushNotificationConfig` | `GET /tasks/{id}/pushNotifications/{configId}` | +| `list_push_configs` | `ListPushNotificationConfigs` | `ListPushNotificationConfigs` | `GET /tasks/{id}/pushNotifications` | +| `delete_push_config` | `DeletePushNotificationConfig` | `DeletePushNotificationConfig` | `DELETE /tasks/{id}/pushNotifications/{configId}` | + +### 4.2. Assertion Root + +Different A2A operations return different response shapes. The **assertion root** — the object against which `expect`, `capture`, and `repeat.until` paths are resolved — depends on the operation: + +| Operation | Response Type | Assertion Root | +|---|---|---| +| `send_message` | `SendMessageResponse` | The `SendMessageResponse` object (contains `task` or `message`) | +| `send_streaming_message` | Stream of `StreamResponse` | Per-event (see §7) | +| `get_task` | `Task` | The `Task` object directly | +| `list_tasks` | `ListTasksResponse` | The `ListTasksResponse` object (contains `tasks` array) | +| `cancel_task` | `Task` | The `Task` object directly | +| `subscribe_to_task` | Stream of `StreamResponse` | Per-event (see §7) | +| `get_agent_card` | `AgentCard` | The `AgentCard` object directly | +| `get_extended_agent_card` | `AgentCard` | The `AgentCard` object directly | +| `create_push_config` | `PushNotificationConfig` | The `PushNotificationConfig` object directly | +| `get_push_config` | `PushNotificationConfig` | The `PushNotificationConfig` object directly | +| `list_push_configs` | `ListPushNotificationConfigsResponse` | The response object (contains `configs` array) | +| `delete_push_config` | *(empty)* | N/A | + +For operations that return a wrapper (like `SendMessageResponse`), assertions reference fields within the wrapper. For operations that return a domain object directly (like `get_task` → `Task`), assertions reference fields on that object. + +**Example — the difference matters:** + +```yaml +# send_message returns SendMessageResponse → root includes task/message discriminator +- id: send + operation: send_message + expect: + status: 200 + body: + task: # SendMessageResponse.task + id: {type: string} + status: + state: TASK_STATE_COMPLETED + capture: + taskId: task.id # path from SendMessageResponse root + +# get_task returns Task directly → root IS the Task +- id: get + operation: get_task + params: + id: "{{send.taskId}}" + expect: + status: 200 + body: + id: "{{send.taskId}}" # Task.id — no "task." prefix needed + status: + state: TASK_STATE_COMPLETED +``` + +The runner MUST unwrap transport-level framing (JSON-RPC envelope, gRPC message wrapper, HTTP response) before applying assertions. The assertion root is always the **A2A domain object**, not the transport wrapper. + +### 4.3. Operation Parameters + +Step `params` use the abstract A2A data model — not the wire format. The runner handles serialization. + +If the `params` object includes a `message` and no `messageId` is specified, the runner MUST auto-generate a unique `messageId` (e.g., a UUID) for the message. The A2A specification requires every message to have a `messageId`. + +```cddl +request-params = { + ; For send_message / send_streaming_message: + ? message: { + role: text, ; e.g., "ROLE_USER" + parts: [+ part], + ? messageId: text, + ? taskId: text, + ? contextId: text, + * text => any ; Additional fields + }, + ? configuration: { + ? returnImmediately: bool, + ? historyLength: int, + * text => any + }, + + ; For get_task / cancel_task / subscribe_to_task: + ? id: text, ; Task ID + ? historyLength: int, + + ; For list_tasks: + ? contextId: text, + ? cursor: text, + ? pageSize: int, + + ; For push config operations: + ? taskId: text, + ? pushNotificationConfig: any, + + ; Catch-all for future fields: + * text => any +} + +part = { + ? text: text, + ? data: any, + ? file: { + ? url: text, + ? raw: text, ; base64-encoded bytes + ? mediaType: text, + ? name: text + }, + * text => any +} +``` + +### 4.4. Raw Wire Steps + +For transport-specific tests that must assert exact wire-level behavior, steps MAY use `raw` and `expect` instead of `operation`/`params`/`expect`: + +```cddl +raw-step = { + id: text, + ? description: text, + raw: { + method: "GET" / "POST" / "PUT" / "DELETE", + path: text, + ? headers: { * text => text }, + ? body: any, + ? body_raw: text ; Raw string body (for malformed JSON tests) + }, + ? expect: expect-block, ; Reuses the standard expect-block + ? expect_error: expect-error, ; Can also expect errors from raw steps + ? capture: { + text => text }, + ? assertions: [+ named-assertion] +} +``` + +Raw steps are used when the test must verify wire-level details that the abstract operation layer intentionally hides (e.g., HTTP status codes, header values, malformed input handling). + +> **Note:** Raw steps are inherently transport-specific. A test containing only raw steps MUST specify a `transport` filter. + +--- + +## 5. Assertions + +Assertions are the core of the format. They describe expected values using a compact DSL that can express exact matches, type checks, existence checks, and more. + +### 5.1. Assertion Grammar + +```cddl +assertion = + exact-match / ; Bare value: exact equality + assertion-object ; Map with assertion operators + +exact-match = text / int / float / bool / null + +assertion-object = { + ; Type checking + ? type: "string" / "number" / "boolean" / "array" / "object" / "null", + + ; Existence + ? exists: true, ; Field is present (any value) + ? absent: true, ; Field is NOT present + + ; String matching + ? contains: text, ; Substring match + ? matches: text, ; Regular expression match (ECMA-262) + ? starts_with: text, + ? ends_with: text, + + ; Numeric comparison + ? gte: number, ; Greater than or equal + ? lte: number, ; Less than or equal + ? gt: number, ; Greater than + ? lt: number, ; Less than + + ; Array length + ? count: int, ; Exact array length + ? count_gte: int, ; Length >= N + ? count_lte: int, ; Length <= N + + ; Enum / alternatives + ? one_of: [+ any], ; Value is one of these + + ; Combinators + ? all_of: [+ assertion], ; All assertions must pass + ? any_of: [+ assertion], ; At least one assertion must pass + ? not: assertion ; Assertion must NOT pass +} + +number = int / float +``` + +### 5.2. Assertion Paths and Nesting + +Assertions in `expect` blocks use **YAML nesting** to mirror the structure of the response object. Each level of nesting corresponds to a level of object nesting in the response. + +```yaml +expect: + task: + status: + state: TASK_STATE_COMPLETED # Asserts response.task.status.state + artifacts: + - parts: + - text: {type: string} # Asserts response.task.artifacts[0].parts[0].text +``` + +**Dot-path notation** (e.g., `artifacts[0].parts[0].text`) MUST NOT be used as YAML keys in `expect` blocks. YAML would parse `artifacts[0].parts[0].text` as a literal string key, not a path expression. Runners MUST interpret `expect` blocks as nested YAML maps that mirror the response structure. + +The one exception is **`capture` paths** and **`repeat.until` expressions**, which use dot-path strings as *values* (not keys): + +```yaml +capture: + taskId: task.id # dot-path as a VALUE — this is correct +``` + +For **array access** in assertions, use YAML array syntax: + +```yaml +expect: + body: + artifacts: + - parts: # First artifact's parts + - text: {type: string} # First part is a text part +``` + +For assertions that must apply to **any element** in an array (rather than a specific index), use `assertion-step` or an `assertions` block with collection assertions (see §5.5). + +When a path appears in a `capture` value, `repeat.until` expression, or `collection-match.path`, it uses dot-path notation with bracket indexing: + +``` +task.id ; Nested field +task.artifacts[0].parts[0].text ; Array indexing +task.artifacts[*].parts[*] ; Wildcard (collection assertions only) +``` + +### 5.3. Exact Match + +A bare scalar value asserts exact equality: + +```yaml +expect: + body: + task: + status: + state: TASK_STATE_COMPLETED # exact string match +``` + +### 5.4. Operator Assertions + +An assertion map applies one or more operators: + +```yaml +expect: + body: + task: + id: {type: string} # type check + status: + state: {one_of: [TASK_STATE_SUBMITTED, TASK_STATE_WORKING]} + artifacts: {type: array, count_gte: 1} # combined operators +``` + +When multiple operators appear in one assertion map, they are combined with AND semantics — all operators must pass. + +### 5.5. Collection Assertions + +For asserting properties across array elements: + +```cddl +inline-assertion = { + source: text, ; Reference to a prior step's response + ? any: collection-match, ; At least one element matches + ? all: collection-match, ; Every element matches + ? none: collection-match ; No element matches +} + +named-assertion = { + ? id: text, + ? description: text, + source: text, ; Dot-path to a captured variable or prior step result + ? path: text, ; Optional sub-path within the source + ? match: assertion, ; Direct assertion applied at source/path + ? any: collection-match, ; At least one element matches + ? all: collection-match, ; Every element matches + ? none: collection-match ; No element matches +} + +collection-match = { + path: text, ; Dot-path with wildcards (e.g., "task.artifacts[*].parts[*]") + match: { * text => assertion } ; Assertions each matching element must satisfy +} +``` + +A step MAY include an `assertions` array of `named-assertion` values that runs immediately after that step completes. Tests MAY also include a top-level `assertions` array evaluated after all steps complete. These forms are useful for cross-step validation and collection checks without introducing a separate trailing `assertion-step`. + +### Example + +```yaml +- id: verify-text-artifact + assertion: + source: "{{send.response}}" + any: + path: task.artifacts[*].parts[*] + match: + text: {type: string} +``` + +--- + +## 6. Expect Blocks + +### 6.1. Standard Expect + +For non-streaming operations, `expect` describes the expected response. + +```cddl +expect-block = { + ? status: assertion, ; Expected HTTP status (default: 200 for success) + ? body: { * text => assertion } ; Response body assertions +} +``` + +For abstract operations, assertions in `expect.body` apply to the unwrapped A2A response object described in §4.2. For raw steps, `expect.body` applies to the raw transport response body. + +### 6.2. Error Expect + +For operations expected to fail, `expect_error` describes the expected error. + +```cddl +expect-error = { + error_type: a2a-error-type, ; Abstract A2A error name + ? message: assertion, ; Assertion on the error message string + ? details: { * text => assertion } ; Assertions on error details/metadata +} + +a2a-error-type = + "TaskNotFoundError" / + "TaskNotCancelableError" / + "UnsupportedOperationError" / + "ContentTypeNotSupportedError" / + "InvalidParamsError" / + "VersionNotSupportedError" / + "PushNotificationNotSupportedError" / + "StreamingNotSupportedError" / + "ExtensionSupportRequiredError" / + "ExtendedCardNotSupportedError" / + "JSONParseError" / + "MethodNotFoundError" / + "InternalError" +``` + +The runner MUST map each abstract error type to the transport-specific representation: + +| Abstract Error | JSON-RPC Code | gRPC Status | REST HTTP Status | +|---|---|---|---| +| `TaskNotFoundError` | -32001 | `NOT_FOUND` | 404 | +| `TaskNotCancelableError` | -32002 | `FAILED_PRECONDITION` | 409 | +| `UnsupportedOperationError` | -32004 | `UNIMPLEMENTED` | 405 | +| `ContentTypeNotSupportedError` | -32005 | `INVALID_ARGUMENT` | 415 | +| `InvalidParamsError` | -32602 | `INVALID_ARGUMENT` | 400 | +| `VersionNotSupportedError` | -32006 | `UNIMPLEMENTED` | 406 | +| `PushNotificationNotSupportedError` | -32003 | `UNIMPLEMENTED` | 501 | +| `StreamingNotSupportedError` | -32007 | `UNIMPLEMENTED` | 501 | +| `JSONParseError` | -32700 | `INVALID_ARGUMENT` | 400 | +| `MethodNotFoundError` | -32601 | `UNIMPLEMENTED` | 501 | +| `InternalError` | -32603 | `INTERNAL` | 500 | + +> **Note:** This mapping table is derived from the A2A specification. Runners MUST consult the normative A2A specification for authoritative error code mappings. If this table conflicts with the A2A specification, the A2A specification takes precedence. + +### Example + +```yaml +- id: CORE-ERR-001 + name: "TaskNotFoundError on missing task" + level: must + steps: + - id: get-missing + operation: get_task + params: + id: "00000000-0000-0000-0000-000000000000" + expect_error: + error_type: TaskNotFoundError + message: {type: string} +``` + +--- + +## 7. Streaming Assertions + +Streaming operations (`send_streaming_message`, `subscribe_to_task`) return an ordered sequence of events rather than a single response. The `expect_stream` block provides assertions for this. + +```cddl +expect-stream = { + ? min_count: int, ; Minimum number of events + ? max_count: int, ; Maximum number of events + ? timeout_ms: int, ; Maximum time to wait for stream completion + ? ordering: ordering-rule, ; Ordering constraint on task states + + ; Event-level assertions + ? events: [+ event-assertion], + + ; Final event assertion + ? final_event: { * text => assertion }, + + ; All-events assertion (applied to every event) + ? each_event: { * text => assertion } +} + +ordering-rule = "monotonic_state" ; Task states never regress + +event-assertion = { + ? description: text, + ? match: "exact_position" / "any_position", ; Default: exact_position + ? index: int, ; Specific event index (0-based) + + ; Exactly one of these StreamResponse payload types: + ? status_update: { * text => assertion }, + ? artifact_update: { * text => assertion }, + ? task: { * text => assertion }, + ? message: { * text => assertion }, + + ; Or a generic assertion on the event: + * text => assertion +} +``` + +### 7.1. Ordering Rules + +When `ordering: monotonic_state` is specified, the runner MUST verify that task state transitions in the event stream follow the A2A state machine and never regress illegally. + +The A2A task states are: + +- **Non-terminal:** `TASK_STATE_SUBMITTED`, `TASK_STATE_WORKING`, `TASK_STATE_INPUT_REQUIRED`, `TASK_STATE_AUTH_REQUIRED` +- **Terminal:** `TASK_STATE_COMPLETED`, `TASK_STATE_FAILED`, `TASK_STATE_CANCELED`, `TASK_STATE_REJECTED` + +The valid transitions are: + +``` +SUBMITTED → WORKING +WORKING → COMPLETED | FAILED | CANCELED | REJECTED | INPUT_REQUIRED | AUTH_REQUIRED +INPUT_REQUIRED → WORKING (client sent more input) +AUTH_REQUIRED → WORKING (authorization was provided) +``` + +A state MAY repeat (e.g., multiple WORKING events). The following transitions are **illegal** and MUST cause a test failure: + +- Any state → SUBMITTED (SUBMITTED is only the initial state) +- Any terminal state → any state (terminal states are final) + +Note that `INPUT_REQUIRED → WORKING` and `AUTH_REQUIRED → WORKING` are valid (the interrupted condition was resolved). The `monotonic_state` rule enforces "no regression from terminal states" and "no illegal transitions," not a simple linear progression. + +### Example + +```yaml +- id: CORE-STREAM-001 + name: "Basic streaming lifecycle" + level: must + preconditions: + capabilities: + streaming: true + steps: + - id: stream + operation: send_streaming_message + params: + message: + role: ROLE_USER + parts: + - text: "generate streaming output" + expect_stream: + min_count: 2 + ordering: monotonic_state + events: + - description: "A status update to WORKING" + match: any_position + status_update: + status: + state: TASK_STATE_WORKING + - description: "At least one artifact update" + match: any_position + artifact_update: + artifact: + parts: {count_gte: 1} + final_event: + one_of: + - task: + status: + state: TASK_STATE_COMPLETED + - status_update: + final: true + status: + state: TASK_STATE_COMPLETED +``` + +--- + +## 8. Variables and Capture + +### 8.1. Variables + +ACTS supports variable interpolation using double-brace syntax: `{{expression}}`. + +```cddl +; Variable reference syntax (within text values): +; {{variableName}} — top-level variable +; {{stepId.varName}} — captured variable from a prior step +; {{env.ENV_VAR_NAME}} — environment variable +; {{$uuid}} — generate a fresh UUID (each occurrence produces a new value) +``` + +Variables defined in the document-level `variables` map are available to all tests. Variables captured in a step (via `capture`) are scoped to that test and available in subsequent steps. + +### 8.2. Capture + +A `capture` block extracts values from a response for use in later steps. + +```cddl +; In a step: +capture = { + text => text } +; Key: variable name to assign +; Value: dot-path into the response +``` + +### Example + +```yaml +steps: + - id: create + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "start a task" + capture: + taskId: task.id + contextId: task.contextId + + - id: retrieve + operation: get_task + params: + id: "{{create.taskId}}" + expect: + status: 200 + body: + id: "{{create.taskId}}" + status: + state: TASK_STATE_COMPLETED +``` + +When the runner resolves `{{create.taskId}}`, it substitutes the value captured from the `create` step's response at the path `task.id`. + +--- + +## 9. Polling and Retry + +Some A2A patterns involve polling (e.g., `returnImmediately: true` followed by `GetTask` until completion). The `repeat` block on a step instructs the runner to re-execute the step until a condition is met. + +```cddl +repeat-config = { + until: text, ; Condition expression (see below) + ? max_attempts: int, ; Maximum retry attempts (default: 10) + ? delay_ms: int, ; Delay between attempts in milliseconds (default: 1000) + ? backoff: "none" / "linear" / "exponential" ; Backoff strategy (default: "none") +} +``` + +### 9.1. Until Expressions + +The `until` field contains a condition expression. The following forms are supported: + +``` +path == value ; Equality +path != value ; Inequality +path in [value1, value2, ...] ; Value is one of the listed values +``` + +Where `path` is a dot-path into the step's response, `value` is a literal to compare against, and `[value1, value2, ...]` is a list of acceptable values. The runner re-executes the step until the expression evaluates to true or `max_attempts` is exhausted. + +If `max_attempts` is exhausted without the condition becoming true, the step MUST be marked as failed. + +### Example + +```yaml +- id: poll + operation: get_task + params: + id: "{{start.taskId}}" + expect: + status: 200 + body: + status: + state: {type: string} + repeat: + until: status.state in [TASK_STATE_COMPLETED, TASK_STATE_FAILED] + max_attempts: 15 + delay_ms: 2000 + backoff: linear +``` + +--- + +## 10. Client Tests + +Most ACTS tests verify server behavior: send a request, check the response. Client tests invert this: they provide a canonical wire payload and assert that the client (SDK) parses it correctly. + +```cddl +client-step = { + id: text, + ? description: text, + client_response: client-response, + expect_parsed: { * text => assertion }, + ? assertions: [+ named-assertion] +} + +client-response = { + operation: abstract-operation, + wire_payload: any +} +``` + +Client tests are valuable for catching interop bugs: they ensure that every SDK can parse responses produced by any other conformant SDK. The `client_response` payloads in the official test suite represent the canonical wire format for each abstract operation. + +### Example + +```yaml +- id: CLIENT-PARSE-001 + name: "Client parses a canonical SendMessage task response" + level: must + steps: + - id: parse + client_response: + operation: send_message + wire_payload: + jsonrpc: "2.0" + id: "1" + result: + task: + id: "abc-123" + contextId: "ctx-456" + status: + state: TASK_STATE_COMPLETED + timestamp: "2026-01-01T00:00:00Z" + artifacts: + - artifactId: "art-1" + parts: + - text: "hello world" + expect_parsed: + task: + id: "abc-123" + status: + state: TASK_STATE_COMPLETED + artifacts: + - parts: + - text: "hello world" +``` + +--- + +## 11. SUT Behavior Contract + +ACTS tests are declarative: they describe what to send and what to expect. But to produce deterministic, verifiable responses, the SUT must exhibit specific behaviors in response to specific inputs. + +ACTS uses a **message-prefix convention**: the text content of the first message part signals to the SUT what behavior to exhibit. This eliminates the need for a side-channel API to control the SUT. + +### 11.1. Behavior Definition + +```cddl +sut-behaviors = { + acts_version: text, + behaviors: [+ behavior] +} + +behavior = { + prefix: text, ; Message text prefix that triggers this behavior + description: text, ; What the SUT should do + ? response_type: "task" / "message", + ? terminal_state: text, ; Expected terminal TaskState + ? artifacts: [+ artifact-spec], ; Artifacts to include in the response + ? delay_ms: int, ; Simulated processing delay + ? streaming: bool ; Whether this applies to streaming requests +} + +artifact-spec = { + ? text: text, + ? data: any, + ? file: { name: text, mediaType: text }, + ? fileUrl: { url: text, name: text, mediaType: text } +} +``` + +### 11.2. Standard Prefixes + +The following prefixes form the standard SUT behavior contract. Any SUT that implements these behaviors can be tested with the official ACTS test suite. + +| Prefix | Behavior | +|--------|----------| +| `tck-complete-task` | Complete the task with a text response message. | +| `tck-input-required` | Return task in `TASK_STATE_INPUT_REQUIRED` state. | +| `tck-reject-task` | Reject the task with an error. | +| `tck-message-response` | Return a `Message` (not a `Task`). | +| `tck-artifact-text` | Complete with a text artifact. | +| `tck-artifact-file` | Complete with a file artifact (inline bytes). | +| `tck-artifact-file-url` | Complete with a file URL artifact. | +| `tck-artifact-data` | Complete with a structured data artifact. | +| `tck-long-running` | Simulate long-running work (delayed completion). | +| `tck-multi-turn` | Require multiple turns (INPUT_REQUIRED → done). | +| `tck-cancel` | Accept and remain in WORKING state until canceled. | +| `tck-stream-basic` | Stream: status(working) → artifact → status(completed). | +| `tck-stream-chunked` | Stream: chunked artifact across multiple events. | +| `tck-task-failure` | Complete with `TASK_STATE_FAILED` and error message. | + +### Example SUT Behaviors File + +```yaml +acts_version: "1.0" +behaviors: + - prefix: "tck-complete-task" + description: "Complete the task with a text response" + response_type: task + terminal_state: TASK_STATE_COMPLETED + + - prefix: "tck-message-response" + description: "Return a direct Message, not a Task" + response_type: message + + - prefix: "tck-artifact-text" + description: "Complete with a text artifact" + response_type: task + terminal_state: TASK_STATE_COMPLETED + artifacts: + - text: "Generated text content" + + - prefix: "tck-long-running" + description: "Simulate long-running processing" + response_type: task + terminal_state: TASK_STATE_COMPLETED + delay_ms: 5000 + + - prefix: "tck-stream-basic" + description: "Stream status updates and an artifact" + response_type: task + terminal_state: TASK_STATE_COMPLETED + streaming: true + artifacts: + - text: "Streamed content" +``` + +--- + +## 12. Runner Requirements + +This section defines requirements for ACTS-compliant test runners. + +### 12.1. Input + +A runner MUST accept one or more ACTS YAML files (or equivalent JSON). A runner MUST validate input against the ACTS structure defined in this specification before executing tests. + +### 12.2. Variable Resolution + +A runner MUST resolve variables in the following order of precedence (highest first): + +1. Step-level captured variables (`{{stepId.varName}}`) +2. Document-level `variables` map +3. Environment variables (`{{env.VAR_NAME}}`) +4. Built-in generators (`{{$uuid}}`) + +If a variable cannot be resolved, the runner MUST fail the step with a clear error message. + +### 12.3. Transport Mapping + +A runner MUST support at least one transport binding. When a test specifies `transport`, the runner MUST skip the test if it does not support any of the listed bindings. When `transport` is omitted, the runner MUST execute the test against whichever binding(s) it supports. + +### 12.4. Protocol Compliance + +When executing abstract operations, the runner MUST produce protocol-compliant requests: + +- **A2A-Version header:** The runner MUST include the `A2A-Version` header (or gRPC metadata equivalent) on all requests, set to the `spec_version` from the ACTS document, unless the test is specifically testing version negotiation behavior (e.g., VER-* tests that deliberately omit or alter the header). +- **messageId generation:** If a request includes a `message` and no `messageId` is specified in the test, the runner MUST auto-generate a unique `messageId` (e.g., a UUID). The A2A specification requires every message to have a `messageId`. +- **JSON-RPC envelope:** For JSON-RPC transport, the runner MUST wrap the request parameters in a valid JSON-RPC 2.0 request object (with `jsonrpc`, `method`, and `id` fields) and unwrap the JSON-RPC envelope from responses before applying assertions. +- **Base URL:** The runner MUST be configurable with the SUT's base URL. The document-level `baseUrl` variable is available for raw wire steps that construct URLs manually, but abstract operations derive the URL from the runner's configuration and the operation's standard path. + +### 12.5. Precondition Evaluation + +Before executing a test, the runner MUST evaluate its `preconditions`. If the SUT does not meet the preconditions (e.g., its agent card does not advertise the required capabilities), the runner MUST mark the test as **skipped**. + +### 12.6. Reporting + +A runner SHOULD produce a structured report containing, for each test: + +- Test ID and name +- Conformance level +- Result: `pass`, `fail`, `skip`, or `error` +- For failures: which step failed, which assertion failed, expected vs. actual values +- Execution time + +Runners MUST support the structured JSON report format defined in §13. Runners MAY additionally produce other formats (JUnit XML, HTML, TAP, etc.). + +### 12.7. Conformance Summary + +A runner SHOULD produce a conformance summary categorized by level: + +``` +MUST: 45/47 passed (2 skipped) +SHOULD: 12/15 passed (1 failed, 2 skipped) +MAY: 8/10 passed (2 skipped) +``` + +An implementation is considered **A2A conformant** if and only if all `must`-level tests pass (excluding skipped tests whose preconditions were not met). + +--- + +## 13. Report Format + +Runners MUST be capable of producing a **JSON report** conforming to the schema below. This enables dashboards, CI systems, and certification portals to consume results from any runner implementation. + +### 13.1. Report Document + +```cddl +acts-report = { + acts_version: text, ; ACTS format version used + spec_version: text, ; A2A protocol version tested + generated_at: text, ; ISO 8601 timestamp + sdk: sdk-info, + transport: transport-binding, + ? environment: { * text => text }, ; OS, runtime version, etc. + summary: summary, + suites: [+ suite-result] +} + +sdk-info = { + name: text, ; e.g., "a2a-python", "a2a-dotnet" + version: text, ; SDK version string + language: text, ; e.g., "python", "csharp", "go" + ? repository: text ; URL to SDK source +} + +transport-binding = "jsonrpc" / "grpc" / "rest" +``` + +### 13.2. Summary + +```cddl +summary = { + total: int, + passed: int, + failed: int, + skipped: int, + errors: int, ; Runner errors (not test failures) + duration_ms: int, + by_level: { + must: level-summary, + should: level-summary, + may: level-summary + } +} + +level-summary = { + total: int, + passed: int, + failed: int, + skipped: int, + errors: int +} +``` + +### 13.3. Suite and Test Results + +```cddl +suite-result = { + id: text, + name: text, + tests: [+ test-result] +} + +test-result = { + id: text, ; Matches the test id from the ACTS file + name: text, + level: "must" / "should" / "may", + result: "pass" / "fail" / "skip" / "error", + duration_ms: int, + ? skip_reason: text, ; Why the test was skipped + ? failure: failure-detail, + ? steps: [+ step-result] ; Optional per-step breakdown +} + +failure-detail = { + message: text, ; Human-readable failure description + ? step_id: text, ; Which step failed + ? expected: text, ; Expected value (stringified) + ? actual: text, ; Actual value (stringified) + ? assertion_path: text ; Dot-path to the failed field +} + +step-result = { + id: text, + result: "pass" / "fail" / "skip" / "error", + duration_ms: int, + ? failure: failure-detail +} +``` + +### 13.4. Example Report + +```json +{ + "acts_version": "1.0", + "spec_version": "1.0", + "generated_at": "2025-05-25T21:00:00Z", + "sdk": { + "name": "a2a-python", + "version": "0.3.1", + "language": "python", + "repository": "https://github.com/a2aproject/a2a-python" + }, + "transport": "jsonrpc", + "environment": { + "os": "ubuntu-24.04", + "runtime": "python 3.12.3", + "ci": "github-actions" + }, + "summary": { + "total": 42, + "passed": 39, + "failed": 2, + "skipped": 1, + "errors": 0, + "duration_ms": 12345, + "by_level": { + "must": { "total": 30, "passed": 28, "failed": 2, "skipped": 0, "errors": 0 }, + "should": { "total": 8, "passed": 8, "failed": 0, "skipped": 0, "errors": 0 }, + "may": { "total": 4, "passed": 3, "failed": 0, "skipped": 1, "errors": 0 } + } + }, + "suites": [ + { + "id": "core-operations", + "name": "Core Operations", + "tests": [ + { + "id": "CORE-SEND-001", + "name": "SendMessage returns a completed task", + "level": "must", + "result": "pass", + "duration_ms": 234 + }, + { + "id": "CORE-GET-001", + "name": "GetTask returns current state", + "level": "must", + "result": "fail", + "duration_ms": 156, + "failure": { + "message": "Expected state TASK_STATE_COMPLETED but got TASK_STATE_WORKING", + "step_id": "verify", + "expected": "TASK_STATE_COMPLETED", + "actual": "TASK_STATE_WORKING", + "assertion_path": "status.state" + } + } + ] + } + ] +} +``` + +### 13.5. Report File Naming + +Report files SHOULD follow the naming convention: + +``` +acts-report-{sdk}-{transport}-{timestamp}.json +``` + +For example: `acts-report-a2a-python-jsonrpc-20250525T210000Z.json` + +--- + +## 14. Requirement ID Scheme + +Every test MUST have a unique `id` that serves as its requirement identifier. IDs follow the pattern: + +``` +CATEGORY-AREA-NNN +``` + +Where: + +| Prefix | Category | +|--------|----------| +| `CORE` | Core operations (SendMessage, GetTask, CancelTask, ListTasks) | +| `STREAM` | Streaming (SendStreamingMessage, SubscribeToTask, event ordering) | +| `PUSH` | Push notifications (config CRUD, webhook delivery) | +| `CARD` | Agent Card discovery and structure | +| `DM` | Data model and serialization (field naming, enums, timestamps) | +| `JSONRPC` | JSON-RPC binding-specific behavior | +| `GRPC` | gRPC binding-specific behavior | +| `REST` | HTTP+JSON/REST binding-specific behavior | +| `VER` | Version negotiation | +| `INTEROP` | Tests derived from cross-SDK interop bugs | +| `CLIENT` | Client-side parsing tests (`client_response` payloads) | + +The `AREA` component provides finer grouping within a category (e.g., `SEND`, `GET`, `CANCEL`, `ERR`, `HIST`, `SSE`, `FMT`). + +--- + +## 15. Complete Example + +The following is a complete, minimal ACTS file demonstrating the key features. + +```yaml +acts_version: "1.0" +spec_version: "1.0" + +metadata: + title: "A2A v1.0 Core Conformance Tests (Excerpt)" + +variables: + baseUrl: "{{env.A2A_BASE_URL}}" + +suites: + - id: core-operations + name: "Core Operations" + tests: + + # ── SendMessage: completed task ───────────────────────────── + - id: CORE-SEND-001 + name: "SendMessage returns a completed task" + description: > + Send a message to the agent. The response MUST contain a Task + in TASK_STATE_COMPLETED state. + spec_ref: "specification.md#3.1.1" + level: must + tags: [send-message, task] + requires_behaviors: [tck-complete-task] + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task hello" + expect: + status: 200 + body: + task: + id: {type: string} + status: + state: TASK_STATE_COMPLETED + capture: + taskId: task.id + contextId: task.contextId + + - id: get + operation: get_task + params: + id: "{{send.taskId}}" + expect: + status: 200 + body: + id: "{{send.taskId}}" + status: + state: TASK_STATE_COMPLETED + + # ── SendMessage: message-only response ────────────────────── + - id: CORE-SEND-003 + name: "SendMessage returns a Message (not a Task)" + description: > + When the agent handles a request without creating a task, + it MUST return a Message with role ROLE_AGENT. + spec_ref: "specification.md#3.1.1" + level: must + tags: [send-message, message-only] + requires_behaviors: [tck-message-response] + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-message-response hello" + expect: + status: 200 + body: + message: + role: ROLE_AGENT + parts: {count_gte: 1} + + # ── CancelTask ───────────────────────────────────────────── + - id: CORE-CANCEL-001 + name: "CancelTask cancels a working task" + spec_ref: "specification.md#3.1.5" + level: must + tags: [cancel, task] + requires_behaviors: [tck-cancel] + steps: + - id: start + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-cancel start work" + configuration: + returnImmediately: true + expect: + status: 200 + body: + task: + status: + state: {one_of: [TASK_STATE_SUBMITTED, TASK_STATE_WORKING]} + capture: + taskId: task.id + + - id: cancel + operation: cancel_task + params: + id: "{{start.taskId}}" + expect: + status: 200 + body: + id: "{{start.taskId}}" + status: + state: TASK_STATE_CANCELED + + # ── Error: TaskNotFound ───────────────────────────────────── + - id: CORE-ERR-001 + name: "GetTask with non-existent ID returns TaskNotFoundError" + spec_ref: "specification.md#4.2" + level: must + tags: [error, task-not-found] + steps: + - id: get-missing + operation: get_task + params: + id: "00000000-0000-0000-0000-000000000000" + expect_error: + error_type: TaskNotFoundError + message: {type: string} + + # ── Streaming ─────────────────────────────────────────────── + - id: CORE-STREAM-001 + name: "Basic streaming lifecycle" + spec_ref: "specification.md#3.1.2" + level: must + tags: [streaming] + requires_behaviors: [tck-stream-basic] + preconditions: + capabilities: + streaming: true + steps: + - id: stream + operation: send_streaming_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-stream-basic hello" + expect_stream: + min_count: 2 + ordering: monotonic_state + final_event: + task: + status: + state: TASK_STATE_COMPLETED + + # ── Multi-turn ────────────────────────────────────────────── + - id: CORE-MULTI-001 + name: "Multi-turn conversation with INPUT_REQUIRED" + spec_ref: "specification.md#3.1.1" + level: must + tags: [multi-turn, input-required] + requires_behaviors: [tck-multi-turn] + steps: + - id: turn1 + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-multi-turn start" + expect: + status: 200 + body: + task: + status: + state: TASK_STATE_INPUT_REQUIRED + capture: + taskId: task.id + contextId: task.contextId + + - id: turn2 + operation: send_message + params: + message: + role: ROLE_USER + taskId: "{{turn1.taskId}}" + contextId: "{{turn1.contextId}}" + parts: + - text: "done" + expect: + status: 200 + body: + task: + id: "{{turn1.taskId}}" + status: + state: TASK_STATE_COMPLETED + + # ── Polling ───────────────────────────────────────────────── + - id: CORE-EXEC-001 + name: "Long-running task with polling" + spec_ref: "specification.md#3.1.1" + level: must + tags: [long-running, polling] + requires_behaviors: [tck-long-running] + steps: + - id: start + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-long-running start" + configuration: + returnImmediately: true + expect: + status: 200 + body: + task: + status: + state: {one_of: [TASK_STATE_SUBMITTED, TASK_STATE_WORKING]} + capture: + taskId: task.id + + - id: poll + operation: get_task + params: + id: "{{start.taskId}}" + expect: + status: 200 + body: + status: + state: {type: string} + repeat: + until: status.state in [TASK_STATE_COMPLETED, TASK_STATE_FAILED] + max_attempts: 15 + delay_ms: 2000 + + - id: discovery + name: "Agent Card Discovery" + tests: + + - id: CARD-DISC-001 + name: "Agent card is retrievable" + spec_ref: "specification.md#5" + level: must + tags: [discovery, agent-card] + steps: + - id: get-card + operation: get_agent_card + params: {} + expect: + status: 200 + body: + name: {type: string} + supportedInterfaces: {type: array, count_gte: 1} + skills: {type: array} + + - id: data-model + name: "Data Model & Serialization" + tests: + + - id: DM-SERIAL-001 + name: "JSON serialization uses camelCase field names" + spec_ref: "specification.md#2.3" + level: must + transport: [jsonrpc, rest] + tags: [serialization, wire-format] + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task camelCase test" + expect: + status: 200 + body: + task: + id: {type: string} + # Runner must additionally verify raw response keys are camelCase + + - id: DM-SERIAL-005 + name: "Server ignores unrecognized fields" + spec_ref: "specification.md#2.3" + level: must + tags: [serialization, tolerance] + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task with extra fields" + futureField: "should be ignored" + extraNested: + foo: "bar" + expect: + status: 200 + body: + task: + status: + state: TASK_STATE_COMPLETED +``` + +--- + +## 16. File Organization + +The official A2A conformance test suite SHOULD be organized as follows: + +``` +a2a-conformance/ +├── acts-spec.md # This specification +├── v1.0/ # Tests for A2A spec v1.0 +│ ├── suite.yaml # Master file listing all test files +│ ├── discovery.yaml # CARD-* tests +│ ├── core-operations.yaml # CORE-* tests +│ ├── streaming.yaml # STREAM-* tests +│ ├── error-handling.yaml # Error tests (CORE-ERR-*, CORE-CAP-*) +│ ├── multi-turn.yaml # CORE-MULTI-* tests +│ ├── push-notifications.yaml # PUSH-* tests +│ ├── data-model.yaml # DM-* tests +│ ├── history.yaml # CORE-HIST-* tests +│ ├── version-negotiation.yaml # VER-* tests +│ ├── transport-jsonrpc.yaml # JSONRPC-* tests +│ ├── transport-grpc.yaml # GRPC-* tests +│ ├── transport-rest.yaml # REST-* tests +│ ├── client-parsing.yaml # CLIENT-* client response tests +│ ├── interop.yaml # INTEROP-* tests (from cross-SDK bugs) +│ └── sut-behaviors.yaml # Standard SUT behavior contract +└── v0.3/ # Backward compatibility tests (if needed) + └── ... +``` + +The `suite.yaml` master file references all test files in the version directory: + +```yaml +acts_version: "1.0" +spec_version: "1.0" +include: + - discovery.yaml + - core-operations.yaml + - streaming.yaml + - error-handling.yaml + - multi-turn.yaml + - push-notifications.yaml + - data-model.yaml + - history.yaml + - version-negotiation.yaml + - transport-jsonrpc.yaml + - transport-grpc.yaml + - transport-rest.yaml + - client-parsing.yaml + - interop.yaml +``` + +--- + +## 17. Versioning + +The ACTS format itself is versioned independently of the A2A protocol. The `acts_version` field in each document identifies the format version. The `spec_version` field identifies which A2A protocol version the tests target. + +When the ACTS format evolves: + +- **Patch** (e.g., 1.0.1): Clarifications, typo fixes. No structural changes. +- **Minor** (e.g., 1.1): New optional fields, new assertion operators. Backward-compatible. +- **Major** (e.g., 2.0): Breaking changes to the format structure. + +Runners MUST reject documents with an `acts_version` major version they do not support. + +--- + +## Appendix A: Full CDDL Grammar + +This appendix consolidates all CDDL definitions from the specification into a single grammar. + +```cddl +; ── Document ────────────────────────────────────────────────────── +acts-document = { + acts_version: text, + spec_version: text, + ? spec_ref: text, + ? metadata: metadata, + ? variables: { * text => text }, + ? include: [+ text], ; For suite.yaml: list of files to include + ? suites: [+ suite] ; At least one of include or suites MUST be present +} + +metadata = { + ? title: text, + ? description: text, + ? authors: [+ text], + ? license: text +} + +; ── Suite ───────────────────────────────────────────────────────── +suite = { + id: text, + name: text, + ? description: text, + ? tags: [+ text], + tests: [+ test] +} + +; ── Test ────────────────────────────────────────────────────────── +test = { + id: text, + name: text, + ? description: text, + ? spec_ref: text, + level: conformance-level, + ? tags: [+ text], + ? transport: [+ transport-binding], + ? preconditions: preconditions, + ? requires_behaviors: [+ text], ; SUT behavior prefixes this test depends on + ? origin: text, + ? runner_requirements: [+ runner-requirement], + steps: [+ step], + ? assertions: [+ named-assertion] +} + +conformance-level = "must" / "should" / "may" +transport-binding = "jsonrpc" / "grpc" / "rest" + +runner-requirement = + "webhook_endpoint" / + "concurrent_streams" / + "stream_disconnect" / + "auth_credentials" / + "header_inspection" + +preconditions = { + ? capabilities: { * text => any }, + ? skills: [+ { id: text }], + ? transport: [+ transport-binding], + ? extensions: [+ text], + ? description: text +} + +; ── Steps ───────────────────────────────────────────────────────── +step = server-step / client-step / assertion-step / raw-step + +server-step = { + id: text, + ? description: text, + operation: abstract-operation, + params: request-params, + ? expect: expect-block, + ? expect_error: expect-error, + ? expect_stream: expect-stream, + ? capture: { + text => text }, + ? assertions: [+ named-assertion], + ? repeat: repeat-config, + ? delay_ms: int +} + +client-step = { + id: text, + ? description: text, + client_response: client-response, + expect_parsed: { * text => assertion }, + ? assertions: [+ named-assertion] +} + +client-response = { + operation: abstract-operation, + wire_payload: any +} + +assertion-step = { + id: text, + ? description: text, + assertion: inline-assertion +} + +raw-step = { + id: text, + ? description: text, + raw: { + method: "GET" / "POST" / "PUT" / "DELETE", + path: text, + ? headers: { * text => text }, + ? body: any, + ? body_raw: text + }, + ? expect: expect-block, + ? expect_error: expect-error, + ? capture: { + text => text }, + ? assertions: [+ named-assertion] +} + +; ── Operations ──────────────────────────────────────────────────── +abstract-operation = + "send_message" / + "send_streaming_message" / + "get_task" / + "list_tasks" / + "cancel_task" / + "subscribe_to_task" / + "get_agent_card" / + "get_extended_agent_card" / + "create_push_config" / + "get_push_config" / + "list_push_configs" / + "delete_push_config" + +; ── Params ──────────────────────────────────────────────────────── +request-params = { + ? message: { + role: text, + parts: [+ part], + ? messageId: text, + ? taskId: text, + ? contextId: text, + * text => any + }, + ? configuration: { + ? returnImmediately: bool, + ? historyLength: int, + * text => any + }, + ? id: text, + ? historyLength: int, + ? contextId: text, + ? cursor: text, + ? pageSize: int, + ? taskId: text, + ? pushNotificationConfig: any, + * text => any +} + +part = { + ? text: text, + ? data: any, + ? file: { + ? url: text, + ? raw: text, + ? mediaType: text, + ? name: text + }, + * text => any +} + +; ── Expect ──────────────────────────────────────────────────────── +expect-block = { + ? status: assertion, + ? body: { * text => assertion } +} + +expect-error = { + error_type: a2a-error-type, + ? message: assertion, + ? details: { * text => assertion } +} + +a2a-error-type = + "TaskNotFoundError" / + "TaskNotCancelableError" / + "UnsupportedOperationError" / + "ContentTypeNotSupportedError" / + "InvalidParamsError" / + "VersionNotSupportedError" / + "PushNotificationNotSupportedError" / + "StreamingNotSupportedError" / + "ExtensionSupportRequiredError" / + "ExtendedCardNotSupportedError" / + "JSONParseError" / + "MethodNotFoundError" / + "InternalError" + +; ── Streaming ───────────────────────────────────────────────────── +expect-stream = { + ? min_count: int, + ? max_count: int, + ? timeout_ms: int, + ? ordering: ordering-rule, + ? events: [+ event-assertion], + ? final_event: { * text => assertion }, + ? each_event: { * text => assertion } +} + +ordering-rule = "monotonic_state" + +event-assertion = { + ? description: text, + ? match: "exact_position" / "any_position", + ? index: int, + ? status_update: { * text => assertion }, + ? artifact_update: { * text => assertion }, + ? task: { * text => assertion }, + ? message: { * text => assertion }, + * text => assertion +} + +; ── Assertions ──────────────────────────────────────────────────── +assertion = exact-match / assertion-object + +exact-match = text / int / float / bool / null + +assertion-object = { + ? type: "string" / "number" / "boolean" / "array" / "object" / "null", + ? exists: true, + ? absent: true, + ? contains: text, + ? matches: text, + ? starts_with: text, + ? ends_with: text, + ? gte: number, + ? lte: number, + ? gt: number, + ? lt: number, + ? count: int, + ? count_gte: int, + ? count_lte: int, + ? one_of: [+ any], + ? all_of: [+ assertion], + ? any_of: [+ assertion], + ? not: assertion +} + +number = int / float + +; ── Collection Assertions ───────────────────────────────────────── +inline-assertion = { + source: text, + ? any: collection-match, + ? all: collection-match, + ? none: collection-match +} + +named-assertion = { + ? id: text, + ? description: text, + source: text, + ? path: text, + ? match: assertion, + ? any: collection-match, + ? all: collection-match, + ? none: collection-match +} + +collection-match = { + path: text, + match: { * text => assertion } +} + +; ── Polling ─────────────────────────────────────────────────────── +repeat-config = { + until: text, + ? max_attempts: int, + ? delay_ms: int, + ? backoff: "none" / "linear" / "exponential" +} + +; ── SUT Behaviors ───────────────────────────────────────────────── +sut-behaviors = { + acts_version: text, + behaviors: [+ behavior] +} + +behavior = { + prefix: text, + description: text, + ? response_type: "task" / "message", + ? terminal_state: text, + ? artifacts: [+ artifact-spec], + ? delay_ms: int, + ? streaming: bool +} + +artifact-spec = { + ? text: text, + ? data: any, + ? file: { name: text, mediaType: text }, + ? fileUrl: { url: text, name: text, mediaType: text } +} + +; ── Report Format ───────────────────────────────────────────────── +acts-report = { + acts_version: text, + spec_version: text, + generated_at: text, + sdk: sdk-info, + transport: transport-binding, + ? environment: { * text => text }, + summary: summary, + suites: [+ suite-result] +} + +sdk-info = { + name: text, + version: text, + language: text, + ? repository: text +} + +summary = { + total: int, + passed: int, + failed: int, + skipped: int, + errors: int, + duration_ms: int, + by_level: { + must: level-summary, + should: level-summary, + may: level-summary + } +} + +level-summary = { + total: int, + passed: int, + failed: int, + skipped: int, + errors: int +} + +suite-result = { + id: text, + name: text, + tests: [+ test-result] +} + +test-result = { + id: text, + name: text, + level: "must" / "should" / "may", + result: "pass" / "fail" / "skip" / "error", + duration_ms: int, + ? skip_reason: text, + ? failure: failure-detail, + ? steps: [+ step-result] +} + +failure-detail = { + message: text, + ? step_id: text, + ? expected: text, + ? actual: text, + ? assertion_path: text +} + +step-result = { + id: text, + result: "pass" / "fail" / "skip" / "error", + duration_ms: int, + ? failure: failure-detail +} +``` + +--- + +## Appendix B: Existing Requirement ID Inventory + +The following requirement IDs are carried forward from existing conformance testing efforts (primarily the A2A TCK). New tests SHOULD follow the same naming pattern. + +### CORE (Core Operations) +| ID | Description | +|----|-------------| +| CORE-SEND-001 | SendMessage returns a completed task | +| CORE-SEND-002 | SendMessage to a terminal task returns UnsupportedOperationError | +| CORE-SEND-003 | ContentTypeNotSupportedError detected by SDK | +| CORE-GET-001 | GetTask returns the current state of an existing task | +| CORE-CANCEL-001 | CancelTask returns the task with updated state | +| CORE-CANCEL-002 | CancelTask on a terminal task returns TaskNotCancelableError | +| CORE-MULTI-001 | Multi-turn conversation with INPUT_REQUIRED | +| CORE-MULTI-005 | SendMessage with only taskId infers contextId | +| CORE-MULTI-006 | SendMessage with taskId + wrong contextId returns error | +| CORE-EXEC-001 | Long-running task with returnImmediately and polling | +| CORE-ERR-001 | Server returns appropriate errors with actionable info | +| CORE-ERR-002 | Server validates input parameters | +| CORE-CAP-001 | Push operations return error when not supported | +| CORE-CAP-002 | Streaming operations return error when not supported | +| CORE-CAP-003 | Extended agent card returns error when not supported | +| CORE-CAP-004 | Required extension missing returns error | +| CORE-HIST-001 | GetTask with historyLength=0 returns no history | +| CORE-HIST-002 | GetTask history count ≤ historyLength | +| CORE-HIST-003 | SendMessage with historyLength=0 returns no history | +| CORE-HIST-004 | GetTask with historyLength>0 may return persisted messages | +| CORE-HIST-005 | History messages in chronological order | +| CORE-HIST-006 | History content matches exchanged messages | + +### STREAM (Streaming) +| ID | Description | +|----|-------------| +| STREAM-ORDER-001 | Task states never regress; last event is terminal | +| STREAM-ORDER-002 | Events broadcast to all active streams | +| STREAM-ORDER-003 | Each stream receives same events in same order | +| STREAM-ORDER-004 | Closing one stream does not affect others | +| STREAM-SUB-001 | First SubscribeToTask event contains a Task | +| STREAM-SUB-002 | SubscribeToTask stream closes at terminal state | +| STREAM-SUB-003 | SubscribeToTask on terminal task returns error | +| STREAM-SUB-004 | SubscribeToTask on non-existent task returns TaskNotFoundError | + +### CARD (Agent Card Discovery) +| ID | Description | +|----|-------------| +| CARD-DISC-001 | Agent card is retrievable | +| CARD-STRUCT-001 | AgentCard contains required fields | +| CARD-PROTO-001 | supportedInterfaces is non-empty | +| CARD-PROTO-002 | Each interface validates against AgentInterface schema | +| CARD-CACHE-001 | Agent Card includes Cache-Control header | +| CARD-CACHE-002 | Agent Card includes ETag header | +| CARD-CACHE-003 | Agent Card may include Last-Modified header | + +### DM (Data Model & Serialization) +| ID | Description | +|----|-------------| +| DM-TASK-001 | Task has required fields (id, status) | +| DM-MSG-001 | Message has required fields (role, parts, messageId) | +| DM-PART-001 | Part oneof semantics | +| DM-ART-001 | Artifact has required fields | +| DM-STATUS-001 | TaskStatus.state field present | +| DM-SERIAL-001 | JSON uses camelCase field names | +| DM-SERIAL-002 | Enum values are strings, not integers | +| DM-SERIAL-003 | Timestamps use ISO 8601 with Z suffix | +| DM-SERIAL-005 | Implementations ignore unrecognized fields | + +### PUSH (Push Notifications) +| ID | Description | +|----|-------------| +| PUSH-CREATE-001 | CreatePushNotificationConfig returns the config | +| PUSH-CREATE-002 | Config persists and can be retrieved | +| PUSH-GET-001 | GetPushNotificationConfig returns details | +| PUSH-GET-002 | GetPushNotificationConfig with nonexistent ID errors | +| PUSH-LIST-001 | ListPushNotificationConfigs includes created config | +| PUSH-DEL-001 | DeletePushNotificationConfig removes config | +| PUSH-DEL-002 | Deleting already-deleted config is idempotent | +| PUSH-DELIVER-001 | Agent includes auth credentials in webhook | +| PUSH-DELIVER-002 | Agent attempts delivery at least once | +| PUSH-DELIVER-003 | Webhook payload uses StreamResponse format | + +### JSONRPC (JSON-RPC Binding) +| ID | Description | +|----|-------------| +| JSONRPC-FMT-001 | Response conforms to JSON-RPC 2.0 | +| JSONRPC-FMT-002 | Content-Type is application/json | +| JSONRPC-SVC-001 | Method names match spec | +| JSONRPC-SVC-002 | Service parameters in HTTP headers | +| JSONRPC-SSE-001 | SSE events are JSON-RPC 2.0 responses | +| JSONRPC-SSE-002 | Error types map to correct JSON-RPC codes | +| JSONRPC-ERR-001 | Error object has code, message, optional data | +| JSONRPC-ERR-002 | A2A errors use codes -32001 to -32099 | +| JSONRPC-ERR-003 | Errors include ErrorInfo in data array | + +### GRPC (gRPC Binding) +| ID | Description | +|----|-------------| +| GRPC-SVC-001 | A2AService responds and validates against proto | +| GRPC-SVC-002 | Uses Protocol Buffers v3 | +| GRPC-META-001 | Service parameters in gRPC metadata | +| GRPC-ERR-001 | Errors include ErrorInfo in status details | +| GRPC-ERR-002 | Errors map to correct gRPC status codes | +| GRPC-ERR-003 | Streaming uses server streaming RPCs | + +### VER (Version Negotiation) +| ID | Description | +|----|-------------| +| VER-SERVER-002 | Unsupported A2A-Version returns VersionNotSupportedError | +| VER-SERVER-003 | Empty A2A-Version treated as 0.3 | + +### BIND (Protocol Binding) +| ID | Description | +|----|-------------| +| BIND-FIELD-001 | All supported protocols declared in AgentCard | diff --git a/tests/acts/auth-security.acts.yaml b/tests/acts/auth-security.acts.yaml new file mode 100644 index 000000000..ec8c1784c --- /dev/null +++ b/tests/acts/auth-security.acts.yaml @@ -0,0 +1,388 @@ +acts_version: "1.0" +spec_version: "1.0" + +metadata: + title: "Authentication and Security" + description: "ACTS coverage for authentication, authorization, and security-sensitive behaviors that require runner-managed credentials or webhooks." + +suites: + - id: auth-security + name: "Authentication and Security" + tests: + - id: SEC-AUTH-001 + name: "Server rejects invalid or missing auth credentials" + description: "Runner MUST attempt a SendMessage request without valid credentials and verify rejection." + spec_ref: "REQ-ERR-002" + level: must + preconditions: + capabilities: + authentication: true + tags: [auth, runner-special] + transport: [jsonrpc] + steps: + - id: send-without-auth + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + A2A-Version: "1.0" + body: + jsonrpc: "2.0" + id: "sec-auth-001" + method: SendMessage + params: + message: + role: ROLE_USER + parts: + - text: "auth should be required" + expect: + status: + one_of: [200, 401, 403] + body: + any_of: + - error: + exists: true + - title: + type: string + + - id: SEC-AUTH-002 + name: "Server returns authorization error for insufficient permissions" + description: "Runner MUST attempt a request with valid but insufficient credentials and verify rejection." + spec_ref: "REQ-ERR-003" + level: must + preconditions: + capabilities: + authentication: true + tags: [auth, runner-special] + transport: [jsonrpc] + steps: + - id: send-insufficient-auth + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + A2A-Version: "1.0" + Authorization: "Bearer {{insufficientAuthToken}}" + body: + jsonrpc: "2.0" + id: "sec-auth-002" + method: SendMessage + params: + message: + role: ROLE_USER + parts: + - text: "authorization should fail" + expect: + status: + one_of: [200, 403] + body: + any_of: + - error: + exists: true + - title: + type: string + + - id: SEC-AUTH-003 + name: "Server does not reveal unauthorized resource existence" + description: "Runner MUST verify that getting a task belonging to another user returns the same error as getting a nonexistent task." + spec_ref: "REQ-ERR-004" + level: must + preconditions: + capabilities: + authentication: true + tags: [auth, security, runner-special] + steps: + - id: get-other-user-task + operation: get_task + params: + id: "{{otherUserTaskId}}" + expect_error: + message: + type: string + + - id: get-missing-task + operation: get_task + params: + id: "00000000-0000-0000-0000-000000000000" + expect_error: + message: + type: string + + - id: SEC-AUTH-004 + name: "Server authenticates every incoming request" + description: "Runner MUST verify that unauthenticated requests to all endpoints are rejected." + spec_ref: "REQ-SEC-004" + level: must + preconditions: + capabilities: + authentication: true + tags: [auth, runner-special] + transport: [jsonrpc] + steps: + - id: get-task-without-auth + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + A2A-Version: "1.0" + body: + jsonrpc: "2.0" + id: "sec-auth-004" + method: GetTask + params: + id: "00000000-0000-0000-0000-000000000000" + expect: + status: + one_of: [200, 401, 403] + body: + any_of: + - error: + exists: true + - title: + type: string + + - id: SEC-AUTH-005 + name: "In-task auth transitions to AUTH_REQUIRED state" + description: "Verifies that an in-task authentication challenge transitions the task into TASK_STATE_AUTH_REQUIRED with a status message." + spec_ref: "REQ-SEC-006" + level: must + requires_behaviors: + - "tck-auth-required" + tags: [auth, task-state] + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-auth-required trigger auth" + expect: + status: 200 + body: + task: + status: + state: TASK_STATE_AUTH_REQUIRED + message: + exists: true + + - id: SEC-PUSH-001 + name: "Push notification webhook includes auth credentials" + description: "Runner MUST set up a webhook and verify auth credentials are present in request headers." + spec_ref: "REQ-SEC-018" + level: must + preconditions: + capabilities: + pushNotifications: true + tags: [auth, push, runner-special] + requires_behaviors: ["tck-long-running"] + steps: + - id: create-task + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-long-running start" + configuration: + returnImmediately: true + expect: + status: 200 + body: + task: + id: + type: string + capture: + taskId: "task.id" + + - id: set-config + operation: set_push_notification_config + params: + taskId: "{{create-task.taskId}}" + pushNotificationConfig: + url: "{{webhookUrl}}" + authentication: + scheme: "Bearer" + credentials: "test-token" + expect: + status: 200 + body: + id: + type: string + + - id: SEC-PUSH-002 + name: "Client validates push notification webhook authenticity" + description: "Runner MUST verify that webhook requests with invalid auth are rejected by the client." + spec_ref: "REQ-SEC-019" + level: must + preconditions: + capabilities: + pushNotifications: true + tags: [auth, push, client, runner-special] + requires_behaviors: ["tck-long-running"] + steps: + - id: create-task + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-long-running start" + configuration: + returnImmediately: true + expect: + status: 200 + body: + task: + id: + type: string + capture: + taskId: "task.id" + + - id: set-config + operation: set_push_notification_config + params: + taskId: "{{create-task.taskId}}" + pushNotificationConfig: + url: "{{webhookUrl}}" + authentication: + scheme: "Bearer" + credentials: "expected-token" + expect: + status: 200 + body: + id: + type: string + + - id: SEC-EXTCARD-001 + name: "Extended Agent Card requires authentication" + description: "Verifies that fetching the extended Agent Card without authentication is rejected." + spec_ref: "REQ-SEC-021" + level: must + preconditions: + capabilities: + extendedAgentCard: true + tags: [auth, discovery, runner-special] + transport: [rest] + steps: + - id: get-extended-card-without-auth + raw: + method: GET + path: "/extendedAgentCard" + headers: + Accept: application/json + A2A-Version: "1.0" + expect: + status: + one_of: [401, 403] + body: + any_of: + - title: + type: string + - error: + exists: true + + - id: SEC-EXTCARD-002 + name: "Extended Agent Card verifies permissions before returning" + description: "Runner MUST verify that an authenticated request with insufficient permissions is rejected." + spec_ref: "REQ-SEC-022" + level: must + preconditions: + capabilities: + extendedAgentCard: true + tags: [auth, discovery, runner-special] + transport: [rest] + steps: + - id: get-extended-card-insufficient-auth + raw: + method: GET + path: "/extendedAgentCard" + headers: + Accept: application/json + A2A-Version: "1.0" + Authorization: "Bearer {{insufficientAuthToken}}" + expect: + status: + one_of: [401, 403] + body: + any_of: + - title: + type: string + - error: + exists: true + + - id: SEC-EXTCARD-003 + name: "Missing extended-card support returns specified error" + description: "Verifies that Get Extended Agent Card returns the specified error when the capability is not advertised." + spec_ref: "REQ-SEC-025" + level: must + preconditions: + capabilities: + extendedAgentCard: false + tags: [auth, discovery] + steps: + - id: get-extended-card + operation: get_agent_card + params: + extended: true + expect_error: + code: UnsupportedOperationError + + - id: SEC-EXTCARD-004 + name: "Extended cards enforce access controls" + description: "Runner MUST verify that extended Agent Cards enforce access controls consistently with the registered endpoint." + spec_ref: "REQ-IANA-008" + level: must + preconditions: + capabilities: + extendedAgentCard: true + tags: [auth, discovery, runner-special] + transport: [rest] + steps: + - id: get-extended-card-without-auth + raw: + method: GET + path: "/extendedAgentCard" + headers: + Accept: application/json + A2A-Version: "1.0" + expect: + status: + one_of: [401, 403] + body: + any_of: + - title: + type: string + - error: + exists: true + + - id: SEC-AUTH-006 + name: "Server uses binding-specific auth challenge or rejection" + description: "Runner SHOULD verify that authentication rejection uses the binding-appropriate mechanism, such as HTTP 401 for HTTP transports." + spec_ref: "REQ-SEC-005" + level: should + preconditions: + capabilities: + authentication: true + tags: [auth, runner-special] + transport: [rest] + steps: + - id: get-agent-card-without-auth + raw: + method: GET + path: "/.well-known/agent-card.json" + headers: + Accept: application/json + A2A-Version: "1.0" + expect: + status: + one_of: [200, 401, 403] + body: + any_of: + - title: + type: string + - error: + exists: true diff --git a/tests/acts/client-parsing.acts.yaml b/tests/acts/client-parsing.acts.yaml new file mode 100644 index 000000000..13bf7b03f --- /dev/null +++ b/tests/acts/client-parsing.acts.yaml @@ -0,0 +1,298 @@ +acts_version: "1.0" +spec_version: "1.0" + +metadata: + title: "Client Parsing (Golden Response) Tests" + description: > + These tests verify that SDK clients can correctly parse canonical A2A + wire payloads. Each test provides a golden JSON response and asserts + that the client SDK parses it into the expected object structure. + +suites: + - id: client-parsing + name: "Client Parsing (Golden Responses)" + tests: + - id: CLIENT-PARSE-001 + name: "Parse SendMessage response with completed task" + description: > + Verifies the SDK client can parse a SendMessage response containing + a completed task with artifacts. + level: must + tags: [client, parsing, task] + steps: + - id: parse + client_response: + operation: send_message + wire_payload: + jsonrpc: "2.0" + id: "req-001" + result: + task: + id: "task-abc-123" + contextId: "ctx-001" + status: + state: TASK_STATE_COMPLETED + timestamp: "2025-05-25T20:00:00Z" + artifacts: + - artifactId: "art-1" + parts: + - text: "hello world" + expect_parsed: + task: + id: "task-abc-123" + contextId: "ctx-001" + status: + state: TASK_STATE_COMPLETED + artifacts: + count_gte: 1 + + - id: CLIENT-PARSE-002 + name: "Parse SendMessage response with message (no task)" + description: > + Verifies the SDK client can parse a SendMessage response containing + a direct message (no task created). + level: must + tags: [client, parsing, message] + steps: + - id: parse + client_response: + operation: send_message + wire_payload: + jsonrpc: "2.0" + id: "req-002" + result: + message: + messageId: "msg-001" + role: ROLE_AGENT + parts: + - text: "I can help with that" + expect_parsed: + message: + messageId: "msg-001" + role: ROLE_AGENT + parts: + count_gte: 1 + + - id: CLIENT-PARSE-003 + name: "Parse GetTask response" + description: > + Verifies the SDK client can parse a GetTask response with full + task state including history and artifacts. + level: must + tags: [client, parsing, get-task] + steps: + - id: parse + client_response: + operation: get_task + wire_payload: + jsonrpc: "2.0" + id: "req-003" + result: + id: "task-xyz-789" + contextId: "ctx-002" + status: + state: TASK_STATE_COMPLETED + timestamp: "2025-05-25T20:05:00Z" + artifacts: + - artifactId: "art-1" + parts: + - text: "result text" + - artifactId: "art-2" + parts: + - data: + key: value + history: + - role: ROLE_USER + parts: + - text: "original request" + - role: ROLE_AGENT + parts: + - text: "processing..." + expect_parsed: + id: "task-xyz-789" + contextId: "ctx-002" + status: + state: TASK_STATE_COMPLETED + artifacts: + count_gte: 2 + history: + count_gte: 2 + + - id: CLIENT-PARSE-004 + name: "Parse error response" + description: > + Verifies the SDK client can parse a JSON-RPC error response and + surface the A2A error code and message. + level: must + tags: [client, parsing, error] + steps: + - id: parse + client_response: + operation: get_task + wire_payload: + jsonrpc: "2.0" + id: "req-004" + error: + code: -32001 + message: "Task not found" + data: + taskId: "nonexistent-task-id" + expect_parsed: + error: + code: -32001 + message: + contains: "not found" + + - id: CLIENT-PARSE-005 + name: "Parse task with INPUT_REQUIRED state" + description: > + Verifies the SDK client correctly parses a task in INPUT_REQUIRED + state with a status message prompting for more input. + level: must + tags: [client, parsing, input-required] + steps: + - id: parse + client_response: + operation: send_message + wire_payload: + jsonrpc: "2.0" + id: "req-005" + result: + task: + id: "task-multi-001" + contextId: "ctx-multi-001" + status: + state: TASK_STATE_INPUT_REQUIRED + timestamp: "2025-05-25T20:10:00Z" + message: + messageId: "status-msg-001" + role: ROLE_AGENT + parts: + - text: "Please provide more details" + expect_parsed: + task: + id: "task-multi-001" + status: + state: TASK_STATE_INPUT_REQUIRED + message: + role: ROLE_AGENT + parts: + count_gte: 1 + + - id: CLIENT-PARSE-006 + name: "Parse task with mixed artifact types" + description: > + Verifies the SDK client can parse artifacts containing text, data, + file, and fileUrl parts in a single response. + level: should + tags: [client, parsing, artifacts, data-types] + steps: + - id: parse + client_response: + operation: get_task + wire_payload: + jsonrpc: "2.0" + id: "req-006" + result: + id: "task-mixed-001" + contextId: "ctx-mixed-001" + status: + state: TASK_STATE_COMPLETED + timestamp: "2025-05-25T20:15:00Z" + artifacts: + - artifactId: "art-text" + parts: + - text: "Plain text result" + - artifactId: "art-data" + parts: + - data: + temperature: 72.5 + unit: "fahrenheit" + - artifactId: "art-file" + parts: + - file: + name: "report.pdf" + mediaType: "application/pdf" + bytes: "JVBERi0xLjQ=" + - artifactId: "art-url" + parts: + - fileUrl: + url: "https://example.com/report.pdf" + name: "report.pdf" + mediaType: "application/pdf" + expect_parsed: + id: "task-mixed-001" + artifacts: + count_gte: 4 + + - id: CLIENT-AUTH-001 + name: "Client replaces cached public card with extended card" + description: "Runner SHOULD verify that after fetching an extended Agent Card, the client uses the extended capabilities for the authenticated session." + spec_ref: "REQ-AGENTCARD-EXT-002" + level: should + tags: [client, discovery, runner-special] + steps: + - id: parse-public-card + client_response: + operation: get_agent_card + wire_payload: + name: "Example Agent" + version: "1.0.0" + capabilities: + streaming: false + extendedAgentCard: true + supportedInterfaces: + - url: "https://example.com/.well-known/agent-card.json" + protocolBinding: REST + protocolVersion: "1.0" + expect_parsed: + capabilities: + extendedAgentCard: true + + - id: parse-extended-card + client_response: + operation: get_extended_agent_card + wire_payload: + name: "Example Agent" + version: "1.0.0" + capabilities: + streaming: true + extendedAgentCard: true + skills: + - id: "extended-skill" + name: "Extended Skill" + supportedInterfaces: + - url: "https://example.com/extendedAgentCard" + protocolBinding: REST + protocolVersion: "1.0" + expect_parsed: + capabilities: + streaming: true + extendedAgentCard: true + skills: + count_gte: 1 + + - id: CLIENT-CAP-001 + name: "Client checks Agent Card before using optional capabilities" + description: "Runner SHOULD verify that the client checks Agent Card capabilities before calling optional operations." + spec_ref: "REQ-CAP-002" + level: should + tags: [client, capabilities, runner-special] + steps: + - id: parse-card + client_response: + operation: get_agent_card + wire_payload: + name: "Capability Gated Agent" + version: "1.0.0" + capabilities: + streaming: false + pushNotifications: false + supportedInterfaces: + - url: "https://example.com/.well-known/agent-card.json" + protocolBinding: REST + protocolVersion: "1.0" + expect_parsed: + capabilities: + streaming: false + pushNotifications: false diff --git a/tests/acts/core-operations.acts.yaml b/tests/acts/core-operations.acts.yaml new file mode 100644 index 000000000..d8c2f6146 --- /dev/null +++ b/tests/acts/core-operations.acts.yaml @@ -0,0 +1,436 @@ +acts_version: "1.0" +spec_version: "1.0" +spec_ref: "docs/specification.md" + +metadata: + title: "Core A2A Operations ACTS" + description: "Core operation conformance tests synthesized from TCK, CSIT task lifecycle, and agentbin scenarios." + +suites: + - id: core-operations + name: "Core A2A operations" + description: "SendMessage, GetTask, and CancelTask conformance coverage for core lifecycle flows." + tests: + - id: CORE-SEND-001 + name: "SendMessage returns a completed task" + description: "Verifies that SendMessage can return a completed task for the complete-task TCK behavior." + spec_ref: "docs/specification.md#311-send-message" + level: must + requires_behaviors: + - "tck-complete-task" + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task hello world" + capture: + taskId: "task.id" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_COMPLETED + + - id: CORE-SEND-002 + name: "SendMessage to a terminal task returns error" + description: "Verifies that sending a follow-up message to a terminal task returns the protocol error for unsupported operation." + spec_ref: "docs/specification.md#311-send-message" + level: must + requires_behaviors: + - "tck-complete-task" + steps: + - id: setup + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task setup" + capture: + taskId: "task.id" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_COMPLETED + - id: resend + operation: send_message + params: + taskId: "{{setup.taskId}}" + message: + role: ROLE_USER + parts: + - text: "tck-complete-task terminal follow-up" + expect_error: + error_type: UnsupportedOperationError + message: + type: string + + - id: CORE-SEND-003 + name: "SendMessage returns a message response (no task)" + description: "Verifies that SendMessage may return a direct agent message response without creating a task." + spec_ref: "docs/specification.md#311-send-message" + level: must + requires_behaviors: + - "tck-message-response" + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-message-response hello" + expect: + status: 200 + body: + message: + role: ROLE_AGENT + task: + absent: true + + - id: CORE-GET-001 + name: "GetTask returns current state of existing task" + description: "Verifies that GetTask returns the current state for an existing completed task." + spec_ref: "docs/specification.md#313-get-task" + level: must + requires_behaviors: + - "tck-complete-task" + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task for get" + capture: + taskId: "task.id" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_COMPLETED + - id: get + operation: get_task + params: + id: "{{send.taskId}}" + expect: + status: 200 + body: + id: "{{send.taskId}}" + status: + state: TASK_STATE_COMPLETED + + - id: CORE-GET-002 + name: "GetTask with historyLength 0 omits history" + description: "Verifies that GetTask omits task history, or returns it as an empty array, when historyLength is zero." + spec_ref: "docs/specification.md#313-get-task" + level: must + requires_behaviors: + - "tck-complete-task" + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task history zero" + capture: + taskId: "task.id" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_COMPLETED + - id: get + operation: get_task + params: + id: "{{send.taskId}}" + historyLength: 0 + expect: + status: 200 + body: + id: "{{send.taskId}}" + history: + any_of: + - absent: true + - type: array + count: 0 + + - id: CORE-CANCEL-001 + name: "CancelTask returns updated state" + description: "Verifies that CancelTask returns the updated canceled task state for a task started in return-immediately mode." + spec_ref: "docs/specification.md#315-cancel-task" + level: must + requires_behaviors: + - "tck-cancel" + steps: + - id: start + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-cancel start" + configuration: + returnImmediately: true + capture: + taskId: "task.id" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: + one_of: + - TASK_STATE_SUBMITTED + - TASK_STATE_WORKING + - id: cancel + operation: cancel_task + params: + id: "{{start.taskId}}" + expect: + status: 200 + body: + id: "{{start.taskId}}" + status: + state: TASK_STATE_CANCELED + + - id: CORE-CANCEL-002 + name: "CancelTask on terminal task returns error" + description: "Verifies that CancelTask on a terminal task fails with TaskNotCancelableError." + spec_ref: "docs/specification.md#315-cancel-task" + level: must + requires_behaviors: + - "tck-complete-task" + steps: + - id: setup + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task setup" + capture: + taskId: "task.id" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_COMPLETED + - id: cancel + operation: cancel_task + params: + id: "{{setup.taskId}}" + expect_error: + error_type: TaskNotCancelableError + message: + type: string + + - id: CORE-FAIL-001 + name: "Task completes with failed state" + description: "Verifies that SendMessage can return a failed task with an explanatory status message." + spec_ref: "docs/specification.md#311-send-message" + level: must + requires_behaviors: + - "tck-task-failure" + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-task-failure trigger error" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_FAILED + message: + exists: true + + - id: CORE-SEND-004 + name: "SendMessage rejects unsupported content type" + description: "Verifies that SendMessage fails with ContentTypeNotSupportedError when a message part uses an unsupported content type." + spec_ref: "docs/specification.md#311-send-message" + level: must + requires_behaviors: + - "tck-complete-task" + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - data: + value: "unsupported" + metadata: + contentType: "application/x-unsupported-type-12345" + expect_error: + code: ContentTypeNotSupportedError + + - id: CORE-LIST-001 + name: "ListTasks returns tasks for a context" + description: "Verifies that ListTasks can return at least one task for an existing context." + spec_ref: "docs/specification.md#314-list-tasks" + level: should + requires_behaviors: + - "tck-complete-task" + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task for list test" + capture: + taskId: "task.id" + contextId: "task.contextId" + expect: + status: 200 + body: + task: + id: + type: string + contextId: + type: string + status: + state: TASK_STATE_COMPLETED + - id: list + operation: list_tasks + params: + contextId: "{{send.contextId}}" + expect: + status: 200 + body: + tasks: + type: array + count_gte: 1 + assertions: + - source: "{{list.response}}" + any: + path: "tasks[*]" + match: + id: "{{send.taskId}}" + contextId: "{{send.contextId}}" + + - id: CORE-LIST-002 + name: "ListTasks with includeArtifacts false omits artifacts" + description: "Verifies that ListTasks omits artifacts, or returns them as null, when includeArtifacts is false." + spec_ref: "REQ-TASK-LIST-001" + level: must + requires_behaviors: + - "tck-complete-task" + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task for list artifacts test" + capture: + taskId: "task.id" + contextId: "task.contextId" + expect: + status: 200 + body: + task: + id: + type: string + contextId: + type: string + status: + state: TASK_STATE_COMPLETED + - id: list + operation: list_tasks + params: + contextId: "{{send.contextId}}" + includeArtifacts: false + expect: + status: 200 + body: + tasks: + type: array + count_gte: 1 + items: + - id: "{{send.taskId}}" + contextId: "{{send.contextId}}" + artifacts: + any_of: + - absent: true + - type: "null" + + - id: CORE-LIST-003 + name: "ListTasks response includes nextPageToken" + description: "Verifies that ListTasks responses always include nextPageToken, using an empty string for the final page." + spec_ref: "REQ-TASK-LIST-002" + level: must + requires_behaviors: + - "tck-complete-task" + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task for list next page token test" + capture: + taskId: "task.id" + contextId: "task.contextId" + expect: + status: 200 + body: + task: + id: + type: string + contextId: + type: string + - id: list + operation: list_tasks + params: + contextId: "{{send.contextId}}" + expect: + status: 200 + body: + tasks: + type: array + count_gte: 1 + nextPageToken: + type: string + assertions: + - source: "{{list.response}}" + any: + path: "tasks[*]" + match: + id: "{{send.taskId}}" + contextId: "{{send.contextId}}" diff --git a/tests/acts/data-types.acts.yaml b/tests/acts/data-types.acts.yaml new file mode 100644 index 000000000..82866663c --- /dev/null +++ b/tests/acts/data-types.acts.yaml @@ -0,0 +1,218 @@ +acts_version: "1.0" +spec_version: "1.0" + +metadata: + title: "Artifact Data Types" + description: "Tests for mixed artifact data types — text, structured data, file, and file URL" + +suites: + - id: data-types + name: "Artifact Data Types" + tests: + - id: DM-ART-001 + name: "Text artifact" + description: > + Sends a request that should produce at least one text artifact and + verifies a returned artifact part includes a text field. + spec_ref: "specification.md#4.1.7" + level: must + requires_behaviors: ["tck-artifact-text"] + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-artifact-text hello" + expect: + status: 200 + body: + task: + status: + state: TASK_STATE_COMPLETED + artifacts: + count_gte: 1 + assertions: + - source: "{{send.response}}" + any: + path: "task.artifacts[*].parts[*]" + match: + text: + type: string + + - id: DM-ART-002 + name: "Data artifact (structured JSON)" + description: > + Sends a request that should produce a structured JSON artifact and + verifies a returned artifact part includes an object-valued data field. + spec_ref: "specification.md#4.1.7" + level: should + requires_behaviors: ["tck-artifact-data"] + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-artifact-data structured" + expect: + status: 200 + body: + task: + status: + state: TASK_STATE_COMPLETED + artifacts: + count_gte: 1 + assertions: + - source: "{{send.response}}" + any: + path: "task.artifacts[*].parts[*]" + match: + data: + type: object + + - id: DM-ART-003 + name: "File artifact (inline)" + description: > + Sends a request that should produce an inline file artifact and + verifies a returned artifact part includes file metadata. + spec_ref: "specification.md#4.1.7" + level: should + requires_behaviors: ["tck-artifact-file"] + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-artifact-file document" + expect: + status: 200 + body: + task: + status: + state: TASK_STATE_COMPLETED + artifacts: + count_gte: 1 + assertions: + - source: "{{send.response}}" + any: + path: "task.artifacts[*].parts[*]" + match: + file: + name: + type: string + mediaType: + type: string + + - id: DM-ART-004 + name: "File URL artifact" + description: > + Sends a request that should produce a file URL artifact and verifies + a returned artifact part includes fileUrl metadata. + spec_ref: "specification.md#4.1.7" + level: may + requires_behaviors: ["tck-artifact-file-url"] + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-artifact-file-url document" + expect: + status: 200 + body: + task: + status: + state: TASK_STATE_COMPLETED + artifacts: + count_gte: 1 + assertions: + - source: "{{send.response}}" + any: + path: "task.artifacts[*].parts[*]" + match: + fileUrl: + url: + type: string + name: + type: string + + - id: DM-SERIAL-001 + name: "Timestamps are ISO 8601 UTC" + level: must + requires_behaviors: ["tck-complete-task"] + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task timestamp" + expect: + status: 200 + body: + task: + status: + state: TASK_STATE_COMPLETED + timestamp: + matches: "^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}" + + - id: DM-SERIAL-002 + name: "Response validates against A2A schema" + description: "Runner SHOULD validate full response against A2A JSON Schema." + level: should + tags: [data-model, schema-validation, runner-special] + requires_behaviors: ["tck-complete-task"] + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task schema" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: + type: string + + - id: DM-EXTRA-001 + name: "Extra fields accepted (tolerance)" + level: should + transport: [jsonrpc] + requires_behaviors: ["tck-complete-task"] + steps: + - id: send-extra-field + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + A2A-Version: "1.0" + body: + jsonrpc: "2.0" + id: 1 + method: "SendMessage" + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task extra field" + x_custom_field: "test" + expect: + status: 200 + body: + result: + exists: true diff --git a/tests/acts/discovery.acts.yaml b/tests/acts/discovery.acts.yaml new file mode 100644 index 000000000..504e174f0 --- /dev/null +++ b/tests/acts/discovery.acts.yaml @@ -0,0 +1,211 @@ +acts_version: "1.0" +spec_version: "1.0" +metadata: + title: "Agent Card Discovery" + description: "Tests for A2A agent card discovery and structure" + +suites: + - id: discovery + name: Agent Card Discovery + tests: + - id: CARD-DISC-001 + name: Agent card is discoverable + description: Agent card is discoverable at the well-known URL and exposes core metadata. + spec_ref: Agent Card Discovery + level: must + requires_behaviors: [] + steps: + - id: step-1 + operation: get_agent_card + params: {} + expect: + status: 200 + body: + name: + exists: true + version: + exists: true + description: + exists: true + + - id: CARD-DISC-002 + name: Agent card contains supported interfaces + description: Agent card lists one or more supported interfaces with binding and version metadata. + spec_ref: Agent Card Discovery + level: must + requires_behaviors: [] + steps: + - id: step-1 + operation: get_agent_card + params: {} + expect: + status: 200 + body: + supportedInterfaces: + type: array + count_gte: 1 + items: + url: + type: string + protocolBinding: + one_of: + - JSONRPC + - GRPC + - REST + protocolVersion: + type: string + + - id: CARD-DISC-003 + name: Agent card contains capabilities + description: Agent card exposes capabilities as an object. + spec_ref: Agent Card Discovery + level: must + requires_behaviors: [] + steps: + - id: step-1 + operation: get_agent_card + params: {} + expect: + status: 200 + body: + capabilities: + exists: true + type: object + + - id: CARD-DISC-004 + name: Agent card contains skills + description: Agent card includes skills with identifiers and names. + spec_ref: Agent Card Discovery + level: should + requires_behaviors: [] + steps: + - id: step-1 + operation: get_agent_card + params: {} + expect: + status: 200 + body: + skills: + type: array + items: + id: + exists: true + name: + exists: true + + - id: CARD-DISC-005 + name: Agent card contains default input and output modes + description: Agent card includes default input and output mode arrays. + spec_ref: Agent Card Discovery + level: should + requires_behaviors: [] + steps: + - id: step-1 + operation: get_agent_card + params: {} + expect: + status: 200 + body: + defaultInputModes: + type: array + defaultOutputModes: + type: array + + - id: CARD-DISC-006 + name: Agent card protocolVersion matches spec version + description: The first supported interface advertises an A2A 1.x protocol version. + spec_ref: Agent Card Discovery + level: must + requires_behaviors: [] + steps: + - id: step-1 + operation: get_agent_card + params: {} + expect: + status: 200 + body: + supportedInterfaces: + type: array + count_gte: 1 + items: + - protocolVersion: + starts_with: "1." + + - id: CARD-CACHE-001 + name: Agent card caching headers + description: Runner SHOULD verify Cache-Control or ETag headers are present. + spec_ref: Agent Card Discovery + level: should + tags: [discovery, caching, runner-special] + requires_behaviors: [] + steps: + - id: step-1 + operation: get_agent_card + params: {} + expect: + status: 200 + body: + name: + exists: true + + - id: CARD-SCHEMA-001 + name: Agent card validates against schema + description: Runner MUST validate response against A2A AgentCard JSON Schema. + spec_ref: Agent Card Discovery + level: should + tags: [discovery, schema-validation] + requires_behaviors: [] + steps: + - id: step-1 + operation: get_agent_card + params: {} + expect: + status: 200 + body: + name: + exists: true + capabilities: + type: object + + - id: CARD-EXT-001 + name: Extended agent card + description: Retrieves the extended agent card when the capability is advertised. + spec_ref: Get Extended Agent Card + level: may + preconditions: + capabilities: + extendedAgentCard: true + requires_behaviors: [] + steps: + - id: step-1 + operation: get_agent_card + params: + extended: true + expect: + status: 200 + body: + skills: + type: array + count_gte: 1 + description: + matches: ".+" + + - id: CARD-DISC-007 + name: Agent card declares all supported protocols + description: Verifies that the public Agent Card declares at least one protocol endpoint through supportedInterfaces or a direct url field. + spec_ref: REQ-BIND-002 + level: must + requires_behaviors: [] + steps: + - id: step-1 + operation: get_agent_card + params: {} + expect: + status: 200 + body: + any_of: + - supportedInterfaces: + type: array + count_gte: 1 + - url: + type: string diff --git a/tests/acts/error-handling.acts.yaml b/tests/acts/error-handling.acts.yaml new file mode 100644 index 000000000..91c46b69b --- /dev/null +++ b/tests/acts/error-handling.acts.yaml @@ -0,0 +1,347 @@ +acts_version: "1.0" +spec_version: "1.0" + +metadata: + title: "A2A Error Handling Conformance Tests" + description: "Error handling coverage derived from TCK CORE-ERR, agentbin error cases, and CSIT." + +suites: + - id: error-handling + name: "Error Handling" + tests: + - id: CORE-ERR-001 + name: "GetTask with non-existent ID returns TaskNotFoundError" + level: must + tags: [error, get-task, task-not-found] + steps: + - id: get-missing + operation: get_task + params: + id: "00000000-0000-0000-0000-000000000000" + expect_error: + code: TaskNotFoundError + message: + type: string + + - id: CORE-ERR-002 + name: "CancelTask on non-existent task returns error" + level: must + tags: [error, cancel-task] + steps: + - id: cancel-missing + operation: cancel_task + params: + id: "00000000-0000-0000-0000-000000000000" + expect_error: + code: + one_of: [TaskNotFoundError, TaskNotCancelableError] + + - id: CORE-ERR-003 + name: "CancelTask on completed task returns TaskNotCancelableError" + level: must + tags: [error, cancel-task, completed-task] + requires_behaviors: [tck-complete-task] + steps: + - id: setup-completed-task + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task setup" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_COMPLETED + capture: + taskId: "task.id" + + - id: cancel-completed-task + operation: cancel_task + params: + id: "{{setup-completed-task.taskId}}" + expect_error: + code: TaskNotCancelableError + + - id: CORE-ERR-004 + name: "SendMessage to terminal task returns UnsupportedOperationError" + level: must + tags: [error, send-message, terminal-task] + requires_behaviors: [tck-complete-task] + steps: + - id: create-terminal-task + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task first" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_COMPLETED + capture: + taskId: "task.id" + + - id: resend-terminal-task + operation: send_message + params: + message: + role: ROLE_USER + taskId: "{{create-terminal-task.taskId}}" + parts: + - text: "tck-complete-task second" + expect_error: + code: UnsupportedOperationError + + - id: JSONRPC-ERR-001 + name: "Unknown method returns MethodNotFoundError" + level: must + tags: [error, jsonrpc, raw] + transport: [jsonrpc] + steps: + - id: bad-method + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + A2A-Version: "1.0" + body: + jsonrpc: "2.0" + id: 1 + method: "DoSomethingUnsupported" + params: {} + expect: + status: 200 + body: + error: + code: -32601 + message: + type: string + + - id: JSONRPC-ERR-002 + name: "Invalid JSON returns ParseError" + level: must + tags: [error, jsonrpc, raw, parse] + transport: [jsonrpc] + steps: + - id: bad-json + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + A2A-Version: "1.0" + body_raw: "{this is not valid json" + expect: + status: 200 + body: + error: + code: -32700 + + - id: CORE-ERR-005 + name: "Error responses include actionable message" + level: should + tags: [error, message-quality] + steps: + - id: get-missing-actionable + operation: get_task + params: + id: "00000000-0000-0000-0000-000000000000" + expect_error: + code: TaskNotFoundError + message: + all_of: + - type: string + - matches: ".{12,}" + - any_of: + - contains: "task" + - contains: "Task" + - contains: "not found" + - contains: "does not exist" + + - id: CORE-ERR-006 + name: "Malformed request returns InvalidRequestError" + level: must + transport: [jsonrpc] + steps: + - id: malformed-jsonrpc + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + A2A-Version: "1.0" + body: + jsonrpc: "2.0" + id: 1 + params: {} + expect: + status: 200 + body: + error: + code: + one_of: [-32600, -32602] + + - id: CORE-CAP-001 + name: "Push not supported returns UnsupportedOperationError" + description: "Overlaps with PUSH-CFG-004 but verifies the capability error at the error-handling suite level." + level: must + preconditions: + capabilities: + pushNotifications: false + requires_behaviors: [tck-complete-task] + steps: + - id: create-task + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task for push" + expect: + status: 200 + body: + task: + id: + type: string + capture: + taskId: "task.id" + - id: set-config + operation: set_push_notification_config + params: + taskId: "{{create-task.taskId}}" + pushNotificationConfig: + url: "https://example.com/webhooks/a2a-tests" + expect_error: + code: UnsupportedOperationError + + - id: CORE-CAP-002 + name: "Streaming not supported returns error" + level: should + preconditions: + capabilities: + streaming: false + requires_behaviors: [tck-complete-task] + steps: + - id: stream + operation: send_streaming_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task streaming unsupported" + expect_error: + code: UnsupportedOperationError + + - id: CORE-ERR-007 + name: "Error data contains structured error info" + level: should + steps: + - id: get-missing-structured + operation: get_task + params: + id: "00000000-0000-0000-0000-000000000000" + expect_error: + code: TaskNotFoundError + data: + type: object + + - id: CORE-ERR-008 + name: "Error details include @type field" + description: "Verifies that JSON-RPC error details include a google.rpc Any type discriminator." + spec_ref: "REQ-ERR-009" + level: must + transport: [jsonrpc] + steps: + - id: send-missing-task + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + A2A-Version: "1.0" + body: + jsonrpc: "2.0" + id: "core-err-008" + method: SendMessage + params: + taskId: "00000000-0000-0000-0000-000000000000" + message: + role: ROLE_USER + parts: + - text: "missing task should return error details" + expect: + status: 200 + body: + error: + message: + type: string + data: + type: array + count_gte: 1 + items: + - "@type": + type: string + + - id: CORE-ERR-009 + name: "Error response should not distinguish missing vs unauthorized" + description: "Runner SHOULD verify that the error response for a missing resource is indistinguishable from one for an unauthorized resource. This requires auth infrastructure." + spec_ref: "REQ-ERR-007" + level: should + tags: [security, error-handling, runner-special] + steps: + - id: get-missing + operation: get_task + params: + id: "00000000-0000-0000-0000-000000000000" + expect_error: + message: + type: string + + - id: JSONRPC-ERR-003 + name: "JSON-RPC error details include google.rpc.ErrorInfo" + description: "Verifies that A2A-specific JSON-RPC errors include google.rpc.ErrorInfo in error.data." + spec_ref: "REQ-RPC-006" + level: must + tags: [error, jsonrpc, raw] + transport: [jsonrpc] + steps: + - id: get-missing-error-info + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + A2A-Version: "1.0" + body: + jsonrpc: "2.0" + id: "jsonrpc-err-003" + method: GetTask + params: + id: "00000000-0000-0000-0000-000000000000" + expect: + status: 200 + body: + error: + message: + type: string + data: + type: array + count_gte: 1 + assertions: + - source: "{{get-missing-error-info.response}}" + any: + path: "error.data[*]" + match: + "@type": + contains: "ErrorInfo" diff --git a/tests/acts/history.acts.yaml b/tests/acts/history.acts.yaml new file mode 100644 index 000000000..edf5093ad --- /dev/null +++ b/tests/acts/history.acts.yaml @@ -0,0 +1,360 @@ +acts_version: "1.0" +spec_version: "1.0" +spec_ref: "docs/specification.md" + +metadata: + title: "Task History ACTS" + description: "ACTS tests for historyLength behavior and persisted task history semantics." + +suites: + - id: history + name: "Task History" + description: "Coverage for GetTask and SendMessage history handling." + tests: + - id: CORE-HIST-001 + name: "GetTask historyLength=0 returns no history" + description: "Verifies that GetTask omits history, or returns it as an empty array, when historyLength is zero." + spec_ref: "docs/specification.md#313-get-task" + level: must + requires_behaviors: + - "tck-complete-task" + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task history zero" + capture: + taskId: "task.id" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_COMPLETED + - id: get + operation: get_task + params: + id: "{{send.taskId}}" + historyLength: 0 + expect: + status: 200 + body: + id: "{{send.taskId}}" + history: + any_of: + - absent: true + - type: array + count: 0 + + - id: CORE-HIST-002 + name: "History count does not exceed historyLength limit" + description: "Verifies that GetTask does not return more history entries than requested." + spec_ref: "docs/specification.md#313-get-task" + level: should + requires_behaviors: + - "tck-multi-turn" + steps: + - id: turn1 + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-multi-turn history limit first" + capture: + taskId: "task.id" + contextId: "task.contextId" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_INPUT_REQUIRED + - id: turn2 + operation: send_message + params: + taskId: "{{turn1.taskId}}" + contextId: "{{turn1.contextId}}" + message: + role: ROLE_USER + parts: + - text: "tck-multi-turn history limit second" + expect: + status: 200 + body: + task: + id: "{{turn1.taskId}}" + status: + state: TASK_STATE_INPUT_REQUIRED + - id: turn3 + operation: send_message + params: + taskId: "{{turn1.taskId}}" + contextId: "{{turn1.contextId}}" + message: + role: ROLE_USER + parts: + - text: "done" + expect: + status: 200 + body: + task: + id: "{{turn1.taskId}}" + status: + state: TASK_STATE_COMPLETED + - id: get + operation: get_task + params: + id: "{{turn1.taskId}}" + historyLength: 2 + expect: + status: 200 + body: + id: "{{turn1.taskId}}" + history: + any_of: + - absent: true + - type: array + count_lte: 2 + + - id: CORE-HIST-003 + name: "SendMessage historyLength=0 returns no history" + description: "Verifies that SendMessage omits task history when configuration.historyLength is zero." + spec_ref: "docs/specification.md#311-send-message" + level: should + requires_behaviors: + - "tck-complete-task" + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task send history zero" + configuration: + historyLength: 0 + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_COMPLETED + history: + absent: true + + - id: CORE-HIST-004 + name: "GetTask may return persisted history" + description: "Verifies that GetTask may omit history entirely, but if it returns history the field is an array." + spec_ref: "docs/specification.md#313-get-task" + level: may + requires_behaviors: + - "tck-complete-task" + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task persisted history" + capture: + taskId: "task.id" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_COMPLETED + - id: get + operation: get_task + params: + id: "{{send.taskId}}" + expect: + status: 200 + body: + id: "{{send.taskId}}" + history: + any_of: + - absent: true + - type: array + + - id: CORE-HIST-005 + name: "History messages are returned in chronological order" + description: "Verifies that persisted history preserves the order of the conversation turns." + spec_ref: "docs/specification.md#313-get-task" + level: should + requires_behaviors: + - "tck-multi-turn" + steps: + - id: turn1 + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-multi-turn chronological first" + capture: + taskId: "task.id" + contextId: "task.contextId" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_INPUT_REQUIRED + - id: turn2 + operation: send_message + params: + taskId: "{{turn1.taskId}}" + contextId: "{{turn1.contextId}}" + message: + role: ROLE_USER + parts: + - text: "tck-multi-turn chronological second" + expect: + status: 200 + body: + task: + id: "{{turn1.taskId}}" + status: + state: TASK_STATE_INPUT_REQUIRED + - id: turn3 + operation: send_message + params: + taskId: "{{turn1.taskId}}" + contextId: "{{turn1.contextId}}" + message: + role: ROLE_USER + parts: + - text: "done" + expect: + status: 200 + body: + task: + id: "{{turn1.taskId}}" + status: + state: TASK_STATE_COMPLETED + - id: get + operation: get_task + params: + id: "{{turn1.taskId}}" + expect: + status: 200 + body: + id: "{{turn1.taskId}}" + history: + type: array + count_gte: 3 + assertions: + - source: "{{get.response}}" + any: + path: "history[0].parts[*]" + match: + text: + contains: "tck-multi-turn chronological first" + - source: "{{get.response}}" + any: + path: "history[2].parts[*]" + match: + text: + contains: "done" + + - id: CORE-HIST-006 + name: "History content matches sent messages" + description: "Verifies that persisted history includes the messages sent during a multi-turn conversation." + spec_ref: "docs/specification.md#313-get-task" + level: should + requires_behaviors: + - "tck-multi-turn" + steps: + - id: turn1 + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-multi-turn first msg" + capture: + taskId: "task.id" + contextId: "task.contextId" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_INPUT_REQUIRED + - id: turn2 + operation: send_message + params: + taskId: "{{turn1.taskId}}" + contextId: "{{turn1.contextId}}" + message: + role: ROLE_USER + parts: + - text: "tck-multi-turn follow up msg" + expect: + status: 200 + body: + task: + id: "{{turn1.taskId}}" + status: + state: + one_of: + - TASK_STATE_INPUT_REQUIRED + - TASK_STATE_COMPLETED + - id: turn3 + operation: send_message + params: + taskId: "{{turn1.taskId}}" + contextId: "{{turn1.contextId}}" + message: + role: ROLE_USER + parts: + - text: "done" + expect: + status: 200 + body: + task: + id: "{{turn1.taskId}}" + status: + state: TASK_STATE_COMPLETED + - id: get + operation: get_task + params: + id: "{{turn1.taskId}}" + expect: + status: 200 + body: + id: "{{turn1.taskId}}" + history: + type: array + count_gte: 2 + assertions: + - source: "{{get.response}}" + any: + path: "history[*].parts[*]" + match: + text: + contains: "tck-multi-turn first msg" + - source: "{{get.response}}" + any: + path: "history[*].parts[*]" + match: + text: + contains: "tck-multi-turn follow up msg" diff --git a/tests/acts/multi-turn.acts.yaml b/tests/acts/multi-turn.acts.yaml new file mode 100644 index 000000000..27b46a486 --- /dev/null +++ b/tests/acts/multi-turn.acts.yaml @@ -0,0 +1,240 @@ +acts_version: "1.0" +spec_version: "1.0" +metadata: + title: "Multi-turn Conversations" + description: "ACTS tests for multi-turn conversations, INPUT_REQUIRED flows, and context inference" + +suites: + - id: multi-turn + name: Multi-turn Conversations + tests: + - id: CORE-MULTI-001 + name: Multi-turn with INPUT_REQUIRED + description: Covers TCK CORE-MULTI-001, CSIT scenario-parity input-required, and agentbin spec-multi-turn behavior. + spec_ref: CORE-MULTI-001 + level: must + requires_behaviors: + - tck-multi-turn + steps: + - id: turn1 + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-multi-turn start conversation" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_INPUT_REQUIRED + capture: + taskId: "task.id" + contextId: "task.contextId" + + - id: turn2 + operation: send_message + params: + taskId: "{{turn1.taskId}}" + contextId: "{{turn1.contextId}}" + message: + role: ROLE_USER + parts: + - text: "here is more input" + expect: + status: 200 + body: + task: + id: "{{turn1.taskId}}" + status: + state: TASK_STATE_INPUT_REQUIRED + + - id: turn3 + operation: send_message + params: + taskId: "{{turn1.taskId}}" + contextId: "{{turn1.contextId}}" + message: + role: ROLE_USER + parts: + - text: "done" + expect: + status: 200 + body: + task: + id: "{{turn1.taskId}}" + status: + state: TASK_STATE_COMPLETED + + - id: CORE-MULTI-005 + name: SendMessage with only taskId infers contextId + description: Verifies follow-up SendMessage can omit contextId and still remain on the original task context. + spec_ref: CORE-MULTI-005 + level: must + requires_behaviors: + - tck-multi-turn + steps: + - id: turn1 + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-multi-turn start" + expect: + status: 200 + body: + task: + id: + type: string + contextId: + type: string + status: + state: TASK_STATE_INPUT_REQUIRED + capture: + taskId: "task.id" + contextId: "task.contextId" + + - id: turn2 + operation: send_message + params: + taskId: "{{turn1.taskId}}" + message: + role: ROLE_USER + parts: + - text: "done" + expect: + status: 200 + body: + task: + id: "{{turn1.taskId}}" + contextId: "{{turn1.contextId}}" + status: + state: TASK_STATE_COMPLETED + + - id: CORE-MULTI-006 + name: SendMessage with taskId and wrong contextId returns error + description: Verifies mismatched contextId is rejected for a follow-up turn. + spec_ref: CORE-MULTI-006 + level: must + requires_behaviors: + - tck-multi-turn + steps: + - id: turn1 + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-multi-turn start" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_INPUT_REQUIRED + capture: + taskId: "task.id" + + - id: turn2 + operation: send_message + params: + taskId: "{{turn1.taskId}}" + contextId: "wrong-context-id-12345" + message: + role: ROLE_USER + parts: + - text: "should fail" + expect: + status: error + error: + exists: true + + - id: CORE-MULTI-003 + name: Mismatched contextId and taskId is rejected + description: Verifies that SendMessage rejects a taskId and contextId pair from different conversations. + spec_ref: REQ-MULTI-002 + level: must + requires_behaviors: + - tck-complete-task + steps: + - id: first + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task first" + expect: + status: 200 + body: + task: + id: + type: string + contextId: + type: string + status: + state: TASK_STATE_COMPLETED + capture: + taskId1: "task.id" + contextId1: "task.contextId" + + - id: second + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task second" + expect: + status: 200 + body: + task: + id: + type: string + contextId: + type: string + status: + state: TASK_STATE_COMPLETED + capture: + taskId2: "task.id" + contextId2: "task.contextId" + + - id: mismatch + operation: send_message + params: + taskId: "{{first.taskId1}}" + contextId: "{{second.contextId2}}" + message: + role: ROLE_USER + parts: + - text: "mismatched context and task" + expect_error: + message: + type: string + + - id: CORE-CTX-001 + name: Rejected client contextId returns an error + description: Runner SHOULD use a client-generated contextId that the server rejects and verify the server errors instead of minting a new contextId. + spec_ref: REQ-CTX-002 + level: must + tags: [context, identity, runner-special] + requires_behaviors: + - tck-complete-task + steps: + - id: send + operation: send_message + params: + contextId: "client-generated-context-that-should-be-rejected" + message: + role: ROLE_USER + parts: + - text: "tck-complete-task reject client context id" + expect_error: + message: + type: string diff --git a/tests/acts/polling.acts.yaml b/tests/acts/polling.acts.yaml new file mode 100644 index 000000000..e84221966 --- /dev/null +++ b/tests/acts/polling.acts.yaml @@ -0,0 +1,140 @@ +acts_version: "1.0" +spec_version: "1.0" + +metadata: + title: "Polling and Long-Running Tasks" + description: "Tests for long-running tasks with returnImmediately and polling patterns" + +suites: + - id: polling + name: "Polling and Long-Running Tasks" + tests: + - id: CORE-EXEC-001 + name: "Long-running task with returnImmediately and polling" + description: > + Sends a long-running request with returnImmediately enabled, polls + until the task reaches a terminal state, and verifies the task + completes with at least one artifact. + spec_ref: "specification.md#3.1.1" + level: must + requires_behaviors: ["tck-long-running"] + steps: + - id: start + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-long-running start process" + configuration: + returnImmediately: true + capture: + taskId: "task.id" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: + one_of: [TASK_STATE_SUBMITTED, TASK_STATE_WORKING] + + - id: poll + operation: get_task + params: + id: "{{start.taskId}}" + expect: + status: 200 + body: + status: + state: + type: string + repeat: + until: "status.state in [TASK_STATE_COMPLETED, TASK_STATE_FAILED]" + max_attempts: 15 + delay_ms: 2000 + + - id: verify + operation: get_task + params: + id: "{{start.taskId}}" + expect: + status: 200 + body: + id: "{{start.taskId}}" + status: + state: TASK_STATE_COMPLETED + artifacts: + type: array + count_gte: 1 + + - id: CORE-EXEC-002 + name: "Polling shows progress through working state" + description: > + Sends a long-running request with returnImmediately enabled, verifies + polling observes TASK_STATE_WORKING during execution, and then confirms + the task reaches TASK_STATE_COMPLETED. + spec_ref: "specification.md#3.1.1" + level: should + requires_behaviors: ["tck-long-running"] + steps: + - id: start + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-long-running start process" + configuration: + returnImmediately: true + capture: + taskId: "task.id" + expect: + status: 200 + body: + task: + status: + state: + one_of: [TASK_STATE_SUBMITTED, TASK_STATE_WORKING] + + - id: wait-for-working + operation: get_task + params: + id: "{{start.taskId}}" + expect: + status: 200 + body: + status: + state: + type: string + repeat: + until: "status.state == TASK_STATE_WORKING" + max_attempts: 10 + delay_ms: 1000 + + - id: poll-to-terminal + operation: get_task + params: + id: "{{start.taskId}}" + expect: + status: 200 + body: + status: + state: + type: string + repeat: + until: "status.state in [TASK_STATE_COMPLETED, TASK_STATE_FAILED]" + max_attempts: 15 + delay_ms: 2000 + + - id: verify + operation: get_task + params: + id: "{{start.taskId}}" + expect: + status: 200 + body: + id: "{{start.taskId}}" + status: + state: TASK_STATE_COMPLETED diff --git a/tests/acts/push-notifications.acts.yaml b/tests/acts/push-notifications.acts.yaml new file mode 100644 index 000000000..b1fa3d19d --- /dev/null +++ b/tests/acts/push-notifications.acts.yaml @@ -0,0 +1,454 @@ +acts_version: "1.0" +spec_version: "1.0" + +metadata: + title: "Push Notification Configuration" + description: "Tests for push notification config CRUD and unsupported-server behavior" + +variables: + webhookUrl: "https://example.com/webhooks/a2a-tests" + +suites: + - id: push-notifications + name: "Push Notification Configuration" + tests: + - id: PUSH-CFG-001 + name: "Set push notification config" + description: > + Creates a task, configures push notifications for it, and verifies the + returned configuration uses the requested webhook URL. + spec_ref: "specification.md#3.1.7" + level: may + preconditions: + capabilities: + pushNotifications: true + requires_behaviors: ["tck-complete-task"] + steps: + - id: create + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task for push" + capture: + taskId: "task.id" + expect: + status: 200 + body: + task: + id: + type: string + + - id: set-config + operation: set_push_notification_config + params: + taskId: "{{create.taskId}}" + pushNotificationConfig: + url: "{{webhookUrl}}" + expect: + status: 200 + body: + id: + type: string + url: "{{webhookUrl}}" + capture: + configId: "id" + + - id: PUSH-CFG-002 + name: "Get push notification config" + description: > + Creates a task, stores a push notification configuration, retrieves it, + and verifies the previously set configuration is returned. + spec_ref: "specification.md#3.1.8" + level: may + preconditions: + capabilities: + pushNotifications: true + requires_behaviors: ["tck-complete-task"] + steps: + - id: create + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task for push" + capture: + taskId: "task.id" + expect: + status: 200 + body: + task: + id: + type: string + + - id: set-config + operation: set_push_notification_config + params: + taskId: "{{create.taskId}}" + pushNotificationConfig: + url: "{{webhookUrl}}" + capture: + configId: "id" + expect: + status: 200 + body: + id: + type: string + + - id: get-config + operation: get_push_notification_config + params: + taskId: "{{create.taskId}}" + id: "{{set-config.configId}}" + expect: + status: 200 + body: + id: "{{set-config.configId}}" + url: "{{webhookUrl}}" + + - id: PUSH-CFG-003 + name: "Delete push notification config" + description: > + Creates a task, stores a push notification configuration, deletes it, + and expects a successful deletion response. + spec_ref: "specification.md#3.1.10" + level: may + preconditions: + capabilities: + pushNotifications: true + requires_behaviors: ["tck-complete-task"] + steps: + - id: create + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task for push" + capture: + taskId: "task.id" + expect: + status: 200 + body: + task: + id: + type: string + + - id: set-config + operation: set_push_notification_config + params: + taskId: "{{create.taskId}}" + pushNotificationConfig: + url: "{{webhookUrl}}" + capture: + configId: "id" + expect: + status: 200 + body: + id: + type: string + + - id: delete-config + operation: delete_push_notification_config + params: + taskId: "{{create.taskId}}" + id: "{{set-config.configId}}" + expect: + status: 200 + + - id: PUSH-CFG-004 + name: "Push config on unsupported server returns error" + description: > + Creates a task on a server that does not advertise push notification + support, then verifies setting a push notification configuration fails. + spec_ref: "specification.md#3.1.7" + level: may + preconditions: + capabilities: + pushNotifications: false + requires_behaviors: ["tck-complete-task"] + steps: + - id: create + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task for push" + capture: + taskId: "task.id" + expect: + status: 200 + body: + task: + id: + type: string + + - id: set-config + operation: set_push_notification_config + params: + taskId: "{{create.taskId}}" + pushNotificationConfig: + url: "{{webhookUrl}}" + expect_error: + code: UnsupportedOperationError + + - id: PUSH-LIST-001 + name: "List push configs" + level: may + preconditions: + capabilities: + pushNotifications: true + requires_behaviors: ["tck-complete-task"] + steps: + - id: create + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task for push" + expect: + status: 200 + body: + task: + id: + type: string + capture: + taskId: "task.id" + + - id: set-config + operation: set_push_notification_config + params: + taskId: "{{create.taskId}}" + pushNotificationConfig: + url: "{{webhookUrl}}" + expect: + status: 200 + body: + id: + type: string + capture: + configId: "id" + + - id: list-configs + operation: list_push_notification_configs + params: + taskId: "{{create.taskId}}" + expect: + status: 200 + body: + configs: + type: array + count_gte: 1 + + - id: PUSH-ERR-001 + name: "Get nonexistent push config returns error" + level: may + preconditions: + capabilities: + pushNotifications: true + requires_behaviors: ["tck-complete-task"] + steps: + - id: create + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task for push" + expect: + status: 200 + body: + task: + id: + type: string + capture: + taskId: "task.id" + + - id: get-config + operation: get_push_notification_config + params: + taskId: "{{create.taskId}}" + id: "00000000-0000-0000-0000-000000000000" + expect_error: + code: TaskNotFoundError + + - id: PUSH-IDEM-001 + name: "Delete push config is idempotent" + level: may + preconditions: + capabilities: + pushNotifications: true + requires_behaviors: ["tck-complete-task"] + steps: + - id: create + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task for push" + expect: + status: 200 + body: + task: + id: + type: string + capture: + taskId: "task.id" + + - id: set-config + operation: set_push_notification_config + params: + taskId: "{{create.taskId}}" + pushNotificationConfig: + url: "{{webhookUrl}}" + expect: + status: 200 + body: + id: + type: string + capture: + configId: "id" + + - id: delete-config + operation: delete_push_notification_config + params: + taskId: "{{create.taskId}}" + id: "{{set-config.configId}}" + expect: + status: 200 + + - id: delete-config-again + operation: delete_push_notification_config + params: + taskId: "{{create.taskId}}" + id: "{{set-config.configId}}" + expect: + status: 200 + + - id: PUSH-DELIV-001 + name: "Push delivery includes auth" + description: "Runner MUST set up a webhook endpoint and verify auth credentials are included." + level: may + tags: [push, delivery, runner-special] + preconditions: + capabilities: + pushNotifications: true + requires_behaviors: ["tck-long-running"] + steps: + - id: create + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-long-running start" + configuration: + returnImmediately: true + expect: + status: 200 + body: + task: + id: + type: string + capture: + taskId: "task.id" + + - id: set-config + operation: set_push_notification_config + params: + taskId: "{{create.taskId}}" + pushNotificationConfig: + url: "{{webhookUrl}}" + authentication: + scheme: "Bearer" + credentials: "test-token" + expect: + status: 200 + body: + id: + type: string + + - id: PUSH-DELIV-002 + name: "Push delivery is at-least-once" + description: "Runner MUST set up a webhook endpoint and verify at-least-once delivery semantics." + level: may + tags: [push, delivery, runner-special] + preconditions: + capabilities: + pushNotifications: true + requires_behaviors: ["tck-long-running"] + steps: + - id: create + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-long-running start" + configuration: + returnImmediately: true + expect: + status: 200 + body: + task: + id: + type: string + capture: + taskId: "task.id" + + - id: set-config + operation: set_push_notification_config + params: + taskId: "{{create.taskId}}" + pushNotificationConfig: + url: "{{webhookUrl}}" + expect: + status: 200 + body: + id: + type: string + + - id: PUSH-DELIV-003 + name: "Push payload format correct" + description: "Runner MUST set up a webhook endpoint and verify the delivered payload format matches the A2A specification." + level: may + tags: [push, delivery, runner-special] + preconditions: + capabilities: + pushNotifications: true + requires_behaviors: ["tck-long-running"] + steps: + - id: create + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-long-running start" + configuration: + returnImmediately: true + expect: + status: 200 + body: + task: + id: + type: string + capture: + taskId: "task.id" + + - id: set-config + operation: set_push_notification_config + params: + taskId: "{{create.taskId}}" + pushNotificationConfig: + url: "{{webhookUrl}}" + expect: + status: 200 + body: + id: + type: string diff --git a/tests/acts/streaming.acts.yaml b/tests/acts/streaming.acts.yaml new file mode 100644 index 000000000..0a03b8834 --- /dev/null +++ b/tests/acts/streaming.acts.yaml @@ -0,0 +1,301 @@ +acts_version: "1.0" +spec_version: "1.0" + +metadata: + title: "Streaming operations ACTS suite" + description: "Streaming conformance scenarios derived from TCK STREAM-*, CSIT task-streaming, and agentbin spec-streaming." + +suites: + - id: streaming + name: "Streaming operations" + description: "Covers send_streaming_message and subscribe_to_task behaviors." + tags: + - streaming + - sse + - subscribe + tests: + - id: STREAM-SSE-001 + name: "SendStreamingMessage produces events ending in terminal state" + level: must + requires_behaviors: + - tck-stream-basic + steps: + - id: stream + operation: send_streaming_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-stream-basic generate output" + expect_stream: + min_count: 2 + ordering: monotonic_state + final_event: + status: + state: + one_of: + - TASK_STATE_COMPLETED + - TASK_STATE_FAILED + + - id: STREAM-SSE-002 + name: "Stream includes at least one artifact update" + level: must + requires_behaviors: + - tck-stream-basic + steps: + - id: stream + operation: send_streaming_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-stream-basic with artifact" + expect_stream: + events: + - description: "At least one streamed event includes an artifact payload" + match: any_position + artifact: + exists: true + final_event: + status: + state: TASK_STATE_COMPLETED + + - id: STREAM-SSE-003 + name: "Chunked streaming delivers multiple artifact events" + level: should + requires_behaviors: + - tck-stream-chunked + steps: + - id: stream + operation: send_streaming_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-stream-chunked multi-part" + expect_stream: + min_count: 3 + ordering: monotonic_state + + - id: STREAM-SUB-001 + name: "SubscribeToTask receives updates for existing task" + level: should + requires_behaviors: + - tck-long-running + steps: + - id: start + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-long-running start" + configuration: + returnImmediately: true + expect: + task: + id: + type: string + status: + state: + one_of: + - TASK_STATE_SUBMITTED + - TASK_STATE_WORKING + capture: + taskId: task.id + - id: subscribe + operation: subscribe_to_task + params: + id: "{{start.taskId}}" + expect_stream: + final_event: + status: + state: + one_of: + - TASK_STATE_COMPLETED + - TASK_STATE_FAILED + - TASK_STATE_CANCELED + - TASK_STATE_REJECTED + + - id: STREAM-SUB-002 + name: "Subscribe stream terminates at terminal state" + description: "The stream must close after emitting the terminal state event." + level: must + requires_behaviors: + - tck-stream-basic + steps: + - id: stream + operation: send_streaming_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-stream-basic test" + expect_stream: + final_event: + status: + state: + one_of: + - TASK_STATE_COMPLETED + - TASK_STATE_FAILED + - TASK_STATE_CANCELED + + - id: STREAM-SUB-003 + name: "Subscribe to terminal task returns error" + level: must + requires_behaviors: + - tck-complete-task + steps: + - id: setup + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task setup" + expect: + task: + id: + type: string + status: + state: + one_of: + - TASK_STATE_COMPLETED + - TASK_STATE_FAILED + - TASK_STATE_CANCELED + - TASK_STATE_REJECTED + capture: + taskId: task.id + - id: subscribe + operation: subscribe_to_task + params: + id: "{{setup.taskId}}" + expect: + error: + exists: true + + - id: STREAM-SSE-004 + name: "First event on subscribe is current task state" + level: must + requires_behaviors: + - tck-long-running + steps: + - id: start + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-long-running start" + configuration: + returnImmediately: true + expect: + status: 200 + body: + task: + id: + type: string + capture: + taskId: "task.id" + - id: subscribe + operation: subscribe_to_task + params: + id: "{{start.taskId}}" + expect_stream: + events: + - description: "First event represents the current task state" + index: 0 + one_of: + - task: + status: + exists: true + - status_update: + status: + exists: true + + - id: STREAM-MSG-001 + name: "Streaming message-only response" + level: should + requires_behaviors: + - tck-message-response + steps: + - id: stream + operation: send_streaming_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-message-response streaming" + expect_stream: + min_count: 1 + final_event: + message: + exists: true + + - id: STREAM-MULTI-001 + name: "Multiple concurrent streams receive same events" + description: "Runner MUST open two concurrent streams and verify both receive the same events. This test may require special runner support." + level: should + tags: [streaming, concurrent, runner-special] + requires_behaviors: + - tck-stream-basic + steps: + - id: stream + operation: send_streaming_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-stream-basic generate output" + expect_stream: + min_count: 2 + + - id: STREAM-MULTI-002 + name: "Closing one stream doesn't affect others" + description: "Runner MUST open two concurrent streams, close one of them mid-stream, and verify the other continues receiving events. This test may require special runner support." + level: should + tags: [streaming, concurrent, runner-special] + requires_behaviors: + - tck-stream-basic + steps: + - id: stream + operation: send_streaming_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-stream-basic generate output" + expect_stream: + min_count: 2 + + - id: STREAM-RESUB-001 + name: "Resubscribe after disconnect" + description: "Runner MUST disconnect from stream mid-flight and resubscribe. This test may require special runner support." + level: should + tags: [streaming, resubscribe, runner-special] + requires_behaviors: + - tck-long-running + steps: + - id: start + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-long-running start" + configuration: + returnImmediately: true + expect: + status: 200 + body: + task: + id: + type: string + capture: + taskId: "task.id" + - id: subscribe + operation: subscribe_to_task + params: + id: "{{start.taskId}}" + expect_stream: + min_count: 1 diff --git a/tests/acts/suite.acts.yaml b/tests/acts/suite.acts.yaml new file mode 100644 index 000000000..6af3f8363 --- /dev/null +++ b/tests/acts/suite.acts.yaml @@ -0,0 +1,28 @@ +acts_version: "1.0" +spec_version: "1.0" + +metadata: + title: "A2A Conformance Test Suite" + description: > + Official ACTS test suite for A2A protocol conformance. SDKs that pass + all must-level tests in this suite are considered A2A conformant and + can interoperate with other conformant SDKs. + +variables: + baseUrl: "{{SUT_BASE_URL}}" + +include: + - discovery.acts.yaml + - core-operations.acts.yaml + - history.acts.yaml + - multi-turn.acts.yaml + - streaming.acts.yaml + - polling.acts.yaml + - error-handling.acts.yaml + - auth-security.acts.yaml + - version-negotiation.acts.yaml + - wire-format.acts.yaml + - data-types.acts.yaml + - push-notifications.acts.yaml + - transport-bindings.acts.yaml + - client-parsing.acts.yaml diff --git a/tests/acts/test-viewer.html b/tests/acts/test-viewer.html new file mode 100644 index 000000000..b3518ceee --- /dev/null +++ b/tests/acts/test-viewer.html @@ -0,0 +1,1776 @@ + + + + + +ACTS — A2A Conformance Test Suite + + + + +
+

ACTS — A2A Conformance Test Suite

+
+
111tests
+
65MUST
+
33SHOULD
+
13MAY
+
16suites
+
14files
+
+
+ + + + + +
+
+ +
+ + +
Authentication and Securityauth-security.acts.yaml12 tests
MUST SEC-AUTH-001 Server rejects invalid or missing auth credentials
auth runner-special

Runner MUST attempt a SendMessage request without valid credentials and verify rejection.

spec_ref: REQ-ERR-002
transport: jsonrpc
preconditions
  capabilities:
+    authentication: true

Steps (1)

send-without-auth RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+    A2A-Version: 1.0
+  body:
+    jsonrpc: 2.0
+    id: sec-auth-001
+    method: SendMessage
+    params:
+      message:
+        role: ROLE_USER
+        parts:
+          - text: auth should be required
expect
  status:
+    one_of:
+      - 200
+      - 401
+      - 403
+  body:
+    any_of:
+      - error:           exists: true
+      - title:           type: string
MUST SEC-AUTH-002 Server returns authorization error for insufficient permissions
auth runner-special

Runner MUST attempt a request with valid but insufficient credentials and verify rejection.

spec_ref: REQ-ERR-003
transport: jsonrpc
preconditions
  capabilities:
+    authentication: true

Steps (1)

send-insufficient-auth RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+    A2A-Version: 1.0
+    Authorization: Bearer {{insufficientAuthToken}}
+  body:
+    jsonrpc: 2.0
+    id: sec-auth-002
+    method: SendMessage
+    params:
+      message:
+        role: ROLE_USER
+        parts:
+          - text: authorization should fail
expect
  status:
+    one_of:
+      - 200
+      - 403
+  body:
+    any_of:
+      - error:           exists: true
+      - title:           type: string
MUST SEC-AUTH-003 Server does not reveal unauthorized resource existence
auth security runner-special

Runner MUST verify that getting a task belonging to another user returns the same error as getting a nonexistent task.

spec_ref: REQ-ERR-004
preconditions
  capabilities:
+    authentication: true

Steps (2)

get-other-user-task get_task
params
  id: {{otherUserTaskId}}
expect_error
  message:
+    type: string
get-missing-task get_task
params
  id: 00000000-0000-0000-0000-000000000000
expect_error
  message:
+    type: string
MUST SEC-AUTH-004 Server authenticates every incoming request
auth runner-special

Runner MUST verify that unauthenticated requests to all endpoints are rejected.

spec_ref: REQ-SEC-004
transport: jsonrpc
preconditions
  capabilities:
+    authentication: true

Steps (1)

get-task-without-auth RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+    A2A-Version: 1.0
+  body:
+    jsonrpc: 2.0
+    id: sec-auth-004
+    method: GetTask
+    params:
+      id: 00000000-0000-0000-0000-000000000000
expect
  status:
+    one_of:
+      - 200
+      - 401
+      - 403
+  body:
+    any_of:
+      - error:           exists: true
+      - title:           type: string
MUST SEC-AUTH-005 In-task auth transitions to AUTH_REQUIRED state
auth task-state

Verifies that an in-task authentication challenge transitions the task into TASK_STATE_AUTH_REQUIRED with a status message.

spec_ref: REQ-SEC-006
requires_behaviors: tck-auth-required

Steps (1)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-auth-required trigger auth
expect
  status: 200
+  body:
+    task:
+      status:
+        state: TASK_STATE_AUTH_REQUIRED
+        message:
+          exists: true
MUST SEC-PUSH-001 Push notification webhook includes auth credentials
auth push runner-special

Runner MUST set up a webhook and verify auth credentials are present in request headers.

spec_ref: REQ-SEC-018
requires_behaviors: tck-long-running
preconditions
  capabilities:
+    pushNotifications: true

Steps (2)

create-task send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-long-running start
+  configuration:
+    returnImmediately: true
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
set-config set_push_notification_config
params
  taskId: {{create-task.taskId}}
+  pushNotificationConfig:
+    url: {{webhookUrl}}
+    authentication:
+      scheme: Bearer
+      credentials: test-token
expect
  status: 200
+  body:
+    id:
+      type: string
MUST SEC-PUSH-002 Client validates push notification webhook authenticity
auth push client runner-special

Runner MUST verify that webhook requests with invalid auth are rejected by the client.

spec_ref: REQ-SEC-019
requires_behaviors: tck-long-running
preconditions
  capabilities:
+    pushNotifications: true

Steps (2)

create-task send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-long-running start
+  configuration:
+    returnImmediately: true
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
set-config set_push_notification_config
params
  taskId: {{create-task.taskId}}
+  pushNotificationConfig:
+    url: {{webhookUrl}}
+    authentication:
+      scheme: Bearer
+      credentials: expected-token
expect
  status: 200
+  body:
+    id:
+      type: string
MUST SEC-EXTCARD-001 Extended Agent Card requires authentication
auth discovery runner-special

Verifies that fetching the extended Agent Card without authentication is rejected.

spec_ref: REQ-SEC-021
transport: rest
preconditions
  capabilities:
+    extendedAgentCard: true

Steps (1)

get-extended-card-without-auth RAW GET /extendedAgentCard
raw
  method: GET
+  path: /extendedAgentCard
+  headers:
+    Accept: application/json
+    A2A-Version: 1.0
expect
  status:
+    one_of:
+      - 401
+      - 403
+  body:
+    any_of:
+      - title:           type: string
+      - error:           exists: true
MUST SEC-EXTCARD-002 Extended Agent Card verifies permissions before returning
auth discovery runner-special

Runner MUST verify that an authenticated request with insufficient permissions is rejected.

spec_ref: REQ-SEC-022
transport: rest
preconditions
  capabilities:
+    extendedAgentCard: true

Steps (1)

get-extended-card-insufficient-auth RAW GET /extendedAgentCard
raw
  method: GET
+  path: /extendedAgentCard
+  headers:
+    Accept: application/json
+    A2A-Version: 1.0
+    Authorization: Bearer {{insufficientAuthToken}}
expect
  status:
+    one_of:
+      - 401
+      - 403
+  body:
+    any_of:
+      - title:           type: string
+      - error:           exists: true
MUST SEC-EXTCARD-003 Missing extended-card support returns specified error
auth discovery

Verifies that Get Extended Agent Card returns the specified error when the capability is not advertised.

spec_ref: REQ-SEC-025
preconditions
  capabilities:
+    extendedAgentCard: false

Steps (1)

get-extended-card get_agent_card
params
  extended: true
expect_error
  code: UnsupportedOperationError
MUST SEC-EXTCARD-004 Extended cards enforce access controls
auth discovery runner-special

Runner MUST verify that extended Agent Cards enforce access controls consistently with the registered endpoint.

spec_ref: REQ-IANA-008
transport: rest
preconditions
  capabilities:
+    extendedAgentCard: true

Steps (1)

get-extended-card-without-auth RAW GET /extendedAgentCard
raw
  method: GET
+  path: /extendedAgentCard
+  headers:
+    Accept: application/json
+    A2A-Version: 1.0
expect
  status:
+    one_of:
+      - 401
+      - 403
+  body:
+    any_of:
+      - title:           type: string
+      - error:           exists: true
SHOULD SEC-AUTH-006 Server uses binding-specific auth challenge or rejection
auth runner-special

Runner SHOULD verify that authentication rejection uses the binding-appropriate mechanism, such as HTTP 401 for HTTP transports.

spec_ref: REQ-SEC-005
transport: rest
preconditions
  capabilities:
+    authentication: true

Steps (1)

get-agent-card-without-auth RAW GET /.well-known/agent-card.json
raw
  method: GET
+  path: /.well-known/agent-card.json
+  headers:
+    Accept: application/json
+    A2A-Version: 1.0
expect
  status:
+    one_of:
+      - 200
+      - 401
+      - 403
+  body:
+    any_of:
+      - title:           type: string
+      - error:           exists: true
+
Client Parsing (Golden Responses)client-parsing.acts.yaml8 tests
MUST CLIENT-PARSE-001 Parse SendMessage response with completed task
client parsing task

Verifies the SDK client can parse a SendMessage response containing a completed task with artifacts. +

Steps (1)

parse CLIENT send_message
client_response
  operation: send_message
+  wire_payload:
+    jsonrpc: 2.0
+    id: req-001
+    result:
+      task:
+        id: task-abc-123
+        contextId: ctx-001
+        status:
+          state: TASK_STATE_COMPLETED
+          timestamp: 2025-05-25T20:00:00Z
+        artifacts:
+          - artifactId: art-1
+            parts:               - text: hello world
expect_parsed
  task:
+    id: task-abc-123
+    contextId: ctx-001
+    status:
+      state: TASK_STATE_COMPLETED
+    artifacts:
+      count_gte: 1
MUST CLIENT-PARSE-002 Parse SendMessage response with message (no task)
client parsing message

Verifies the SDK client can parse a SendMessage response containing a direct message (no task created). +

Steps (1)

parse CLIENT send_message
client_response
  operation: send_message
+  wire_payload:
+    jsonrpc: 2.0
+    id: req-002
+    result:
+      message:
+        messageId: msg-001
+        role: ROLE_AGENT
+        parts:
+          - text: I can help with that
expect_parsed
  message:
+    messageId: msg-001
+    role: ROLE_AGENT
+    parts:
+      count_gte: 1
MUST CLIENT-PARSE-003 Parse GetTask response
client parsing get-task

Verifies the SDK client can parse a GetTask response with full task state including history and artifacts. +

Steps (1)

parse CLIENT get_task
client_response
  operation: get_task
+  wire_payload:
+    jsonrpc: 2.0
+    id: req-003
+    result:
+      id: task-xyz-789
+      contextId: ctx-002
+      status:
+        state: TASK_STATE_COMPLETED
+        timestamp: 2025-05-25T20:05:00Z
+      artifacts:
+        - artifactId: art-1
+          parts:             - text: result text
+        - artifactId: art-2
+          parts:             - data:                 key: value
+      history:
+        - role: ROLE_USER
+          parts:             - text: original request
+        - role: ROLE_AGENT
+          parts:             - text: processing...
expect_parsed
  id: task-xyz-789
+  contextId: ctx-002
+  status:
+    state: TASK_STATE_COMPLETED
+  artifacts:
+    count_gte: 2
+  history:
+    count_gte: 2
MUST CLIENT-PARSE-004 Parse error response
client parsing error

Verifies the SDK client can parse a JSON-RPC error response and surface the A2A error code and message. +

Steps (1)

parse CLIENT get_task
client_response
  operation: get_task
+  wire_payload:
+    jsonrpc: 2.0
+    id: req-004
+    error:
+      code: -32001
+      message: Task not found
+      data:
+        taskId: nonexistent-task-id
expect_parsed
  error:
+    code: -32001
+    message:
+      contains: not found
MUST CLIENT-PARSE-005 Parse task with INPUT_REQUIRED state
client parsing input-required

Verifies the SDK client correctly parses a task in INPUT_REQUIRED state with a status message prompting for more input. +

Steps (1)

parse CLIENT send_message
client_response
  operation: send_message
+  wire_payload:
+    jsonrpc: 2.0
+    id: req-005
+    result:
+      task:
+        id: task-multi-001
+        contextId: ctx-multi-001
+        status:
+          state: TASK_STATE_INPUT_REQUIRED
+          timestamp: 2025-05-25T20:10:00Z
+          message:
+            messageId: status-msg-001
+            role: ROLE_AGENT
+            parts:
+              - text: Please provide more details
expect_parsed
  task:
+    id: task-multi-001
+    status:
+      state: TASK_STATE_INPUT_REQUIRED
+      message:
+        role: ROLE_AGENT
+        parts:
+          count_gte: 1
SHOULD CLIENT-PARSE-006 Parse task with mixed artifact types
client parsing artifacts data-types

Verifies the SDK client can parse artifacts containing text, data, file, and fileUrl parts in a single response. +

Steps (1)

parse CLIENT get_task
client_response
  operation: get_task
+  wire_payload:
+    jsonrpc: 2.0
+    id: req-006
+    result:
+      id: task-mixed-001
+      contextId: ctx-mixed-001
+      status:
+        state: TASK_STATE_COMPLETED
+        timestamp: 2025-05-25T20:15:00Z
+      artifacts:
+        - artifactId: art-text
+          parts:             - text: Plain text result
+        - artifactId: art-data
+          parts:             - data:                 temperature: 72.5
+                unit: fahrenheit
+        - artifactId: art-file
+          parts:             - file:                 name: report.pdf
+                mediaType: application/pdf
+                bytes: JVBERi0xLjQ=
+        - artifactId: art-url
+          parts:             - fileUrl:                 url: https://example.com/report.pdf
+                name: report.pdf
+                mediaType: application/pdf
expect_parsed
  id: task-mixed-001
+  artifacts:
+    count_gte: 4
SHOULD CLIENT-AUTH-001 Client replaces cached public card with extended card
client discovery runner-special

Runner SHOULD verify that after fetching an extended Agent Card, the client uses the extended capabilities for the authenticated session.

spec_ref: REQ-AGENTCARD-EXT-002

Steps (2)

parse-public-card CLIENT get_agent_card
client_response
  operation: get_agent_card
+  wire_payload:
+    name: Example Agent
+    version: 1.0.0
+    capabilities:
+      streaming: false
+      extendedAgentCard: true
+    supportedInterfaces:
+      - url: https://example.com/.well-known/agent-card.json
+        protocolBinding: REST
+        protocolVersion: 1.0
expect_parsed
  capabilities:
+    extendedAgentCard: true
parse-extended-card CLIENT get_extended_agent_card
client_response
  operation: get_extended_agent_card
+  wire_payload:
+    name: Example Agent
+    version: 1.0.0
+    capabilities:
+      streaming: true
+      extendedAgentCard: true
+    skills:
+      - id: extended-skill
+        name: Extended Skill
+    supportedInterfaces:
+      - url: https://example.com/extendedAgentCard
+        protocolBinding: REST
+        protocolVersion: 1.0
expect_parsed
  capabilities:
+    streaming: true
+    extendedAgentCard: true
+  skills:
+    count_gte: 1
SHOULD CLIENT-CAP-001 Client checks Agent Card before using optional capabilities
client capabilities runner-special

Runner SHOULD verify that the client checks Agent Card capabilities before calling optional operations.

spec_ref: REQ-CAP-002

Steps (1)

parse-card CLIENT get_agent_card
client_response
  operation: get_agent_card
+  wire_payload:
+    name: Capability Gated Agent
+    version: 1.0.0
+    capabilities:
+      streaming: false
+      pushNotifications: false
+    supportedInterfaces:
+      - url: https://example.com/.well-known/agent-card.json
+        protocolBinding: REST
+        protocolVersion: 1.0
expect_parsed
  capabilities:
+    streaming: false
+    pushNotifications: false
+
Core A2A operationscore-operations.acts.yaml12 tests

SendMessage, GetTask, and CancelTask conformance coverage for core lifecycle flows.

MUST CORE-SEND-001 SendMessage returns a completed task

Verifies that SendMessage can return a completed task for the complete-task TCK behavior.

spec_ref: docs/specification.md#311-send-message
requires_behaviors: tck-complete-task

Steps (1)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task hello world
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
capture
  taskId: task.id
MUST CORE-SEND-002 SendMessage to a terminal task returns error

Verifies that sending a follow-up message to a terminal task returns the protocol error for unsupported operation.

spec_ref: docs/specification.md#311-send-message
requires_behaviors: tck-complete-task

Steps (2)

setup send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task setup
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
capture
  taskId: task.id
resend send_message
params
  taskId: {{setup.taskId}}
+  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task terminal follow-up
expect_error
  error_type: UnsupportedOperationError
+  message:
+    type: string
MUST CORE-SEND-003 SendMessage returns a message response (no task)

Verifies that SendMessage may return a direct agent message response without creating a task.

spec_ref: docs/specification.md#311-send-message
requires_behaviors: tck-message-response

Steps (1)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-message-response hello
expect
  status: 200
+  body:
+    message:
+      role: ROLE_AGENT
+    task:
+      absent: true
MUST CORE-GET-001 GetTask returns current state of existing task

Verifies that GetTask returns the current state for an existing completed task.

spec_ref: docs/specification.md#313-get-task
requires_behaviors: tck-complete-task

Steps (2)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task for get
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
capture
  taskId: task.id
get get_task
params
  id: {{send.taskId}}
expect
  status: 200
+  body:
+    id: {{send.taskId}}
+    status:
+      state: TASK_STATE_COMPLETED
MUST CORE-GET-002 GetTask with historyLength 0 omits history

Verifies that GetTask omits task history, or returns it as an empty array, when historyLength is zero.

spec_ref: docs/specification.md#313-get-task
requires_behaviors: tck-complete-task

Steps (2)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task history zero
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
capture
  taskId: task.id
get get_task
params
  id: {{send.taskId}}
+  historyLength: 0
expect
  status: 200
+  body:
+    id: {{send.taskId}}
+    history:
+      any_of:
+        - absent: true
+        - type: array
+          count: 0
MUST CORE-CANCEL-001 CancelTask returns updated state

Verifies that CancelTask returns the updated canceled task state for a task started in return-immediately mode.

spec_ref: docs/specification.md#315-cancel-task
requires_behaviors: tck-cancel

Steps (2)

start send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-cancel start
+  configuration:
+    returnImmediately: true
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state:
+          one_of:
+            - TASK_STATE_SUBMITTED
+            - TASK_STATE_WORKING
capture
  taskId: task.id
cancel cancel_task
params
  id: {{start.taskId}}
expect
  status: 200
+  body:
+    id: {{start.taskId}}
+    status:
+      state: TASK_STATE_CANCELED
MUST CORE-CANCEL-002 CancelTask on terminal task returns error

Verifies that CancelTask on a terminal task fails with TaskNotCancelableError.

spec_ref: docs/specification.md#315-cancel-task
requires_behaviors: tck-complete-task

Steps (2)

setup send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task setup
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
capture
  taskId: task.id
cancel cancel_task
params
  id: {{setup.taskId}}
expect_error
  error_type: TaskNotCancelableError
+  message:
+    type: string
MUST CORE-FAIL-001 Task completes with failed state

Verifies that SendMessage can return a failed task with an explanatory status message.

spec_ref: docs/specification.md#311-send-message
requires_behaviors: tck-task-failure

Steps (1)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-task-failure trigger error
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_FAILED
+        message:
+          exists: true
MUST CORE-SEND-004 SendMessage rejects unsupported content type

Verifies that SendMessage fails with ContentTypeNotSupportedError when a message part uses an unsupported content type.

spec_ref: docs/specification.md#311-send-message
requires_behaviors: tck-complete-task

Steps (1)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - data:           value: unsupported
+        metadata:           contentType: application/x-unsupported-type-12345
expect_error
  code: ContentTypeNotSupportedError
SHOULD CORE-LIST-001 ListTasks returns tasks for a context

Verifies that ListTasks can return at least one task for an existing context.

spec_ref: docs/specification.md#314-list-tasks
requires_behaviors: tck-complete-task

Steps (2)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task for list test
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      contextId:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
capture
  taskId: task.id
+  contextId: task.contextId
list list_tasks
params
  contextId: {{send.contextId}}
expect
  status: 200
+  body:
+    tasks:
+      type: array
+      count_gte: 1
MUST CORE-LIST-002 ListTasks with includeArtifacts false omits artifacts

Verifies that ListTasks omits artifacts, or returns them as null, when includeArtifacts is false.

spec_ref: REQ-TASK-LIST-001
requires_behaviors: tck-complete-task

Steps (2)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task for list artifacts test
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      contextId:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
capture
  taskId: task.id
+  contextId: task.contextId
list list_tasks
params
  contextId: {{send.contextId}}
+  includeArtifacts: false
expect
  status: 200
+  body:
+    tasks:
+      type: array
+      count_gte: 1
+      items:
+        - id: {{send.taskId}}
+          contextId: {{send.contextId}}
+          artifacts:             any_of:
+              - absent: true
+              - type: null
MUST CORE-LIST-003 ListTasks response includes nextPageToken

Verifies that ListTasks responses always include nextPageToken, using an empty string for the final page.

spec_ref: REQ-TASK-LIST-002
requires_behaviors: tck-complete-task

Steps (2)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task for list next page token test
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      contextId:
+        type: string
capture
  taskId: task.id
+  contextId: task.contextId
list list_tasks
params
  contextId: {{send.contextId}}
expect
  status: 200
+  body:
+    tasks:
+      type: array
+      count_gte: 1
+    nextPageToken:
+      type: string
+
Artifact Data Typesdata-types.acts.yaml7 tests
MUST DM-ART-001 Text artifact

Sends a request that should produce at least one text artifact and verifies a returned artifact part includes a text field. +

spec_ref: specification.md#4.1.7
requires_behaviors: tck-artifact-text

Steps (1)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-artifact-text hello
expect
  status: 200
+  body:
+    task:
+      status:
+        state: TASK_STATE_COMPLETED
+      artifacts:
+        count_gte: 1
SHOULD DM-ART-002 Data artifact (structured JSON)

Sends a request that should produce a structured JSON artifact and verifies a returned artifact part includes an object-valued data field. +

spec_ref: specification.md#4.1.7
requires_behaviors: tck-artifact-data

Steps (1)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-artifact-data structured
expect
  status: 200
+  body:
+    task:
+      status:
+        state: TASK_STATE_COMPLETED
+      artifacts:
+        count_gte: 1
SHOULD DM-ART-003 File artifact (inline)

Sends a request that should produce an inline file artifact and verifies a returned artifact part includes file metadata. +

spec_ref: specification.md#4.1.7
requires_behaviors: tck-artifact-file

Steps (1)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-artifact-file document
expect
  status: 200
+  body:
+    task:
+      status:
+        state: TASK_STATE_COMPLETED
+      artifacts:
+        count_gte: 1
MAY DM-ART-004 File URL artifact

Sends a request that should produce a file URL artifact and verifies a returned artifact part includes fileUrl metadata. +

spec_ref: specification.md#4.1.7
requires_behaviors: tck-artifact-file-url

Steps (1)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-artifact-file-url document
expect
  status: 200
+  body:
+    task:
+      status:
+        state: TASK_STATE_COMPLETED
+      artifacts:
+        count_gte: 1
MUST DM-SERIAL-001 Timestamps are ISO 8601 UTC
requires_behaviors: tck-complete-task

Steps (1)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task timestamp
expect
  status: 200
+  body:
+    task:
+      status:
+        state: TASK_STATE_COMPLETED
+        timestamp:
+          matches: ^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}
SHOULD DM-SERIAL-002 Response validates against A2A schema
data-model schema-validation runner-special

Runner SHOULD validate full response against A2A JSON Schema.

requires_behaviors: tck-complete-task

Steps (1)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task schema
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state:
+          type: string
SHOULD DM-EXTRA-001 Extra fields accepted (tolerance)
transport: jsonrpc
requires_behaviors: tck-complete-task

Steps (1)

send-extra-field RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+    A2A-Version: 1.0
+  body:
+    jsonrpc: 2.0
+    id: 1
+    method: SendMessage
+    params:
+      message:
+        role: ROLE_USER
+        parts:
+          - text: tck-complete-task extra field
+      x_custom_field: test
expect
  status: 200
+  body:
+    result:
+      exists: true
+
Agent Card Discoverydiscovery.acts.yaml10 tests
MUST CARD-DISC-001 Agent card is discoverable

Agent card is discoverable at the well-known URL and exposes core metadata.

spec_ref: Agent Card Discovery
requires_behaviors:

Steps (1)

step-1 get_agent_card
expect
  status: 200
+  body:
+    name:
+      exists: true
+    version:
+      exists: true
+    description:
+      exists: true
MUST CARD-DISC-002 Agent card contains supported interfaces

Agent card lists one or more supported interfaces with binding and version metadata.

spec_ref: Agent Card Discovery
requires_behaviors:

Steps (1)

step-1 get_agent_card
expect
  status: 200
+  body:
+    supportedInterfaces:
+      type: array
+      count_gte: 1
+      items:
+        url:
+          type: string
+        protocolBinding:
+          one_of:
+            - JSONRPC
+            - GRPC
+            - REST
+        protocolVersion:
+          type: string
MUST CARD-DISC-003 Agent card contains capabilities

Agent card exposes capabilities as an object.

spec_ref: Agent Card Discovery
requires_behaviors:

Steps (1)

step-1 get_agent_card
expect
  status: 200
+  body:
+    capabilities:
+      exists: true
+      type: object
SHOULD CARD-DISC-004 Agent card contains skills

Agent card includes skills with identifiers and names.

spec_ref: Agent Card Discovery
requires_behaviors:

Steps (1)

step-1 get_agent_card
expect
  status: 200
+  body:
+    skills:
+      type: array
+      items:
+        id:
+          exists: true
+        name:
+          exists: true
SHOULD CARD-DISC-005 Agent card contains default input and output modes

Agent card includes default input and output mode arrays.

spec_ref: Agent Card Discovery
requires_behaviors:

Steps (1)

step-1 get_agent_card
expect
  status: 200
+  body:
+    defaultInputModes:
+      type: array
+    defaultOutputModes:
+      type: array
MUST CARD-DISC-006 Agent card protocolVersion matches spec version

The first supported interface advertises an A2A 1.x protocol version.

spec_ref: Agent Card Discovery
requires_behaviors:

Steps (1)

step-1 get_agent_card
expect
  status: 200
+  body:
+    supportedInterfaces:
+      type: array
+      count_gte: 1
+      items:
+        - protocolVersion:             starts_with: 1.
SHOULD CARD-CACHE-001 Agent card caching headers
discovery caching runner-special

Runner SHOULD verify Cache-Control or ETag headers are present.

spec_ref: Agent Card Discovery
requires_behaviors:

Steps (1)

step-1 get_agent_card
expect
  status: 200
+  body:
+    name:
+      exists: true
SHOULD CARD-SCHEMA-001 Agent card validates against schema
discovery schema-validation

Runner MUST validate response against A2A AgentCard JSON Schema.

spec_ref: Agent Card Discovery
requires_behaviors:

Steps (1)

step-1 get_agent_card
expect
  status: 200
+  body:
+    name:
+      exists: true
+    capabilities:
+      type: object
MAY CARD-EXT-001 Extended agent card

Retrieves the extended agent card when the capability is advertised.

spec_ref: Get Extended Agent Card
requires_behaviors:
preconditions
  capabilities:
+    extendedAgentCard: true

Steps (1)

step-1 get_agent_card
params
  extended: true
expect
  status: 200
+  body:
+    skills:
+      type: array
+      count_gte: 1
+    description:
+      matches: .+
MUST CARD-DISC-007 Agent card declares all supported protocols

Verifies that the public Agent Card declares at least one protocol endpoint through supportedInterfaces or a direct url field.

spec_ref: REQ-BIND-002
requires_behaviors:

Steps (1)

step-1 get_agent_card
expect
  status: 200
+  body:
+    any_of:
+      - supportedInterfaces:           type: array
+          count_gte: 1
+      - url:           type: string
+
Error Handlingerror-handling.acts.yaml14 tests
MUST CORE-ERR-001 GetTask with non-existent ID returns TaskNotFoundError
error get-task task-not-found

Steps (1)

get-missing get_task
params
  id: 00000000-0000-0000-0000-000000000000
expect_error
  code: TaskNotFoundError
+  message:
+    type: string
MUST CORE-ERR-002 CancelTask on non-existent task returns error
error cancel-task

Steps (1)

cancel-missing cancel_task
params
  id: 00000000-0000-0000-0000-000000000000
expect_error
  code:
+    one_of:
+      - TaskNotFoundError
+      - TaskNotCancelableError
MUST CORE-ERR-003 CancelTask on completed task returns TaskNotCancelableError
error cancel-task completed-task
requires_behaviors: tck-complete-task

Steps (2)

setup-completed-task send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task setup
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
capture
  taskId: task.id
cancel-completed-task cancel_task
params
  id: {{setup-completed-task.taskId}}
expect_error
  code: TaskNotCancelableError
MUST CORE-ERR-004 SendMessage to terminal task returns UnsupportedOperationError
error send-message terminal-task
requires_behaviors: tck-complete-task

Steps (2)

create-terminal-task send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task first
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
capture
  taskId: task.id
resend-terminal-task send_message
params
  message:
+    role: ROLE_USER
+    taskId: {{create-terminal-task.taskId}}
+    parts:
+      - text: tck-complete-task second
expect_error
  code: UnsupportedOperationError
MUST JSONRPC-ERR-001 Unknown method returns MethodNotFoundError
error jsonrpc raw
transport: jsonrpc

Steps (1)

bad-method RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+    A2A-Version: 1.0
+  body:
+    jsonrpc: 2.0
+    id: 1
+    method: DoSomethingUnsupported
+    params:
+{}
expect
  status: 200
+  body:
+    error:
+      code: -32601
+      message:
+        type: string
MUST JSONRPC-ERR-002 Invalid JSON returns ParseError
error jsonrpc raw parse
transport: jsonrpc

Steps (1)

bad-json RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+    A2A-Version: 1.0
+  body_raw: {this is not valid json
expect
  status: 200
+  body:
+    error:
+      code: -32700
SHOULD CORE-ERR-005 Error responses include actionable message
error message-quality

Steps (1)

get-missing-actionable get_task
params
  id: 00000000-0000-0000-0000-000000000000
expect_error
  code: TaskNotFoundError
+  message:
+    all_of:
+      - type: string
+      - matches: .{12,}
+      - any_of:           - contains: task
+          - contains: Task
+          - contains: not found
+          - contains: does not exist
MUST CORE-ERR-006 Malformed request returns InvalidRequestError
transport: jsonrpc

Steps (1)

malformed-jsonrpc RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+    A2A-Version: 1.0
+  body:
+    jsonrpc: 2.0
+    id: 1
+    params:
+{}
expect
  status: 200
+  body:
+    error:
+      code:
+        one_of:
+          - -32600
+          - -32602
MUST CORE-CAP-001 Push not supported returns UnsupportedOperationError

Overlaps with PUSH-CFG-004 but verifies the capability error at the error-handling suite level.

requires_behaviors: tck-complete-task
preconditions
  capabilities:
+    pushNotifications: false

Steps (2)

create-task send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task for push
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
set-config set_push_notification_config
params
  taskId: {{create-task.taskId}}
+  pushNotificationConfig:
+    url: https://example.com/webhooks/a2a-tests
expect_error
  code: UnsupportedOperationError
SHOULD CORE-CAP-002 Streaming not supported returns error
requires_behaviors: tck-complete-task
preconditions
  capabilities:
+    streaming: false

Steps (1)

stream send_streaming_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task streaming unsupported
expect_error
  code: UnsupportedOperationError
SHOULD CORE-ERR-007 Error data contains structured error info

Steps (1)

get-missing-structured get_task
params
  id: 00000000-0000-0000-0000-000000000000
expect_error
  code: TaskNotFoundError
+  data:
+    type: object
MUST CORE-ERR-008 Error details include @type field

Verifies that JSON-RPC error details include a google.rpc Any type discriminator.

spec_ref: REQ-ERR-009
transport: jsonrpc

Steps (1)

send-missing-task RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+    A2A-Version: 1.0
+  body:
+    jsonrpc: 2.0
+    id: core-err-008
+    method: SendMessage
+    params:
+      taskId: 00000000-0000-0000-0000-000000000000
+      message:
+        role: ROLE_USER
+        parts:
+          - text: missing task should return error details
expect
  status: 200
+  body:
+    error:
+      message:
+        type: string
+      data:
+        type: array
+        count_gte: 1
+        items:
+          - @type:               type: string
SHOULD CORE-ERR-009 Error response should not distinguish missing vs unauthorized
security error-handling runner-special

Runner SHOULD verify that the error response for a missing resource is indistinguishable from one for an unauthorized resource. This requires auth infrastructure.

spec_ref: REQ-ERR-007

Steps (1)

get-missing get_task
params
  id: 00000000-0000-0000-0000-000000000000
expect_error
  message:
+    type: string
MUST JSONRPC-ERR-003 JSON-RPC error details include google.rpc.ErrorInfo
error jsonrpc raw

Verifies that A2A-specific JSON-RPC errors include google.rpc.ErrorInfo in error.data.

spec_ref: REQ-RPC-006
transport: jsonrpc

Steps (1)

get-missing-error-info RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+    A2A-Version: 1.0
+  body:
+    jsonrpc: 2.0
+    id: jsonrpc-err-003
+    method: GetTask
+    params:
+      id: 00000000-0000-0000-0000-000000000000
expect
  status: 200
+  body:
+    error:
+      message:
+        type: string
+      data:
+        type: array
+        count_gte: 1
+
Task Historyhistory.acts.yaml6 tests

Coverage for GetTask and SendMessage history handling.

MUST CORE-HIST-001 GetTask historyLength=0 returns no history

Verifies that GetTask omits history, or returns it as an empty array, when historyLength is zero.

spec_ref: docs/specification.md#313-get-task
requires_behaviors: tck-complete-task

Steps (2)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task history zero
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
capture
  taskId: task.id
get get_task
params
  id: {{send.taskId}}
+  historyLength: 0
expect
  status: 200
+  body:
+    id: {{send.taskId}}
+    history:
+      any_of:
+        - absent: true
+        - type: array
+          count: 0
SHOULD CORE-HIST-002 History count does not exceed historyLength limit

Verifies that GetTask does not return more history entries than requested.

spec_ref: docs/specification.md#313-get-task
requires_behaviors: tck-multi-turn

Steps (4)

turn1 send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-multi-turn history limit first
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_INPUT_REQUIRED
capture
  taskId: task.id
+  contextId: task.contextId
turn2 send_message
params
  taskId: {{turn1.taskId}}
+  contextId: {{turn1.contextId}}
+  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-multi-turn history limit second
expect
  status: 200
+  body:
+    task:
+      id: {{turn1.taskId}}
+      status:
+        state: TASK_STATE_INPUT_REQUIRED
turn3 send_message
params
  taskId: {{turn1.taskId}}
+  contextId: {{turn1.contextId}}
+  message:
+    role: ROLE_USER
+    parts:
+      - text: done
expect
  status: 200
+  body:
+    task:
+      id: {{turn1.taskId}}
+      status:
+        state: TASK_STATE_COMPLETED
get get_task
params
  id: {{turn1.taskId}}
+  historyLength: 2
expect
  status: 200
+  body:
+    id: {{turn1.taskId}}
+    history:
+      any_of:
+        - absent: true
+        - type: array
+          count_lte: 2
SHOULD CORE-HIST-003 SendMessage historyLength=0 returns no history

Verifies that SendMessage omits task history when configuration.historyLength is zero.

spec_ref: docs/specification.md#311-send-message
requires_behaviors: tck-complete-task

Steps (1)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task send history zero
+  configuration:
+    historyLength: 0
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
+      history:
+        absent: true
MAY CORE-HIST-004 GetTask may return persisted history

Verifies that GetTask may omit history entirely, but if it returns history the field is an array.

spec_ref: docs/specification.md#313-get-task
requires_behaviors: tck-complete-task

Steps (2)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task persisted history
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
capture
  taskId: task.id
get get_task
params
  id: {{send.taskId}}
expect
  status: 200
+  body:
+    id: {{send.taskId}}
+    history:
+      any_of:
+        - absent: true
+        - type: array
SHOULD CORE-HIST-005 History messages are returned in chronological order

Verifies that persisted history preserves the order of the conversation turns.

spec_ref: docs/specification.md#313-get-task
requires_behaviors: tck-multi-turn

Steps (4)

turn1 send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-multi-turn chronological first
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_INPUT_REQUIRED
capture
  taskId: task.id
+  contextId: task.contextId
turn2 send_message
params
  taskId: {{turn1.taskId}}
+  contextId: {{turn1.contextId}}
+  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-multi-turn chronological second
expect
  status: 200
+  body:
+    task:
+      id: {{turn1.taskId}}
+      status:
+        state: TASK_STATE_INPUT_REQUIRED
turn3 send_message
params
  taskId: {{turn1.taskId}}
+  contextId: {{turn1.contextId}}
+  message:
+    role: ROLE_USER
+    parts:
+      - text: done
expect
  status: 200
+  body:
+    task:
+      id: {{turn1.taskId}}
+      status:
+        state: TASK_STATE_COMPLETED
get get_task
params
  id: {{turn1.taskId}}
expect
  status: 200
+  body:
+    id: {{turn1.taskId}}
+    history:
+      type: array
+      count_gte: 3
SHOULD CORE-HIST-006 History content matches sent messages

Verifies that persisted history includes the messages sent during a multi-turn conversation.

spec_ref: docs/specification.md#313-get-task
requires_behaviors: tck-multi-turn

Steps (4)

turn1 send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-multi-turn first msg
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_INPUT_REQUIRED
capture
  taskId: task.id
+  contextId: task.contextId
turn2 send_message
params
  taskId: {{turn1.taskId}}
+  contextId: {{turn1.contextId}}
+  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-multi-turn follow up msg
expect
  status: 200
+  body:
+    task:
+      id: {{turn1.taskId}}
+      status:
+        state:
+          one_of:
+            - TASK_STATE_INPUT_REQUIRED
+            - TASK_STATE_COMPLETED
turn3 send_message
params
  taskId: {{turn1.taskId}}
+  contextId: {{turn1.contextId}}
+  message:
+    role: ROLE_USER
+    parts:
+      - text: done
expect
  status: 200
+  body:
+    task:
+      id: {{turn1.taskId}}
+      status:
+        state: TASK_STATE_COMPLETED
get get_task
params
  id: {{turn1.taskId}}
expect
  status: 200
+  body:
+    id: {{turn1.taskId}}
+    history:
+      type: array
+      count_gte: 2
+
Multi-turn Conversationsmulti-turn.acts.yaml5 tests
MUST CORE-MULTI-001 Multi-turn with INPUT_REQUIRED

Covers TCK CORE-MULTI-001, CSIT scenario-parity input-required, and agentbin spec-multi-turn behavior.

spec_ref: CORE-MULTI-001
requires_behaviors: tck-multi-turn

Steps (3)

turn1 send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-multi-turn start conversation
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_INPUT_REQUIRED
capture
  taskId: task.id
+  contextId: task.contextId
turn2 send_message
params
  taskId: {{turn1.taskId}}
+  contextId: {{turn1.contextId}}
+  message:
+    role: ROLE_USER
+    parts:
+      - text: here is more input
expect
  status: 200
+  body:
+    task:
+      id: {{turn1.taskId}}
+      status:
+        state: TASK_STATE_INPUT_REQUIRED
turn3 send_message
params
  taskId: {{turn1.taskId}}
+  contextId: {{turn1.contextId}}
+  message:
+    role: ROLE_USER
+    parts:
+      - text: done
expect
  status: 200
+  body:
+    task:
+      id: {{turn1.taskId}}
+      status:
+        state: TASK_STATE_COMPLETED
MUST CORE-MULTI-005 SendMessage with only taskId infers contextId

Verifies follow-up SendMessage can omit contextId and still remain on the original task context.

spec_ref: CORE-MULTI-005
requires_behaviors: tck-multi-turn

Steps (2)

turn1 send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-multi-turn start
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      contextId:
+        type: string
+      status:
+        state: TASK_STATE_INPUT_REQUIRED
capture
  taskId: task.id
+  contextId: task.contextId
turn2 send_message
params
  taskId: {{turn1.taskId}}
+  message:
+    role: ROLE_USER
+    parts:
+      - text: done
expect
  status: 200
+  body:
+    task:
+      id: {{turn1.taskId}}
+      contextId: {{turn1.contextId}}
+      status:
+        state: TASK_STATE_COMPLETED
MUST CORE-MULTI-006 SendMessage with taskId and wrong contextId returns error

Verifies mismatched contextId is rejected for a follow-up turn.

spec_ref: CORE-MULTI-006
requires_behaviors: tck-multi-turn

Steps (2)

turn1 send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-multi-turn start
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_INPUT_REQUIRED
capture
  taskId: task.id
turn2 send_message
params
  taskId: {{turn1.taskId}}
+  contextId: wrong-context-id-12345
+  message:
+    role: ROLE_USER
+    parts:
+      - text: should fail
expect
  status: error
+  error:
+    exists: true
MUST CORE-MULTI-003 Mismatched contextId and taskId is rejected

Verifies that SendMessage rejects a taskId and contextId pair from different conversations.

spec_ref: REQ-MULTI-002
requires_behaviors: tck-complete-task

Steps (3)

first send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task first
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      contextId:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
capture
  taskId1: task.id
+  contextId1: task.contextId
second send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task second
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      contextId:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
capture
  taskId2: task.id
+  contextId2: task.contextId
mismatch send_message
params
  taskId: {{first.taskId1}}
+  contextId: {{second.contextId2}}
+  message:
+    role: ROLE_USER
+    parts:
+      - text: mismatched context and task
expect_error
  message:
+    type: string
MUST CORE-CTX-001 Rejected client contextId returns an error
context identity runner-special

Runner SHOULD use a client-generated contextId that the server rejects and verify the server errors instead of minting a new contextId.

spec_ref: REQ-CTX-002
requires_behaviors: tck-complete-task

Steps (1)

send send_message
params
  contextId: client-generated-context-that-should-be-rejected
+  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task reject client context id
expect_error
  message:
+    type: string
+
Polling and Long-Running Taskspolling.acts.yaml2 tests
MUST CORE-EXEC-001 Long-running task with returnImmediately and polling

Sends a long-running request with returnImmediately enabled, polls until the task reaches a terminal state, and verifies the task completes with at least one artifact. +

spec_ref: specification.md#3.1.1
requires_behaviors: tck-long-running

Steps (3)

start send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-long-running start process
+  configuration:
+    returnImmediately: true
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state:
+          one_of:
+            - TASK_STATE_SUBMITTED
+            - TASK_STATE_WORKING
capture
  taskId: task.id
poll get_task
params
  id: {{start.taskId}}
expect
  status: 200
+  body:
+    status:
+      state:
+        type: string
repeat
  until: status.state in [TASK_STATE_COMPLETED, TASK_STATE_FAILED]
+  max_attempts: 15
+  delay_ms: 2000
verify get_task
params
  id: {{start.taskId}}
expect
  status: 200
+  body:
+    id: {{start.taskId}}
+    status:
+      state: TASK_STATE_COMPLETED
+    artifacts:
+      type: array
+      count_gte: 1
SHOULD CORE-EXEC-002 Polling shows progress through working state

Sends a long-running request with returnImmediately enabled, verifies polling observes TASK_STATE_WORKING during execution, and then confirms the task reaches TASK_STATE_COMPLETED. +

spec_ref: specification.md#3.1.1
requires_behaviors: tck-long-running

Steps (4)

start send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-long-running start process
+  configuration:
+    returnImmediately: true
expect
  status: 200
+  body:
+    task:
+      status:
+        state:
+          one_of:
+            - TASK_STATE_SUBMITTED
+            - TASK_STATE_WORKING
capture
  taskId: task.id
wait-for-working get_task
params
  id: {{start.taskId}}
expect
  status: 200
+  body:
+    status:
+      state:
+        type: string
repeat
  until: status.state == TASK_STATE_WORKING
+  max_attempts: 10
+  delay_ms: 1000
poll-to-terminal get_task
params
  id: {{start.taskId}}
expect
  status: 200
+  body:
+    status:
+      state:
+        type: string
repeat
  until: status.state in [TASK_STATE_COMPLETED, TASK_STATE_FAILED]
+  max_attempts: 15
+  delay_ms: 2000
verify get_task
params
  id: {{start.taskId}}
expect
  status: 200
+  body:
+    id: {{start.taskId}}
+    status:
+      state: TASK_STATE_COMPLETED
+
Push Notification Configurationpush-notifications.acts.yaml10 tests
MAY PUSH-CFG-001 Set push notification config

Creates a task, configures push notifications for it, and verifies the returned configuration uses the requested webhook URL. +

spec_ref: specification.md#3.1.7
requires_behaviors: tck-complete-task
preconditions
  capabilities:
+    pushNotifications: true

Steps (2)

create send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task for push
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
set-config set_push_notification_config
params
  taskId: {{create.taskId}}
+  pushNotificationConfig:
+    url: {{webhookUrl}}
expect
  status: 200
+  body:
+    id:
+      type: string
+    url: {{webhookUrl}}
capture
  configId: id
MAY PUSH-CFG-002 Get push notification config

Creates a task, stores a push notification configuration, retrieves it, and verifies the previously set configuration is returned. +

spec_ref: specification.md#3.1.8
requires_behaviors: tck-complete-task
preconditions
  capabilities:
+    pushNotifications: true

Steps (3)

create send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task for push
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
set-config set_push_notification_config
params
  taskId: {{create.taskId}}
+  pushNotificationConfig:
+    url: {{webhookUrl}}
expect
  status: 200
+  body:
+    id:
+      type: string
capture
  configId: id
get-config get_push_notification_config
params
  taskId: {{create.taskId}}
+  id: {{set-config.configId}}
expect
  status: 200
+  body:
+    id: {{set-config.configId}}
+    url: {{webhookUrl}}
MAY PUSH-CFG-003 Delete push notification config

Creates a task, stores a push notification configuration, deletes it, and expects a successful deletion response. +

spec_ref: specification.md#3.1.10
requires_behaviors: tck-complete-task
preconditions
  capabilities:
+    pushNotifications: true

Steps (3)

create send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task for push
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
set-config set_push_notification_config
params
  taskId: {{create.taskId}}
+  pushNotificationConfig:
+    url: {{webhookUrl}}
expect
  status: 200
+  body:
+    id:
+      type: string
capture
  configId: id
delete-config delete_push_notification_config
params
  taskId: {{create.taskId}}
+  id: {{set-config.configId}}
expect
  status: 200
MAY PUSH-CFG-004 Push config on unsupported server returns error

Creates a task on a server that does not advertise push notification support, then verifies setting a push notification configuration fails. +

spec_ref: specification.md#3.1.7
requires_behaviors: tck-complete-task
preconditions
  capabilities:
+    pushNotifications: false

Steps (2)

create send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task for push
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
set-config set_push_notification_config
params
  taskId: {{create.taskId}}
+  pushNotificationConfig:
+    url: {{webhookUrl}}
expect_error
  code: UnsupportedOperationError
MAY PUSH-LIST-001 List push configs
requires_behaviors: tck-complete-task
preconditions
  capabilities:
+    pushNotifications: true

Steps (3)

create send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task for push
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
set-config set_push_notification_config
params
  taskId: {{create.taskId}}
+  pushNotificationConfig:
+    url: {{webhookUrl}}
expect
  status: 200
+  body:
+    id:
+      type: string
capture
  configId: id
list-configs list_push_notification_configs
params
  taskId: {{create.taskId}}
expect
  status: 200
+  body:
+    configs:
+      type: array
+      count_gte: 1
MAY PUSH-ERR-001 Get nonexistent push config returns error
requires_behaviors: tck-complete-task
preconditions
  capabilities:
+    pushNotifications: true

Steps (2)

create send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task for push
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
get-config get_push_notification_config
params
  taskId: {{create.taskId}}
+  id: 00000000-0000-0000-0000-000000000000
expect_error
  code: TaskNotFoundError
MAY PUSH-IDEM-001 Delete push config is idempotent
requires_behaviors: tck-complete-task
preconditions
  capabilities:
+    pushNotifications: true

Steps (4)

create send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task for push
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
set-config set_push_notification_config
params
  taskId: {{create.taskId}}
+  pushNotificationConfig:
+    url: {{webhookUrl}}
expect
  status: 200
+  body:
+    id:
+      type: string
capture
  configId: id
delete-config delete_push_notification_config
params
  taskId: {{create.taskId}}
+  id: {{set-config.configId}}
expect
  status: 200
delete-config-again delete_push_notification_config
params
  taskId: {{create.taskId}}
+  id: {{set-config.configId}}
expect
  status: 200
MAY PUSH-DELIV-001 Push delivery includes auth
push delivery runner-special

Runner MUST set up a webhook endpoint and verify auth credentials are included.

requires_behaviors: tck-long-running
preconditions
  capabilities:
+    pushNotifications: true

Steps (2)

create send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-long-running start
+  configuration:
+    returnImmediately: true
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
set-config set_push_notification_config
params
  taskId: {{create.taskId}}
+  pushNotificationConfig:
+    url: {{webhookUrl}}
+    authentication:
+      scheme: Bearer
+      credentials: test-token
expect
  status: 200
+  body:
+    id:
+      type: string
MAY PUSH-DELIV-002 Push delivery is at-least-once
push delivery runner-special

Runner MUST set up a webhook endpoint and verify at-least-once delivery semantics.

requires_behaviors: tck-long-running
preconditions
  capabilities:
+    pushNotifications: true

Steps (2)

create send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-long-running start
+  configuration:
+    returnImmediately: true
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
set-config set_push_notification_config
params
  taskId: {{create.taskId}}
+  pushNotificationConfig:
+    url: {{webhookUrl}}
expect
  status: 200
+  body:
+    id:
+      type: string
MAY PUSH-DELIV-003 Push payload format correct
push delivery runner-special

Runner MUST set up a webhook endpoint and verify the delivered payload format matches the A2A specification.

requires_behaviors: tck-long-running
preconditions
  capabilities:
+    pushNotifications: true

Steps (2)

create send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-long-running start
+  configuration:
+    returnImmediately: true
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
set-config set_push_notification_config
params
  taskId: {{create.taskId}}
+  pushNotificationConfig:
+    url: {{webhookUrl}}
expect
  status: 200
+  body:
+    id:
+      type: string
+
Streaming operationsstreaming.acts.yaml11 tests

Covers send_streaming_message and subscribe_to_task behaviors.

MUST STREAM-SSE-001 SendStreamingMessage produces events ending in terminal state
requires_behaviors: tck-stream-basic

Steps (1)

stream send_streaming_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-stream-basic generate output
expect_stream
  min_count: 2
+  ordering: monotonic_state
+  final_event:
+    status:
+      state:
+        one_of:
+          - TASK_STATE_COMPLETED
+          - TASK_STATE_FAILED
MUST STREAM-SSE-002 Stream includes at least one artifact update
requires_behaviors: tck-stream-basic

Steps (1)

stream send_streaming_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-stream-basic with artifact
expect_stream
  events:
+    - description: At least one streamed event includes an artifact payload
+      match: any_position
+      artifact:         exists: true
+  final_event:
+    status:
+      state: TASK_STATE_COMPLETED
SHOULD STREAM-SSE-003 Chunked streaming delivers multiple artifact events
requires_behaviors: tck-stream-chunked

Steps (1)

stream send_streaming_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-stream-chunked multi-part
expect_stream
  min_count: 3
+  ordering: monotonic_state
SHOULD STREAM-SUB-001 SubscribeToTask receives updates for existing task
requires_behaviors: tck-long-running

Steps (2)

start send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-long-running start
+  configuration:
+    returnImmediately: true
expect
  task:
+    id:
+      type: string
+    status:
+      state:
+        one_of:
+          - TASK_STATE_SUBMITTED
+          - TASK_STATE_WORKING
capture
  taskId: task.id
subscribe subscribe_to_task
params
  id: {{start.taskId}}
expect_stream
  final_event:
+    status:
+      state:
+        one_of:
+          - TASK_STATE_COMPLETED
+          - TASK_STATE_FAILED
+          - TASK_STATE_CANCELED
+          - TASK_STATE_REJECTED
MUST STREAM-SUB-002 Subscribe stream terminates at terminal state

The stream must close after emitting the terminal state event.

requires_behaviors: tck-stream-basic

Steps (1)

stream send_streaming_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-stream-basic test
expect_stream
  final_event:
+    status:
+      state:
+        one_of:
+          - TASK_STATE_COMPLETED
+          - TASK_STATE_FAILED
+          - TASK_STATE_CANCELED
MUST STREAM-SUB-003 Subscribe to terminal task returns error
requires_behaviors: tck-complete-task

Steps (2)

setup send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task setup
expect
  task:
+    id:
+      type: string
+    status:
+      state:
+        one_of:
+          - TASK_STATE_COMPLETED
+          - TASK_STATE_FAILED
+          - TASK_STATE_CANCELED
+          - TASK_STATE_REJECTED
capture
  taskId: task.id
subscribe subscribe_to_task
params
  id: {{setup.taskId}}
expect
  error:
+    exists: true
MUST STREAM-SSE-004 First event on subscribe is current task state
requires_behaviors: tck-long-running

Steps (2)

start send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-long-running start
+  configuration:
+    returnImmediately: true
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
subscribe subscribe_to_task
params
  id: {{start.taskId}}
expect_stream
  events:
+    - description: First event represents the current task state
+      index: 0
+      one_of:         - task:             status:
+              exists: true
+        - status_update:             status:
+              exists: true
SHOULD STREAM-MSG-001 Streaming message-only response
requires_behaviors: tck-message-response

Steps (1)

stream send_streaming_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-message-response streaming
expect_stream
  min_count: 1
+  final_event:
+    message:
+      exists: true
SHOULD STREAM-MULTI-001 Multiple concurrent streams receive same events
streaming concurrent runner-special

Runner MUST open two concurrent streams and verify both receive the same events. This test may require special runner support.

requires_behaviors: tck-stream-basic

Steps (1)

stream send_streaming_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-stream-basic generate output
expect_stream
  min_count: 2
SHOULD STREAM-MULTI-002 Closing one stream doesn't affect others
streaming concurrent runner-special

Runner MUST open two concurrent streams, close one of them mid-stream, and verify the other continues receiving events. This test may require special runner support.

requires_behaviors: tck-stream-basic

Steps (1)

stream send_streaming_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-stream-basic generate output
expect_stream
  min_count: 2
SHOULD STREAM-RESUB-001 Resubscribe after disconnect
streaming resubscribe runner-special

Runner MUST disconnect from stream mid-flight and resubscribe. This test may require special runner support.

requires_behaviors: tck-long-running

Steps (2)

start send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-long-running start
+  configuration:
+    returnImmediately: true
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
capture
  taskId: task.id
subscribe subscribe_to_task
params
  id: {{start.taskId}}
expect_stream
  min_count: 1
+
JSON-RPC transport bindingstransport-bindings.acts.yaml3 tests
MUST JSONRPC-ENV-001 Response has valid JSON-RPC 2.0 envelope
transport jsonrpc raw envelope
transport: jsonrpc
requires_behaviors: tck-complete-task

Steps (1)

send-raw RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+    A2A-Version: 1.0
+  body:
+    jsonrpc: 2.0
+    id: jsonrpc-env-001
+    method: SendMessage
+    params:
+      message:
+        role: ROLE_USER
+        parts:
+          - text: tck-complete-task verify jsonrpc envelope
expect
  status: 200
+  body:
+    jsonrpc: 2.0
+    id: jsonrpc-env-001
+    result:
+      exists: true
MUST JSONRPC-CT-001 Response Content-Type is application/json
transport jsonrpc raw content-type

ACTS cannot reliably assert response headers in this raw format, so the test uses a valid JSON-RPC body as a proxy for the expected application/json response.

transport: jsonrpc
requires_behaviors: tck-complete-task

Steps (1)

send-raw RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+    A2A-Version: 1.0
+  body:
+    jsonrpc: 2.0
+    id: jsonrpc-ct-001
+    method: SendMessage
+    params:
+      message:
+        role: ROLE_USER
+        parts:
+          - text: tck-complete-task verify content type
expect
  status: 200
+  body:
+    jsonrpc: 2.0
+    id: jsonrpc-ct-001
+    result:
+      exists: true
MUST JSONRPC-SSE-001 SSE streaming events have JSON-RPC envelope
transport jsonrpc raw sse streaming
transport: jsonrpc
requires_behaviors: tck-stream-basic

Steps (1)

stream-raw RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+    Accept: text/event-stream
+    A2A-Version: 1.0
+  body:
+    jsonrpc: 2.0
+    id: jsonrpc-sse-001
+    method: SendStreamingMessage
+    params:
+      message:
+        role: ROLE_USER
+        parts:
+          - text: tck-stream-basic generate output
expect
  status: 200
expect_stream
  min_count: 2
+  each_event:
+    jsonrpc: 2.0
+
REST transport bindingstransport-bindings.acts.yaml3 tests
MUST REST-STATUS-001 REST errors map to correct HTTP status codes
transport rest raw status
transport: rest

Steps (1)

get-missing RAW GET /tasks/nonexistent-id
raw
  method: GET
+  path: /tasks/nonexistent-id
+  headers:
+    Accept: application/json
+    A2A-Version: 1.0
expect
  status: 404
SHOULD REST-PD-001 REST errors use problem details format
transport rest raw problem-details
transport: rest

Steps (1)

get-missing-problem RAW GET /tasks/nonexistent-id
raw
  method: GET
+  path: /tasks/nonexistent-id
+  headers:
+    Accept: application/json
+    A2A-Version: 1.0
expect
  status: 404
+  body:
+    type:
+      type: string
+    title:
+      type: string
+    status:
+      type: number
SHOULD REST-CT-001 REST responses use application/a2a+json content type
transport content-type runner-special

Runner SHOULD verify the Content-Type response header is application/a2a+json for REST responses.

transport: rest
requires_behaviors: tck-complete-task

Steps (1)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task verify rest content type
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state: TASK_STATE_COMPLETED
+
gRPC transport bindingstransport-bindings.acts.yaml3 tests
MUST GRPC-STATUS-001 gRPC errors map to correct status codes
transport grpc grpc-status-verification

Verifies that missing-task errors surface through the gRPC binding with the expected status semantics (for example, NOT_FOUND) even though the ACTS assertion is expressed as an abstract TaskNotFoundError.

transport: grpc

Steps (1)

get-missing get_task
params
  id: 00000000-0000-0000-0000-000000000000
expect_error
  code: TaskNotFoundError
MUST GRPC-STREAM-001 gRPC streaming delivers proper message structure
transport grpc streaming
transport: grpc
requires_behaviors: tck-stream-basic

Steps (1)

stream send_streaming_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-stream-basic generate output
expect_stream
  min_count: 2
SHOULD GRPC-STREAM-002 gRPC streaming cancellation works
transport grpc streaming cancel
transport: grpc
requires_behaviors: tck-cancel

Steps (2)

start send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-cancel start
+  configuration:
+    returnImmediately: true
expect
  status: 200
+  body:
+    task:
+      id:
+        type: string
+      status:
+        state:
+          one_of:
+            - TASK_STATE_SUBMITTED
+            - TASK_STATE_WORKING
capture
  taskId: task.id
cancel cancel_task
params
  id: {{start.taskId}}
expect
  status: 200
+  body:
+    id: {{start.taskId}}
+    status:
+      state: TASK_STATE_CANCELED
+
Version Negotiationversion-negotiation.acts.yaml2 tests

Coverage for unsupported and omitted A2A-Version header handling.

MUST VER-NEG-001 Unsupported A2A-Version returns VersionNotSupportedError

Verifies that a JSON-RPC request with an unsupported A2A-Version header is rejected.

spec_ref: docs/specification.md
transport: jsonrpc

Steps (1)

send-unsupported-version RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+    A2A-Version: 99.0
+  body:
+    jsonrpc: 2.0
+    id: ver-neg-001
+    method: SendMessage
+    params:
+      message:
+        role: ROLE_USER
+        parts:
+          - text: version negotiation unsupported
expect
  status: 200
+  body:
+    error:
+      code: -32009
+      message:
+        type: string
SHOULD VER-NEG-002 Missing A2A-Version is treated as default

Verifies that a JSON-RPC request without an A2A-Version header succeeds using the server's default version behavior.

spec_ref: docs/specification.md
transport: jsonrpc
requires_behaviors: tck-complete-task

Steps (1)

send-without-version RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+  body:
+    jsonrpc: 2.0
+    id: ver-neg-002
+    method: SendMessage
+    params:
+      message:
+        role: ROLE_USER
+        parts:
+          - text: tck-complete-task version default
expect
  status: 200
+  body:
+    result:
+      task:
+        id:
+          type: string
+
Wire Format Compliancewire-format.acts.yaml3 tests
MUST DM-FMT-001 Wire format uses camelCase field names

Sends a raw JSON-RPC SendMessage request and verifies the wire-format response does not expose snake_case field names. +

spec_ref: specification.md#2.3
transport: jsonrpc
requires_behaviors: tck-complete-task

Steps (1)

send-raw RAW POST /
raw
  method: POST
+  path: /
+  headers:
+    Content-Type: application/json
+    A2A-Version: 1.0
+  body:
+    jsonrpc: 2.0
+    id: wire-format-001
+    method: SendMessage
+    params:
+      message:
+        role: ROLE_USER
+        parts:
+          - text: tck-complete-task verify wire format
expect
  status: 200
+  body:
+    result:
+      task:
+        id:
+          type: string
+    context_id:
+      absent: true
+    message_id:
+      absent: true
+    task_id:
+      absent: true
+    artifact_id:
+      absent: true
+    status_update:
+      absent: true
+    artifact_update:
+      absent: true
+    protocol_version:
+      absent: true
+    media_type:
+      absent: true
MUST DM-FMT-002 Enum values use prefixed format

Verifies enum values are returned using the prefixed wire-format names for message roles and task states. +

spec_ref: specification.md#2.3
requires_behaviors: tck-message-response, tck-complete-task

Steps (2)

check-role send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-message-response check role
expect
  status: 200
+  body:
+    message:
+      role: ROLE_AGENT
check-state send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-complete-task check state
expect
  status: 200
+  body:
+    task:
+      status:
+        state: TASK_STATE_COMPLETED
SHOULD DM-FMT-003 Default and empty values are omitted from wire

Verifies message responses omit default taskId and contextId fields, or at minimum never serialize them as empty strings. +

spec_ref: specification.md#2.3
requires_behaviors: tck-message-response

Steps (1)

send send_message
params
  message:
+    role: ROLE_USER
+    parts:
+      - text: tck-message-response test defaults
expect
  status: 200
+  body:
+    message:
+      role: ROLE_AGENT
+      taskId:
+        absent: true
+      contextId:
+        absent: true
+
+ + + + \ No newline at end of file diff --git a/tests/acts/transport-bindings.acts.yaml b/tests/acts/transport-bindings.acts.yaml new file mode 100644 index 000000000..cb58ce617 --- /dev/null +++ b/tests/acts/transport-bindings.acts.yaml @@ -0,0 +1,245 @@ +acts_version: "1.0" +spec_version: "1.0" + +metadata: + title: "A2A Transport Binding Conformance Tests" + description: "Transport-specific ACTS coverage for JSON-RPC, REST, and gRPC bindings." + +suites: + - id: jsonrpc-transport + name: "JSON-RPC transport bindings" + tests: + - id: JSONRPC-ENV-001 + name: "Response has valid JSON-RPC 2.0 envelope" + level: must + tags: [transport, jsonrpc, raw, envelope] + transport: [jsonrpc] + requires_behaviors: ["tck-complete-task"] + steps: + - id: send-raw + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + A2A-Version: "1.0" + body: + jsonrpc: "2.0" + id: "jsonrpc-env-001" + method: SendMessage + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task verify jsonrpc envelope" + expect: + status: 200 + body: + jsonrpc: "2.0" + id: "jsonrpc-env-001" + result: + exists: true + + - id: JSONRPC-CT-001 + name: "Response Content-Type is application/json" + description: "ACTS cannot reliably assert response headers in this raw format, so the test uses a valid JSON-RPC body as a proxy for the expected application/json response." + level: must + tags: [transport, jsonrpc, raw, content-type] + transport: [jsonrpc] + requires_behaviors: ["tck-complete-task"] + steps: + - id: send-raw + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + A2A-Version: "1.0" + body: + jsonrpc: "2.0" + id: "jsonrpc-ct-001" + method: SendMessage + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task verify content type" + expect: + status: 200 + body: + jsonrpc: "2.0" + id: "jsonrpc-ct-001" + result: + exists: true + + - id: JSONRPC-SSE-001 + name: "SSE streaming events have JSON-RPC envelope" + level: must + tags: [transport, jsonrpc, raw, sse, streaming] + transport: [jsonrpc] + requires_behaviors: ["tck-stream-basic"] + steps: + - id: stream-raw + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + Accept: text/event-stream + A2A-Version: "1.0" + body: + jsonrpc: "2.0" + id: "jsonrpc-sse-001" + method: SendStreamingMessage + params: + message: + role: ROLE_USER + parts: + - text: "tck-stream-basic generate output" + expect: + status: 200 + expect_stream: + min_count: 2 + each_event: + jsonrpc: "2.0" + + - id: rest-transport + name: "REST transport bindings" + tests: + - id: REST-STATUS-001 + name: "REST errors map to correct HTTP status codes" + level: must + tags: [transport, rest, raw, status] + transport: [rest] + steps: + - id: get-missing + raw: + method: GET + path: "/tasks/nonexistent-id" + headers: + Accept: application/json + A2A-Version: "1.0" + expect: + status: 404 + + - id: REST-PD-001 + name: "REST errors use problem details format" + level: should + tags: [transport, rest, raw, problem-details] + transport: [rest] + steps: + - id: get-missing-problem + raw: + method: GET + path: "/tasks/nonexistent-id" + headers: + Accept: application/json + A2A-Version: "1.0" + expect: + status: 404 + body: + type: + type: string + title: + type: string + status: + type: number + + - id: REST-CT-001 + name: "REST responses use application/a2a+json content type" + description: "Runner SHOULD verify the Content-Type response header is application/a2a+json for REST responses." + level: should + tags: [transport, content-type, runner-special] + transport: [rest] + requires_behaviors: ["tck-complete-task"] + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task verify rest content type" + expect: + status: 200 + body: + task: + id: + type: string + status: + state: TASK_STATE_COMPLETED + + - id: grpc-transport + name: "gRPC transport bindings" + tests: + - id: GRPC-STATUS-001 + name: "gRPC errors map to correct status codes" + description: "Verifies that missing-task errors surface through the gRPC binding with the expected status semantics (for example, NOT_FOUND) even though the ACTS assertion is expressed as an abstract TaskNotFoundError." + level: must + tags: [transport, grpc, grpc-status-verification] + transport: [grpc] + steps: + - id: get-missing + operation: get_task + params: + id: "00000000-0000-0000-0000-000000000000" + expect_error: + code: TaskNotFoundError + + - id: GRPC-STREAM-001 + name: "gRPC streaming delivers proper message structure" + level: must + tags: [transport, grpc, streaming] + transport: [grpc] + requires_behaviors: ["tck-stream-basic"] + steps: + - id: stream + operation: send_streaming_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-stream-basic generate output" + expect_stream: + min_count: 2 + + - id: GRPC-STREAM-002 + name: "gRPC streaming cancellation works" + level: should + tags: [transport, grpc, streaming, cancel] + transport: [grpc] + requires_behaviors: ["tck-cancel"] + steps: + - id: start + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-cancel start" + configuration: + returnImmediately: true + expect: + status: 200 + body: + task: + id: + type: string + status: + state: + one_of: + - TASK_STATE_SUBMITTED + - TASK_STATE_WORKING + capture: + taskId: "task.id" + + - id: cancel + operation: cancel_task + params: + id: "{{start.taskId}}" + expect: + status: 200 + body: + id: "{{start.taskId}}" + status: + state: TASK_STATE_CANCELED diff --git a/tests/acts/version-negotiation.acts.yaml b/tests/acts/version-negotiation.acts.yaml new file mode 100644 index 000000000..add830eff --- /dev/null +++ b/tests/acts/version-negotiation.acts.yaml @@ -0,0 +1,75 @@ +acts_version: "1.0" +spec_version: "1.0" +spec_ref: "docs/specification.md" + +metadata: + title: "Version Negotiation ACTS" + description: "ACTS tests for A2A-Version negotiation behavior over JSON-RPC." + +suites: + - id: version-negotiation + name: "Version Negotiation" + description: "Coverage for unsupported and omitted A2A-Version header handling." + tests: + - id: VER-NEG-001 + name: "Unsupported A2A-Version returns VersionNotSupportedError" + description: "Verifies that a JSON-RPC request with an unsupported A2A-Version header is rejected." + spec_ref: "docs/specification.md" + level: must + transport: [jsonrpc] + steps: + - id: send-unsupported-version + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + A2A-Version: "99.0" + body: + jsonrpc: "2.0" + id: "ver-neg-001" + method: SendMessage + params: + message: + role: ROLE_USER + parts: + - text: "version negotiation unsupported" + expect: + status: 200 + body: + error: + code: -32009 + message: + type: string + + - id: VER-NEG-002 + name: "Missing A2A-Version is treated as default" + description: "Verifies that a JSON-RPC request without an A2A-Version header succeeds using the server's default version behavior." + spec_ref: "docs/specification.md" + level: should + transport: [jsonrpc] + requires_behaviors: + - "tck-complete-task" + steps: + - id: send-without-version + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + body: + jsonrpc: "2.0" + id: "ver-neg-002" + method: SendMessage + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task version default" + expect: + status: 200 + body: + result: + task: + id: + type: string diff --git a/tests/acts/wire-format.acts.yaml b/tests/acts/wire-format.acts.yaml new file mode 100644 index 000000000..7fcc3f90c --- /dev/null +++ b/tests/acts/wire-format.acts.yaml @@ -0,0 +1,122 @@ +acts_version: "1.0" +spec_version: "1.0" + +metadata: + title: "Wire Format Compliance" + description: "Tests for camelCase field naming, enum value format, and default value omission" + +suites: + - id: wire-format + name: "Wire Format Compliance" + tests: + - id: DM-FMT-001 + name: "Wire format uses camelCase field names" + description: > + Sends a raw JSON-RPC SendMessage request and verifies the wire-format + response does not expose snake_case field names. + spec_ref: "specification.md#2.3" + level: must + transport: [jsonrpc] + requires_behaviors: ["tck-complete-task"] + steps: + - id: send-raw + raw: + method: POST + path: "/" + headers: + Content-Type: application/json + A2A-Version: "1.0" + body: + jsonrpc: "2.0" + id: "wire-format-001" + method: SendMessage + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task verify wire format" + expect: + status: 200 + body: + result: + task: + id: + type: string + context_id: + absent: true + message_id: + absent: true + task_id: + absent: true + artifact_id: + absent: true + status_update: + absent: true + artifact_update: + absent: true + protocol_version: + absent: true + media_type: + absent: true + + - id: DM-FMT-002 + name: "Enum values use prefixed format" + description: > + Verifies enum values are returned using the prefixed wire-format names + for message roles and task states. + spec_ref: "specification.md#2.3" + level: must + requires_behaviors: ["tck-message-response", "tck-complete-task"] + steps: + - id: check-role + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-message-response check role" + expect: + status: 200 + body: + message: + role: ROLE_AGENT + + - id: check-state + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-complete-task check state" + expect: + status: 200 + body: + task: + status: + state: TASK_STATE_COMPLETED + + - id: DM-FMT-003 + name: "Default and empty values are omitted from wire" + description: > + Verifies message responses omit default taskId and contextId fields, + or at minimum never serialize them as empty strings. + spec_ref: "specification.md#2.3" + level: should + requires_behaviors: ["tck-message-response"] + steps: + - id: send + operation: send_message + params: + message: + role: ROLE_USER + parts: + - text: "tck-message-response test defaults" + expect: + status: 200 + body: + message: + role: ROLE_AGENT + taskId: + absent: true + contextId: + absent: true