Add declarative YAML JSON and JSONL schemas with enforced memory validation #67435

Goal

Allow workflows to declare JSON and JSONL schemas directly in YAML frontmatter and require gh-aw to enforce them automatically. Authors must not need a validation.script or a helper call to activate schema validation.

Related: #58144. This is a concrete implementation plan for its native declarative approach, backed by the validator already shipped in actions/setup/js rather than a new dependency.

Updated requirement from pelikhan: YAML schema declarations with mandatory validation are the primary feature. Script-only helpers do not satisfy this issue and are not a prerequisite.

Do not use Ajv, ajv-cli, a new schema-validator dependency, runtime package downloads, or a generated validator bundle. Preserve pre-existing dependencies used by unrelated tooling.

Proposed YAML interface

Add validation.json-schemas as a list of per-file declarations alongside the existing optional validation.script and validation.timeout-minutes. Each declaration has an exact relative file, an explicit format (json or jsonl), and an inline schema object. Declaring an entry requires that file to exist and pass validation; do not introduce an opt-out flag or silently skip missing files. Start with exact file paths, not globs or external schema files.

tools:
  repo-memory:
    allowed-extensions: [".json", ".jsonl"]
    validation:
      json-schemas:
        - file: state.json
          format: json
          schema:
            type: object
            required: [version, items]
            additionalProperties: false
            properties:
              version:
                enum: [1]
              items:
                type: array
                items:
                  type: string
        - file: archive/events.jsonl
          format: jsonl
          schema:
            type: object
            required: [id, status]
            additionalProperties: false
            properties:
              id:
                type: integer
              status:
                enum: [open, closed]

The same declaration must work for cache-memory and drive-memory. This example intentionally contains no script: schema validation must still run and gate acceptance/persistence.

Implementation plan

1. Define and validate frontmatter. Extend memory validation configuration types, parsing, and the frontmatter schema for json-schemas. Reject unknown entry fields, invalid formats, empty declarations, duplicate file targets, absolute/traversing paths, invalid schemas, and schema targets excluded by configured extension/glob policies. Preserve existing script-only workflows. Check supported schema vocabulary at compile time so ignored constraints cannot become a false guarantee.
2. Reuse existing runtime validation. Reuse validateValueAgainstSchema in actions/setup/js/mcp_scripts_validation.cjs. Extract and share the supported-schema guard currently inside compileSchema in actions/setup/js/ledger_store.cjs, preserving ledger behavior. Keep the compiler and runtime support checks consistent using shared conformance fixtures. Do not build a second general-purpose validator.
3. Carry trusted schemas through compilation. Serialize inline YAML schemas and file/format declarations into the compiled workflow/runtime configuration using existing safe transport patterns. Wire every validation surface, including the agent-time repo-memory safe-output/MCP validation path, post-agent upload/cache/drive validation, and the repo-memory push job. Schema-only configuration must create the necessary validation steps and hard gates even when validation.script is absent. Validation must not depend on a source checkout or schemas stored in agent-editable memory.
4. Enforce declarative validation automatically. Validate the complete configured candidate file after existing filtering/normalization and relevant temporary-ID substitution. Fail if a declared file is missing, unreadable, malformed, or violates its schema. Run schema validation before optional custom validation.script, which remains available for domain/cross-file invariants. At runtime, recheck schema configuration and reject missing, malformed, or unsupported transported configuration explicitly; never fall back to skipping validation.
5. Preserve persistence safety and consistency. Retain temporary-copy validation, credential filtering, timeout/resource limits, output bounds/redaction, and mutation detection. Return actionable schema errors from the recoverable agent-time validation path. Hard failures must block artifact upload/cache or drive acceptance and repo-memory commit/push. Ensure the final candidate that is actually persisted is validated, including after reconciliation if that changes the candidate. Preserve existing retry/merge mechanisms and do not broaden scope to invalid-baseline recovery policy.
6. Document, test, and publish. Update repo-memory, cache-memory, and drive-memory references and .github/aw/memory.md; add a patch changeset. Document required-file semantics, the limited vocabulary, JSONL record behavior, ordering, and optional script composition. Run targeted Go/compiler and JavaScript tests, type checking, lint/format checks, and repository-required validation. Compile the exact schema-only example and exercise runtime acceptance/rejection. Publish one draft PR linked to this issue using repository-approved tools. Do not trigger workflow runs.

Supported schema contract

Preserve the existing simplified validator's enforceable subset:

• type, including arrays of types and the string "null".
• Primitive enum, required, nested properties, and additionalProperties: false.
• Object-schema items and standalone oneOf or anyOf.
• Existing 32-level schema nesting bound; reject alternatives combined with sibling constraints.

Schemas must be objects. Reject unsupported keywords explicitly at compile time and defensively at runtime, even for empty JSONL, including $ref, const, numeric/string/array bounds, patterns, uniqueItems, and format within a schema. The declaration's format: jsonl selects file parsing and is distinct from the unsupported JSON Schema format keyword. Do not claim full JSON Schema draft support or expand the validator vocabulary in this task. Positive-ID checks, item-count limits, timestamps, uniqueness, and cross-file relationships may remain in the optional custom script.

File and diagnostic semantics

• JSON validates one document. Reject empty, whitespace-only, and malformed documents.
• JSONL validates each record independently, not the array of records. Accept LF/CRLF, an optional final newline, and an empty existing file as zero records. Reject blank physical record lines. A missing declared file always fails, even though an empty JSONL file may pass.
• Resolve declared relative paths within memoryRoot and reject traversal, external symlinks, directories, missing files, and unreadable files with contextual diagnostics.
• Schema validation must not coerce, default, strip, or rewrite data. Existing independently configured normalization remains unchanged.
• Failures identify the memory kind/ID and file; schema failures include the existing validator's failing field path; JSONL errors include the physical line number. Preserve repository-standard failure outputs and redaction.
• Inline workflow schemas are trusted configuration, not agent-editable memory data. External references, schema files, file globs, and runtime downloads are out of scope for this first implementation.

Acceptance criteria

• A workflow can declare inline schemas in YAML and enforce them with no custom JavaScript. The schema-only example compiles and generates all required validation configuration and gates.
• Compiler tests cover valid declarations, YAML scalar/type handling, invalid formats and paths, duplicate/excluded targets, unknown fields, invalid schemas, unsupported constraints, and nesting bounds. No unsupported constraint is silently accepted.
• Valid nested JSON/JSONL passes unchanged; invalid types, enums, required fields, extra properties, and nested items fail with contextual diagnostics. JSONL tests cover physical line numbers, LF/CRLF, final newlines, empty files, blank lines, and malformed records.
• Missing declared files fail explicitly. Runtime configuration loss/corruption and unsupported schemas fail closed, including for empty JSONL. Path-containment tests cover traversal, directories, and external symlinks.
• Declarative validation runs consistently in repo-memory, cache-memory, and drive-memory, including repo-memory MCP validation, pre-upload validation, and push-job validation. Tests prove schema-only configurations cannot bypass gates because script is absent.
• Invalid candidate memory is never uploaded/accepted or committed/pushed. The persisted candidate is checked after transformations and any reconciliation that changes it. Existing filtered/nested-memory behavior remains intact.
• When both declarative schemas and a script exist, schemas run first; a script failure still rejects persistence. Existing script-only and no-validation configurations preserve their behavior.
• Runtime validation works with the copied setup files without node_modules, credentials, network access, or an additional source checkout. Existing timeout and non-mutation protections remain covered.
• Existing schema-validator and ledger compatibility tests pass. Compare unrelated failures against baseline; do not silently skip them or expand scope to fix unrelated defects.
• Focused compiler/runtime tests, JavaScript type checking, lint/format checks, and repository-required validation pass. The exact documented YAML example is compiled and its runtime behavior is exercised with both valid and invalid data.
• No new validator dependencies, runtime downloads, or generated validator bundles are introduced. Docs and a patch changeset describe the implemented declarative API and limited vocabulary accurately.

Handoff context

An earlier local no-Ajv script-helper implementation was prepared in the originating Slack conversation, but no branch was pushed and no PR was created. It does not satisfy the updated declarative requirement on its own. Implement from main; do not assume access to that sandbox or its patch. This issue is an implementation plan, not a launched Copilot agent task.