Architectural direction for CQL-with-FHIR, `$evaluate`, and shared fixtures

[preston]

Today, the CQL testing projects only address expression-level CQL via $cql, with no retrieves and no patient/model data in use. That was the great starting point, but is not enough for the next tier of conformance: CQL with FHIR (Using CQL with FHIR) and $evaluate over a FHIR model.

In addition to the absense of retrieval-oriented $cql expressions, queries using FHIR imply a baseline set of test fixture data—stable, versioned resources that engines can load so retrieve-based tests are comparable across implementations. This directly cross-cuts across:

• cql-tests
• cql-tests-runner
• cql-tests-results
• cql-studio

Any strategy should respect that these projects have different governance controls and stakeholders.

Proposed Goals

1. Support optional CQL-with-FHIR testing scenarios (retrieves, terminology hooks as needed, FHIR type mapping, various parameter combinations).
2. Test FHIR $evaluate (Library- and Measure-based), not only $cql.
3. Ship a small, curated baseline fixture set that tests can reference by some form of scenario preset identifier
4. Keep authorship, execution, and result publication cleanly separated.

Goals to keep in mind for initial refactoring, but out of scope for initial refactoring: full measure evaluation, arbitrary large populations, or replacing vendor sandboxes.

Some Options for Consideration

A. Keep fixtures in cql-tests next to the cases

Tests declare which fixture scenario set (or Bundle) they need; runner loads it before $evaluate.

• Pros: Single source of truth for “what this test means.” Versioned with the cases. Easy for authors.
• Cons: cql-tests projects grows much, MUCH heavier. Runner (and studio) must know how to load/seed. Coupling of FHIR-specific test data files into the schema/tests repo.

B. Own fixtures in cql-tests-runner

Runner ships or fetches fixtures and is responsible for load/teardown against the target FHIR server.

• Pros: Execution concerns stay in the executor. cql-tests stays lean (logic + expectations only).
• Cons: Needlessly creates test/data drift across repos. Studio/other consumers duplicate load logic or depend on the runner. Harder for pure test authors who never touch the runner.

C. New small repo (e.g. cql-tests-fhir)

Versioned Bundles/ndjson + a manifest mostly centered on CQL over the FHIR data model and CQL-with-FHIR IG testing resources.

• Pros: Clear ownership and seperation of FHIR. Reusable by runner, studio, CI, and external engines. Independent release cadence. Schema doesn't have to match the one used for $cql tests
• Cons: Yet another repo and moving part. Pinning/submodules/coordination overhead. Easy to under-maintain if nobody “owns” it day to day.

D. Publish fixtures as a FHIR package / npm (or similar) artifact

Same content as C, but consumed as a package rather than a git submodule.

• Pros: Clean dependency story; semver; works well in CI and Docker. Studio and runner install the same thing.
• Cons: Packaging/release process to stand up. Less friendly for drive-by PR edits to a single Patient resource.

E. Minimal fixtures in cql-tests; richer sets elsewhere later

Start with a tiny shared baseline in cql-tests (enough for first $evaluate + retrieve tests). Defer bulk/synthetic data to a fixtures package or runner-only packs once we know the shape.

• Pros: Unblocks architecture work without boiling the ocean. Forces us to design the reference/manifest early.
• Cons: Temporary dual homes if we outgrow the baseline. Need discipline so “temporary” does not become permanent sprawl.

Runner implications (regardless of home)

• Config for $cql vs $evaluate (we already hint at $evaluate in places but it needs to be implemented).
• Pre-flight: load fixture Bundle(s) setup into the target server (or document that the server must already contain them).
• Clear failure modes when data is missing vs when CQL is wrong.
• Results payload should record fixture set id/version so published runs in cql-tests-results are interpretable.
• Possibly some sort of server "teardown" semantic for re-seeding