Support multi-file spec loading, --entity filter, and --emit-spec

[amc-corey-cox]

Summary

Add the ability to load and merge multiple transformation spec files, filter execution by entity class, and emit the resolved spec.

Multi--T spec loading

Both map-data and validate-spec should accept multiple -T flags (and/or directories) and merge them into a single TransformationSpecification at load time.

Merge semantics:

• class_derivations: append (order preserved per file, files in argument order)
• enum_derivations: union by name (error on conflict)
• slot_derivations: union by name (error on conflict)

This enables modular spec authoring — keeping enum derivations in a shared file, splitting class derivations by domain, or maintaining per-variable specs that get merged at load time.

linkml-map map-data -T enums.yaml -T measurements/*.yaml -s schema.yaml input/
linkml-map validate-spec -T specs/

--entity filter

Both map-data and validate-spec should accept --entity <class_name> to restrict processing to class_derivations matching that name. Only filters top-level class_derivations — nested object_derivations (e.g., Quantity inside MeasurementObservation) are unaffected and process normally as part of their parent.

linkml-map map-data -T specs/*.yaml --entity MeasurementObservation -s schema.yaml input/

This enables external parallelization (e.g., make -j) by running filtered transforms concurrently, and is useful for debugging individual entity transforms.

--emit-spec

• On validate-spec: emit the resolved (merged + filtered) spec to stdout. Enables inspection of exactly what linkml-map would execute.
• On map-data: emit the resolved spec to a file path (--emit-spec merged.yaml) as a side-effect alongside normal transformation. Useful for reproducibility logging.

# Preview the merged + filtered spec
linkml-map validate-spec -T specs/*.yaml --entity MeasurementObservation --emit-spec

# Transform and log what spec was used
linkml-map map-data -T specs/*.yaml --entity Foo --emit-spec resolved.yaml -s schema.yaml input/

Motivation

Transformation specs for large-scale ETL pipelines are naturally modular — per-variable, per-domain, or with shared enum definitions. Today, users must pre-compose these into a single file externally before passing to linkml-map. This is particularly painful for enum_derivations which need to be shared across entity transforms but duplicated into each composed file.

This follows the same pattern as other declarative config systems (Docker Compose -f, LinkML schema imports, Terraform modules): composable parts, merged at load time.

Design notes

• The merge logic lives in spec loading (shared by both commands), not as a separate CLI command
• --entity is distinct from --source-type (which overrides the type name passed to map_object, not which class_derivations are executed)
• validate-spec with --entity + --emit-spec serves as a complete preview of what map-data would execute

[github-actions]

Issue Summary

This is a well-scoped CLI enhancement proposal from @amc-corey-cox (who also contributed #162 --continue-on-error, now merged), targeting large-scale ETL pipeline use cases. Three related features are bundled together:

---

Feature 1: Multi--T spec loading

Allow map-data and validate-spec to accept multiple -T flags and/or a directory path, merging them into one TransformationSpecification at load time.

Merge semantics (well-matched to the existing data model):

• class_derivations → append (already a list[ClassDerivation] in the model, so this is natural)
• enum_derivations → dict union by name, error on conflict (already a dict[str, EnumDerivation])
• slot_derivations → dict union by name, error on conflict (already a dict[str, SlotDerivation])

Current state: -T is a single-valued Click option (transformer_specification_option at cli.py:41). validate-spec currently takes only positional args and validates files individually — no merging occurs.

---

Feature 2: --entity filter

Restrict map-data and validate-spec to only the named class_derivation(s). Nested object_derivations inside a matching parent are unaffected.

Primary use cases:

• External parallelization with make -j across entity types
• Faster debugging of individual entity transforms

Note: this is explicitly distinct from --source-type, which overrides the type name passed to map_object rather than filtering which derivations are loaded.

---

Feature 3: --emit-spec

Dump the resolved (merged + filtered) spec for inspection or reproducibility logging:

• validate-spec --emit-spec → emit to stdout (flag)
• map-data --emit-spec resolved.yaml → emit to file path as a side-effect (option)

Combined with --entity, validate-spec -T specs/*.yaml --entity Foo --emit-spec gives a complete preview of exactly what map-data would execute.

---

Implementation notes

• Merge logic should live in a shared loader utility (not duplicated across commands), called before the transformer is initialized.
• Click change: -T needs multiple=True to accept multiple values; also needs to accept directories (glob/expand at load time).
• validate-spec harmonization: currently uses positional args (spec_files tuple), not -T; this proposal unifies both commands under -T flag semantics.
• load_transformer_specification (transformer/transformer.py:100) currently takes a single path — the multi-file merge would wrap or replace this call at the CLI layer.

Related issues

• Add --continue-on-error flag to map-data CLI #162 (closed ✅) — --continue-on-error for map-data, same contributor, same ETL pipeline motivation

Pattern precedent

The design follows the same composability pattern as Docker Compose -f, LinkML schema imports, and Terraform modules: independent declarative parts merged at load time, without requiring external pre-composition.