Back-port FOCUS 1.4 columns from older export versions during ingestion

[flanakin]

Context

Follow-up from PR #2126 (FOCUS 1.4 ingestion). The FOCUS 1.4 schema introduces several columns that older FOCUS export versions do not carry. In IngestionSetup_v1_4.kql, Costs_transform_v1_4() currently leaves these null/empty when ingesting older exports, because no source-data mapping exists yet. FOCUS 1.4 requires these to be populated where derivable.

Gaps to back-port

Tracked from PR #2126 review threads:

• CommitmentProgramEligibilityDetails (thread 3309107501) — likely needs the CommitmentDiscountEligibility open dataset to generate.
• ContractCommitment* per-row columns (thread 3309131626) — flatten from contract-commitment source where available.
• InvoiceDetailId (thread 3309133481) — derive/link from invoice source.

Scope

Research the source-data mappings and implement the back-fill in the _v1_4() transforms (and equivalent hub functions). Until then, these columns remain null when ingesting pre-1.4 exports — a known gap.

Related

• Research: Handle FOCUS 1.4 native-columns requirement to avoid raw table bloat #2159 — native-columns bloat (distinct: unknown/native columns, not known FOCUS 1.4 columns)
• PR Add FOCUS 1.4 ingestion (IngestionSetup_v1_4.kql) #2126 — FOCUS 1.4 ingestion

[microsoft-github-policy-service]

Thank you for submitting this issue! We try to respond within 3 business days. In the meantime:

• 📖 Check the troubleshooting guide for common issues
• 💬 Ask questions in Discussions
• 📅 Join our biweekly office hours for live help from the team

[flanakin]

Closing — consolidating all FOCUS 1.4 tracking into the umbrella (PR #2207 / FOCUS-1.4-PLAN.md) per maintainer direction. The back-port work is now Q5/W11 in the plan, owned by concrete PR(s) in the stack rather than a standalone research issue.