Skip to content

nyc-taxi: add_metadata.py tag/term attachments overwrite each other; ownership points to a group that is never created #218

Description

@simpleciki

Environment: DataHub quickstart (server v1.5.0.6, CLI acryl-datahub 1.6.0.16), Windows 11, Docker Desktop 4.84. Dataset at edbfb6d (2026-07-23). Verified via GMS REST API end-state and the UI — script log output was not used as evidence.

Bug 1: tag/term attachments silently overwrite each other (last-write-wins)

attach_tags() emits a full GlobalTagsClass aspect containing a single tag per (tag, table) pair:

aspect=GlobalTagsClass(tags=[TagAssociationClass(tag=f"urn:li:tag:{tag}")])

MCP emit is upsert-with-replace at aspect granularity, so each emit replaces the previous one. The last tag in TAG_ASSIGNMENTS iteration order (pipeline_stage) wins on every table. attach_glossary() has the same pattern.

Result after python add_metadata.py --all (exit 0, log prints ✓ for every attachment): all six tables across both instances end with exactly tags: [pipeline_stage] and terms: [pipeline_stage], instead of the documented 2–4 tags / 2–3 terms per table (README "Metadata" section). Notably:

  • daily_refresh — the anchor of the freshness scenario — is attached to zero tables
  • Empty Load is attached to zero tables, including mart_daily_summary where its own definition says it belongs

The tag/term entities are all created correctly; only the attachments are lost.

Verify: GET /openapi/v3/entity/dataset/{urn}globalTags / glossaryTerms, or open any of the six tables in the UI.

Suggested fix: accumulate per table and emit once per table with the full list, or use patch semantics (DatasetPatchBuilder.add_tag / add_glossary_term). Happy to send a PR.

Bug 2: ownership references a corpGroup that is never created

emit_ownership() attaches urn:li:corpGroup:data_platform_team as DATAOWNER to all tables, but nothing ever creates the corpGroup entity (the script has create_tags() and create_glossary() but no group creation).

Result: GET /openapi/v3/entity/corpGroup → 0 entities, while every table's ownership aspect references the group. Clicking the owner chip in the UI → "Not Found — Sorry, we are unable to find this entity in DataHub."

Suggested fix: emit corpGroupInfo for data_platform_team before attaching ownership. (Included in the same PR.)

Bug 3 (minor): --dry-run prints "✅ Metadata complete" and previews nothing

With --dry-run the script lists discovered dataset URNs, skips every Creating/Attaching code path, then unconditionally prints the same "✅ Metadata complete" banner as a real run — no preview of what would change, and a misleading completion claim. (Confirmed it performs no writes.) add_lineage.py --dry-run already does this right ("DRY RUN — 8 lineage edges would be created").

Suggested fix: print the planned tag/term/ownership operations and a "DRY RUN — no changes made" banner. (Included in the same PR.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions