Skip to content

fix(nyc-taxi): match instance name exactly to avoid prefix collision - #213

Merged
lakshay-nasa merged 1 commit into
datahub-project:mainfrom
yanou16:fix/nyc-taxi-instance-name-prefix-collision
Jul 23, 2026
Merged

fix(nyc-taxi): match instance name exactly to avoid prefix collision#213
lakshay-nasa merged 1 commit into
datahub-project:mainfrom
yanou16:fix/nyc-taxi-instance-name-prefix-collision

Conversation

@yanou16

@yanou16 yanou16 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

TL;DR

The instance filter uses a substring test, which collides when one instance
name is a prefix of another ("nyc_taxi""nyc_taxi_pipeline"). Fixed by
matching the exact instance name.

Bug fix diagram: instance-name prefix collision

Problem

The nyc-taxi helper scripts filter discovered dataset URNs by instance using a
plain substring test:

if platform_instance not in urn:      # add_lineage.py
if platform_instance not in entity.get("urn", ""):   # add_metadata.py

This is wrong when one instance name is a prefix of another. The dataset
ships two instances, nyc_taxi and nyc_taxi_pipeline, and "nyc_taxi" is a
substring of "nyc_taxi_pipeline". So when running for --instance=nyc_taxi,
the filter also lets nyc_taxi_pipeline URNs through. Same-named tables
(staging_trips, mart_daily_summary, ...) then overwrite each other in
urn_map, and lineage/metadata for the clean nyc_taxi instance gets emitted
onto nyc_taxi_pipeline URNs — leaving the nyc_taxi instance with no lineage.

Fix

Match the exact instance by checking for ",{instance}.". Inside a dataset URN
the instance name is delimited by a comma and a dot
(...,nyc_taxi.main.staging_trips,...), so ",nyc_taxi." does not match
",nyc_taxi_pipeline.". Applied in both add_lineage.py and add_metadata.py.

Impact

  • datahub ... add_lineage.py --instance=nyc_taxi / --all now writes lineage
    to the correct instance.
  • No behavior change for single-instance users or for nyc_taxi_pipeline.

How to reproduce (before the fix)

python add_lineage.py --all
# then query lineage for nyc_taxi.staging_trips -> 0 downstream (should be 4)

The instance filter used a plain substring test (`platform_instance in urn`),
which wrongly matches when one instance name is a prefix of another - e.g.
"nyc_taxi" is contained in "nyc_taxi_pipeline". Same-named tables from
different instances then collided in urn_map, so lineage/metadata for the
`nyc_taxi` instance was emitted onto `nyc_taxi_pipeline` URNs and the clean
instance ended up with no lineage.

Match the exact instance by checking for ",{instance}." (the instance name is
delimited by a comma and a dot inside the dataset URN). Applied in both
add_lineage.py and add_metadata.py.
yanou16 added a commit to yanou16/static-assets that referenced this pull request Jul 23, 2026
@lakshay-nasa

Copy link
Copy Markdown
Contributor

Thanks @yanou16! Clean fix, applied consistently across both scripts 🙌

@lakshay-nasa
lakshay-nasa merged commit edbfb6d into datahub-project:main Jul 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants