fix(nyc-taxi): match instance name exactly to avoid prefix collision - #213
Merged
lakshay-nasa merged 1 commit intoJul 23, 2026
Conversation
The instance filter used a plain substring test (`platform_instance in urn`),
which wrongly matches when one instance name is a prefix of another - e.g.
"nyc_taxi" is contained in "nyc_taxi_pipeline". Same-named tables from
different instances then collided in urn_map, so lineage/metadata for the
`nyc_taxi` instance was emitted onto `nyc_taxi_pipeline` URNs and the clean
instance ended up with no lineage.
Match the exact instance by checking for ",{instance}." (the instance name is
delimited by a comma and a dot inside the dataset URN). Applied in both
add_lineage.py and add_metadata.py.
yanou16
added a commit
to yanou16/static-assets
that referenced
this pull request
Jul 23, 2026
Contributor
|
Thanks @yanou16! Clean fix, applied consistently across both scripts 🙌 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
The instance filter uses a substring test, which collides when one instance
name is a prefix of another (
"nyc_taxi"⊂"nyc_taxi_pipeline"). Fixed bymatching the exact instance name.
Problem
The nyc-taxi helper scripts filter discovered dataset URNs by instance using a
plain substring test:
This is wrong when one instance name is a prefix of another. The dataset
ships two instances,
nyc_taxiandnyc_taxi_pipeline, and"nyc_taxi"is asubstring of
"nyc_taxi_pipeline". So when running for--instance=nyc_taxi,the filter also lets
nyc_taxi_pipelineURNs through. Same-named tables(
staging_trips,mart_daily_summary, ...) then overwrite each other inurn_map, and lineage/metadata for the cleannyc_taxiinstance gets emittedonto
nyc_taxi_pipelineURNs — leaving thenyc_taxiinstance with no lineage.Fix
Match the exact instance by checking for
",{instance}.". Inside a dataset URNthe instance name is delimited by a comma and a dot
(
...,nyc_taxi.main.staging_trips,...), so",nyc_taxi."does not match",nyc_taxi_pipeline.". Applied in bothadd_lineage.pyandadd_metadata.py.Impact
datahub ... add_lineage.py --instance=nyc_taxi/--allnow writes lineageto the correct instance.
nyc_taxi_pipeline.How to reproduce (before the fix)
python add_lineage.py --all # then query lineage for nyc_taxi.staging_trips -> 0 downstream (should be 4)