fix(ingest/powerbi): follow a native query's source argument to the connector

[puneetagarwal-datahub]

Summary

Value.NativeQuery names its data source as the first argument, and in real M that argument is usually a navigation step rather than the connector call itself:

Source       = Snowflake.Databases(host, warehouse),
DatabaseStep = Source{[Name="MYDB", Kind="Database"]}[Data],
QueryStep    = Value.NativeQuery(DatabaseStep, "select ... from ...")

_get_data_source_tokens resolved that identifier once, landed on Source{...}[Data], read its head as the literal name Source, and gave up — no connector is called that. Measured on master:

source argument	tokens[0]
inline Snowflake.Databases(...){...}[Data]	Snowflake.Databases ✅
one hop — NativeQuery(Source, ...)	Snowflake.Databases ✅
two hops — NativeQuery(DatabaseStep, ...)	Source ❌

Not Databricks-specific. The same shape was losing lineage on Snowflake, Redshift, MSSQL, Postgres and BigQuery. It survived because every existing native-query test uses the inline form, where the head is the connector.

Approach

Follow the head back until it reaches something the let does not bind — that is what distinguishes a connector call from a step name — and append each hop's navigation record after the connector's own arguments. The platform stays at index 0 and the server at index 1 however deep the chain runs, so get_db_name and create_lineage are unaffected. Aliases of aliases are followed too.

Safety

The inline path is byte-for-byte unchanged; I verified the emitted token list is identical before and after.

Two properties are pinned by tests:

• Cycles. a = b, b = a would otherwise spin in the alias loop rather than raise — hanging an ingestion run instead of failing it. Guarded by a seen-set; returns no lineage.
• Unresolved parameters. When the connector's positional arguments are parameter references that cannot be resolved, the navigation record shifts into the server position. The guard added in fix(ingest/powerbi): resolve Snowflake database for parameterized Value.NativeQuery #18724 still declines to guess a host, and this change does not undermine it.

Also verified: three-hop chains accumulate both navigation records; multi-table native SQL (join / union / CTE) still yields one upstream per table.

Relationship to other PRs

This is the last piece of the M-Query native-query work. #19364 registers the Databricks.Catalogs connector for native queries; both are needed for a two-hop Databricks native query, since this PR fixes the traversal and that one fixes the registration. They touch different regions of pattern_handler.py and can merge in either order.

Testing

360 passed, 1 xfailed · ruff, format and mypy clean · no existing goldens touched.

New tests use Snowflake and DatabricksMultiCloud.Catalogs, since plain Databricks.Catalogs is not a registered native-query platform until #19364 lands.

Checklist

• PR conforms to the Contributing Guideline (particularly PR Title Format)
• Tests for the changes have been added
• Docs — n/a, no user-facing config change
• Breaking changes — none

---

Summary by cubic

Fixes lineage extraction for Power BI M Value.NativeQuery when the source argument is a multi-hop step. Previously, resolving only one hop treated the head as a step name (e.g., "Source") and dropped lineage for Snowflake, Redshift, MSSQL, Postgres, and BigQuery; now the parser walks back through aliases/navigation to the connector and preserves argument positions.

• Follows let-bound aliases until the unbound connector, then appends each hop’s navigation record after the connector’s args, keeping platform at index 0 and server at index 1. Inline paths are unchanged byte-for-byte.
• Guards cycles with a seen-set (no hangs) and keeps the “no guessed host” behavior when connector args are unresolved parameters; multi-table SQL lineage unaffected.
• Adds integration tests for Snowflake and DatabricksMultiCloud.Catalogs multi-hop, cycle handling, and unresolved parameters; no existing goldens changed.
• For two-hop Databricks native queries, this and fix(ingest/powerbi): recover Databricks lineage lost to three M-Query shapes #19364 (registers Databricks.Catalogs) are both required; they touch different parts of pattern_handler.py and can merge in any order.

^{Written for commit e6414fd. Summary will update on new commits.}

[github-actions]

Linear: ING-3350

Thanks for your contribution! We have created an internal ticket to track this PR. A member of the core DataHub team will be assigned to review it within the next few business days - you will get a follow-up comment once a reviewer is assigned.

[cursor]

PR Summary

Overview
Fixes PowerBI Value.NativeQuery lineage when the first argument is a navigation step (e.g. db = Source{[Name=...]}[Data]) instead of the connector call itself. Previously _get_data_source_tokens resolved one hop and treated the step name as the platform, dropping lineage for Snowflake, Redshift, MSSQL, Postgres, BigQuery, and Databricks.

It now walks let-bound aliases until it hits a name that is not a let variable (the connector), then appends each hop’s navigation record after the connector arguments so platform stays at index 0 and server at index 1. Cycles are cut with a seen set (no lineage, no hang). Unresolved connector parameters still produce no guessed host.

Adds integration tests for multi-hop Snowflake and Databricks MultiCloud native queries, cyclic step aliases, and unresolved parameters.

^{Reviewed by Cursor Bugbot for commit e6414fd. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

❌ Patch coverage is 86.20690% with 4 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...ngestion/source/powerbi/m_query/pattern_handler.py	86.20%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e6414fd. Configure here.}

[cursor]

Databricks DB lookup hits Kind value

Medium Severity

Multi-hop token assembly appends later navigation pairs after the database hop’s Kind/Database pair. For DatabricksMultiCloud.Catalogs with Database=null (so no real Database key is emitted), get_db_name’s get_next_item(..., "Database") matches that Kind value and returns the following token (Name) as the database, producing wrong upstream URNs on three-hop (or longer) native queries.

Additional Locations (1)

• metadata-ingestion/src/datahub/ingestion/source/powerbi/m_query/pattern_handler.py#L1570-L1590

 

^{Reviewed by Cursor Bugbot for commit e6414fd. Configure here.}

[datahub-connector-tests]

Connector Tests Results

All connector tests passed for commit e6414fd

View full test logs →

To skip connector tests, add the skip-connector-tests label (org members only).

Autogenerated by the connector-tests CI pipeline.

[cubic-dev-ai]

1 issue found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="metadata-ingestion/src/datahub/ingestion/source/powerbi/m_query/pattern_handler.py">

<violation number="1" location="metadata-ingestion/src/datahub/ingestion/source/powerbi/m_query/pattern_handler.py:173">
P2: When `DatabricksMultiCloud.Catalogs` omits `Database` with `null` and another navigation hop follows, this recursive token assembly places the later `Name` after the navigation's `Kind=Database` value. `get_db_name` then treats `Name` as the database and emits an incorrect upstream URN. Preserve navigation field boundaries or require a real `Database` key/value pair before appending later hops.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: When DatabricksMultiCloud.Catalogs omits Database with null and another navigation hop follows, this recursive token assembly places the later Name after the navigation's Kind=Database value. get_db_name then treats Name as the database and emits an incorrect upstream URN. Preserve navigation field boundaries or require a real Database key/value pair before appending later hops.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At metadata-ingestion/src/datahub/ingestion/source/powerbi/m_query/pattern_handler.py, line 173:

<comment>When `DatabricksMultiCloud.Catalogs` omits `Database` with `null` and another navigation hop follows, this recursive token assembly places the later `Name` after the navigation's `Kind=Database` value. `get_db_name` then treats `Name` as the database and emits an incorrect upstream URN. Preserve navigation field boundaries or require a real `Database` key/value pair before appending later hops.</comment>

<file context>
@@ -113,36 +113,73 @@ def _get_record_args(node_map: Dict[int, dict], invoke_node: dict) -> Dict[str,
+    if resolved_head is not None:
+        # The head is another step, so the connector is further back. Its tokens
+        # come first; this level's navigation steps are appended below.
+        tokens = _get_data_source_tokens(
+            node_map, resolved_head, parameters=parameters, seen=seen
+        )
</file context>

[puneetagarwal-datahub]

Moved to draft while a scope question is settled — not because of anything found in review.

This fixes a real traversal bug, but I have not confirmed the reported customer M actually hits it. Their example table for that symptom fails silently, so it appears nowhere in their ingestion log and I have no sample of its M.

Measured which branch state fixes which of their real shapes:

customer M shape	master	#19364	this PR	both
navigation chain, no Kind on leaf	✗	✅	✗	✅
native query, multi-hop source arg	✗	✗	✗	✅
native query, inline source arg	✗	✅	✗	✅

Two things follow. This PR fixes nothing on its own for that customer, because Databricks.Catalogs is not a registered native-query platform until #19364 lands. And if their native query turns out to be the inline form, #19364 closes the whole ticket by itself and this becomes an optional fix rather than a required one.

Waiting on the M text for that table. If it is multi-hop this is required and I will mark it ready; if it is inline it stands on its own merits as a latent bug — the same shape loses lineage on Snowflake, Redshift, MSSQL, Postgres and BigQuery — and can be scheduled independently.

Code is complete and verified either way: 360 passed, lint and mypy clean, inline token output byte-identical to before, and the #18724 unresolved-parameter guard confirmed still firing.