feat(ingest/powerbi): follow references into queries the model does not load

[puneetagarwal-datahub]

Closes the "Enable load switched off" report — 70 tables on the reporting tenant.

Base branch is #19370, not master. That in turn stacks on #19364. Reviewing down the stack is the shorter path; each PR's diff shows only its own changes.

Summary

A query with "Enable load" switched off is not a table in the model. It gets no entity of its own, so a loaded table built on one had nothing to point an edge at and lost its lineage entirely:

Merged Rows               load ON   ← the only table in the model
  = Table.Combine({ #"Rows Kept", #"Rows Dropped" })
        │                 both load OFF, both reading
        └──────────────► #"Base Rows"   load OFF
                          holds the Databricks.Catalogs navigation

Nothing in Merged Rows' own expression binds those names, so the walk stopped at the first one.

The M was already in hand

The scan is already asked for datasetExpressions, and the response was being discarded. So this reads what we were already paying for. Microsoft's own docs example is the same shape — a table whose M is let Source = Revenues … in Source, with Revenues a shared expression holding Sql.Database(…). The reporter confirmed it independently from their side before we looked.

Approach

Keep the expressions on the dataset, and when no enclosing let binds a name, follow it into the query that defines it. Because those queries are not entities, the chain is followed inline and the data source it ends at is attributed to the loaded table — emitting a Power BI → Power BI edge is not an option, there is nothing at the other end.

Guards, each with a test:

• A parameter is not a query. Following one leads to a literal, so expressions carrying IsParameterQuery are left alone.
• Cycles. Two queries referencing each other stop rather than recurse.
• Depth. A chain longer than ten references is not a real model shape.
• Casing. Names match the way identifier lookup does elsewhere, case-insensitively.
• Cost. A query reached by several routes is parsed once, not once per route.

Verified against the reported tenant's own M

Their ingestion log carries the real template — the connector bound to a named step, with the catalog, schema and table names all supplied as parameters rather than literals:

SourceStep   = Databricks.Catalogs(#"Param A", #"Param B", null),
DatabaseStep = SourceStep{[Name=ParamC, Kind="Database"]}[Data],
...

Running that shape as the hidden query resolves. Worth knowing for anyone reading the fixtures: an earlier version of my test used string literals, and the parameterised form is what they actually write.

One caveat that is not fixable here — if the hidden query's own M omits a Kind= on a navigation step, which three of their tables do, the chain still dead-ends until #19364 is in. Measured both ways: Kind present resolves on this branch alone, Kind absent needs #19364 too.

Two changes beyond the feature

Converging routes should name an upstream once. Reaching the same upstream by more than one route is legitimate — two queries filtering a common source, or both branches of a conditional — but the aspect should name it once. Upstreams and their column edges are both deduplicated as they are collected, keyed on the fields they join so a route contributing a mapping the others did not still reaches the aspect.

This was needed here (the diamond above reaches the same table twice) and it also covers Table.Combine({A, A}), which could already duplicate before this change.

The golden could not see the duplicate. check_golden_file defaults to ignore_order=True, so DeepDiff compares lists as sets and a repeated edge reads as equal to a single one. The golden held two column edges while the pipeline emitted four, and the test passed. The integration test now asserts the counts directly rather than trusting the golden. That blindspot applies to every golden in this suite.

Shared queries through parentheses. Every branch of the walk threads the shared queries except the parenthesized one, which arrives from #19364 in the base and has no such argument. Because the parameter defaults to None, a parenthesized step whose head is a hidden query resolved to nothing, silently. Fixed with a test that fails when the argument is dropped — #19364's own parenthesized tests pass either way, so only the new one catches it.

Testing

Six parser-level tests and a golden. The golden asserts the shape that matters: one upstream on the loaded table plus column-level lineage, from a fixture whose dataset carries the three hidden queries and one parameter expression.

Verified the golden guards the feature — stubbing the expressions out makes it fail.

382 passed, 2 xfailed · ruff, format and mypy clean · no existing goldens touched.

Known limitation, not introduced here

A hidden query written as a bare expression with no let wrapper yields no lineage and no warning. That is the pre-existing behaviour of the root-expression path — resolve_to_data_access_functions requires a LetExpression and returns [] equally quietly — and it is already documented at parser.py:37 from #18720. Measured on both paths; this PR inherits the limitation rather than adding it. Handling let-less roots changes behaviour for every table's own expression and belongs in its own PR.

Checklist

• PR conforms to the Contributing Guideline (particularly PR Title Format)
• Tests for the changes have been added
• Docs — n/a, no user-facing config change
• Breaking changes — none

---

Summary by cubic

Restores lineage for Power BI tables that reference queries with “Enable load” off by following those shared expressions inline. Before, unbound names in a table’s M stopped the walk and dropped lineage; now we resolve into the hidden query and attribute its data source to the loaded table.

• Reads dataset expressions and follows unbound identifiers into referenced queries, honoring native_query_parsing=False.
• Skips parameter queries, matches names case-insensitively, detects cycles, limits chains to 10, and parses each shared query once while caching failures.
• Reports unparseable referenced queries once per table instead of silently dropping lineage.
• Carries shared-query context through parenthesized steps so hidden queries still resolve.
• Deduplicates upstream URNs and column-level edges when routes converge, preserving unique mappings.
• Adds integration coverage for hidden-query lineage and asserts unique upstream/edge counts; parser tests cover cycles, parameters, case-insensitive lookup, native-query opt-out, and parentheses.

^{Written for commit 3e094a0. Summary will update on new commits.}

[github-actions]

Linear: ING-3352

Thanks for your contribution! We have created an internal ticket to track this PR. A member of the core DataHub team will be assigned to review it within the next few business days - you will get a follow-up comment once a reviewer is assigned.

[cursor]

PR Summary

Overview
Power BI lineage now resolves tables that reference queries with “Enable load” off by reading datasetExpressions from the scan response (previously discarded) and following those shared M queries inline, since they have no DataHub entity to link to.

The M-query resolver gains a SharedExpressions helper: unbound identifiers jump into the matching dataset query, with cycle and depth (10) guards, case-insensitive names, skip parameter queries (IsParameterQuery), parse-once caching, and native_query_parsing=False filtering for Value.NativeQuery in shared text as well as on the table. Unparseable referenced queries surface as ingestion warnings.

When building upstream lineage aspects, duplicate dataset upstreams and fine-grained column edges are deduplicated when multiple M paths converge on the same source.

Integration and parser tests cover diamond Table.Combine shapes, parentheses, native-query opt-out, and golden output uniqueness checks.

^{Reviewed by Cursor Bugbot for commit 3e094a0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

❌ Patch coverage is 36.17021% with 60 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...tahub/ingestion/source/powerbi/m_query/resolver.py	7.14%	26 Missing ⚠️
...stion/source/powerbi/m_query/shared_expressions.py	52.50%	19 Missing ⚠️
...on/src/datahub/ingestion/source/powerbi/powerbi.py	0.00%	13 Missing ⚠️
...datahub/ingestion/source/powerbi/m_query/parser.py	80.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[cubic-dev-ai]

All reported issues were addressed across 11 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[datahub-connector-tests]

Connector Tests Results

All connector tests passed for commit 3e094a0

View full test logs →

To skip connector tests, add the skip-connector-tests label (org members only).

Autogenerated by the connector-tests CI pipeline.

[cubic-dev-ai]

1 issue found across 6 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="metadata-ingestion/src/datahub/ingestion/source/powerbi/m_query/parser.py">

<violation number="1" location="metadata-ingestion/src/datahub/ingestion/source/powerbi/m_query/parser.py:78">
P2: When `native_query_parsing=False`, a hidden regular query containing `Value.NativeQuery` in SQL text or an M comment is removed before parsing, losing valid lineage. Detect an actual NativeQuery function call rather than scanning raw expression text.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: When native_query_parsing=False, a hidden regular query containing Value.NativeQuery in SQL text or an M comment is removed before parsing, losing valid lineage. Detect an actual NativeQuery function call rather than scanning raw expression text.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At metadata-ingestion/src/datahub/ingestion/source/powerbi/m_query/parser.py, line 78:

<comment>When `native_query_parsing=False`, a hidden regular query containing `Value.NativeQuery` in SQL text or an M comment is removed before parsing, losing valid lineage. Detect an actual NativeQuery function call rather than scanning raw expression text.</comment>

<file context>
@@ -66,10 +66,30 @@ def get_upstream_tables(
+    shared_texts = expressions or {}
+    if not config.native_query_parsing:
+        withheld = [
+            name for name, text in shared_texts.items() if "Value.NativeQuery" in text
+        ]
+        if withheld:
</file context>

[puneetagarwal-datahub]

Valid in principle, but leaving it as-is deliberately — the raw-text scan is what native_query_parsing=False already means in this file.

The pre-existing check on the table's own expression, six lines below at parser.py:115, is the same scan:

if not config.native_query_parsing and "Value.NativeQuery" in expression:

So a comment or a SQL string mentioning Value.NativeQuery already suppresses the root expression today. This PR makes references to hidden queries followable, which would otherwise let a native query one hop away be parsed despite the flag — the filter exists to keep the two paths behaving the same. Making it precise on one side only would leave them inconsistent.

Detecting an actual Value.NativeQuery invoke node requires parsing the expression first, which is the thing the flag is there to avoid. And the failure mode is conservative: a false positive skips a query the operator has already opted out of parsing, so no wrong lineage is produced — it just isn't followed. Tightening both checks to be AST-based is a reasonable change, but it belongs in its own PR since it alters behaviour for the root expression too.

[puneetagarwal-datahub]

On the red codecov/patch (39% patch coverage) — that number is a measurement gap, not missing tests.

Coverage is only uploaded from the Gradle matrix legs, and those run -m 'not integration'. The leg that actually exercises this code, ci (powerbi, tests/integration/powerbi/, 3.11), runs bare pytest -m integration "$TEST_PATH" with no --cov flags (.github/workflows/metadata-ingestion.yml, the elif [ -n "$TEST_PATH" ] branch), so it produces no coverage report for Codecov to combine. Every test added here is integration_batch_2, so from Codecov's view the new lines are untouched.

Measured locally over tests/integration/powerbi + tests/unit/powerbi (368 passed, 1 xfailed):

file	coverage	Codecov patch %
m_query/shared_expressions.py	100%	52.50%
m_query/resolver.py	82%	7.14%
m_query/parser.py	81%	80.00%
powerbi.py	97%	0.00%

Adding --cov to the selective connector leg would fix the reporting for every connector PR, but that is a CI change and does not belong in this one.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 392a098. Configure here.}