fix(ingest/powerbi): recover Databricks lineage lost to three M-Query shapes

[puneetagarwal-datahub]

Summary

Three shapes of valid M-Query silently produced no lineage. Each is a separate defect; all three were reproduced against the real M-Query bridge before fixing.

1. Redundant parentheses around a navigation step

tbl = (sch{[Name="my_table",Kind="Table"]}[Data])   // or: then (sch{...}[Data])

_walk had no ParenthesizedExpression case, so it hit the unhandled-node fallback and returned. The parse itself succeeded — only the walk gave up.

2. A navigation step written without Kind=

tbl = sch{[Name="my_table"]}[Data]

table_detail[items["Kind"]] raised a bare KeyError, which the caller's broad except turned into an "Unknown M-Query Pattern" warning — so the report named the pattern rather than the missing key. Databricks chains are ordered catalog → schema → table, so the level now follows from position. A step with neither Kind nor a resolvable Name is genuinely unusable and is reported as such instead of raising.

3. Native SQL on the Databricks.Catalogs connector

Value.NativeQuery was registered for DatabricksMultiCloud.Catalogs but not for plain Databricks.Catalogs, so Azure Databricks native queries bailed at the is_native_parsing_supported gate. get_db_name needed widening in the same change — both connectors take (host, http path, [Database=…, Catalog=…]), and without that half the catalog comes back empty and the emitted URN points at a dataset that does not exist.

Scope

Covers the inline and single-hop source-argument forms of (3). A source argument reached through more than one navigation step is a separate limitation in _get_data_source_tokens — that one is platform-agnostic (it affects Snowflake identically) and is left to its own PR.

Testing

Every fix has a test that was watched failing first, plus a golden integration test asserting the emitted MCPs — table-level upstream and column-level lineage. Verified the golden fails if the Databricks.Catalogs registration is removed, so it genuinely guards the change.

The golden uses a dedicated mock response rather than extending default_mock_response.json. That fixture is loaded by 34 test invocations, so adding a table there regenerates 13 goldens for ~3,400 lines of churn — and would assert nothing, because test_extract_lineage narrows dataset_type_mapping to four platforms and powerbi.py drops upstreams for any platform outside it. The per-platform fixture/golden pair follows the three existing mysql and ODBC pairs.

363 passed, 1 xfailed · ruff, format and mypy clean · no existing goldens touched.

Note for reviewers

This and #19365 both touch resolver.py — this PR adds the ParenthesizedExpression case, the other renames the walk's accumulator. Whichever merges second needs a one-block rebase with no logic overlap. Either order is fine.

Checklist

• PR conforms to the Contributing Guideline (particularly PR Title Format)
• Tests for the changes have been added
• Docs — n/a, no user-facing config change
• Breaking changes — none

---

Summary by cubic

Recovers Power BI lineage lost to three valid M-Query shapes and enables native SQL lineage on Azure Databricks. Previously: parenthesized navigation steps aborted the walk; Databricks steps without Kind raised and were masked as “Unknown M-Query Pattern”; Value.NativeQuery on Databricks.Catalogs was unsupported. Now: the walk unwraps parentheses, Databricks levels are inferred by position (catalog → schema → table), unusable steps warn and skip, and native queries on both Databricks connectors emit correct 3-part URNs.

• Adds ParenthesizedExpression handling in the resolver walk (covers bare (Source{...}[Data]) and then (...)/else (...)).
• Infers Databricks navigation level when Kind is missing; warns (does not raise) if both Kind and Name are missing.
• Registers Databricks.Catalogs for native-query parsing and widens database resolution to handle both connectors; fixes URN catalog resolution.
• Scope: handles inline and single-hop source arguments; multi-hop source arguments remain unsupported and will be addressed separately.
• Tests: focused unit tests, an integration golden asserting upstream and column-level lineage for Databricks.Catalogs native queries, and an added check pinning mid-chain level inference; no existing goldens changed.
• Review note: touches resolver.py; expect a trivial one-block rebase if concurrent changes land first.

^{Written for commit 2e9c885. Summary will update on new commits.}

[github-actions]

Linear: ING-3347

Thanks for your contribution! We have created an internal ticket to track this PR. A member of the core DataHub team will be assigned to review it within the next few business days - you will get a follow-up comment once a reviewer is assigned.

[cursor]

PR Summary

Overview
Restores Power BI → Databricks lineage for three valid M-Query shapes that previously dropped silently.

The M-Query walker now continues through ParenthesizedExpression nodes, so extra parentheses around a navigation step (including then (...)) no longer abort the walk.

DatabricksLineage infers catalog/schema/table from chain position when a step omits Kind, and reports a warning instead of raising KeyError when a step has neither Kind nor Name.

Value.NativeQuery on Databricks.Catalogs is now treated like the MultiCloud connector, so Azure Databricks native SQL emits 3-part table URNs (and column lineage). Multi-hop source arguments remain unsupported.

^{Reviewed by Cursor Bugbot for commit 2e9c885. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

❌ Patch coverage is 78.57143% with 3 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...tahub/ingestion/source/powerbi/m_query/resolver.py	0.00%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

[cubic-dev-ai]

No issues found across 7 files

_{Re-trigger cubic}

[datahub-connector-tests]

Connector Tests Results

All connector tests passed for commit 2e9c885

View full test logs →

To skip connector tests, add the skip-connector-tests label (org members only).

Autogenerated by the connector-tests CI pipeline.