fix(ingestion/grafana): don't break SQL parsing for quoted template variables

[daha]

Fixes #19346

Problem

_clean_grafana_template_variables() replaces ${var} with the quoted literal 'grafana_var'
without checking whether the variable is already inside quotes. Panel SQL containing '${var}' —
the common way to interpolate a Grafana variable into a SQL string or cast — becomes
''grafana_var'', which no dialect can parse:

-- input (valid, parses as-is)
WHERE run_id = CAST('${run_id}' AS INTEGER)
-- after cleaning
WHERE run_id = CAST(''grafana_var'' AS INTEGER)   -- ParseError: Expected AS after CAST

The irony is that the unmodified query is fine — sqlglot treats '${var}' as an ordinary string
literal. The cleaning step is what breaks it.

The failure is then invisible. create_lineage_sql_parsed_result swallows the ParseError and
returns a truthy SqlParsingResult with in_tables=[], so _parse_sql returns non-None,
_create_column_lineage bails at its if not parsed_sql.in_tables guard, and
extract_panel_lineage falls through to _create_basic_lineage — which builds the upstream URN
from the Grafana datasource UID:

actual:   urn:li:dataset:(urn:li:dataPlatform:postgres,test_db.postgres_uid,PROD)     <- datasource UID
expected: urn:li:dataset:(urn:li:dataPlatform:postgres,test_db.public.test_table,PROD)

That dataset does not exist in the upstream platform, so ingestion materialises a phantom entity
that looks like genuine lineage. Nothing warns — panel_parsing_warnings stays at 0 and
warnings stays empty.

Measured on a production Grafana instance with ~800 dashboards: 143 of 940 upstreamLineage
aspects (15.2%) pointed at a fabricated UUID-named dataset; 101 of those are caused by this
issue.

Fix

The sibling _GRAFANA_SIMPLE_VAR_PATTERN already carries (?<!')...(?!') for exactly this
reason, and the function's docstring documents that '$var' is left unchanged. This gives the
braced pattern the same guard, so the documented contract holds for both syntaxes:

-_GRAFANA_BRACED_VAR_PATTERN = re.compile(r"\$\{[^}]+\}")
+_GRAFANA_BRACED_VAR_PATTERN = re.compile(r"(?<!')\$\{[^}]+\}(?!')")

Both lookarounds must hold for a substitution, so '${var}' is left untouched — which is correct,
because it is already a valid SQL string literal. Everything else in the diff is the matching
docstring and comment update.

The ordering of the other passes makes this sufficient on its own:
_GRAFANA_GENERIC_MACRO_PATTERN (\$__\w+) runs earlier but cannot match ${__from}, because
$ is followed by { rather than _; and _GRAFANA_SIMPLE_VAR_PATTERN runs later but cannot
match '${run_id}' for the same reason. A quoted braced variable now survives all passes
unchanged.

Known limitation, shared with the existing sibling pattern and deliberately not addressed here:
a single-quote lookaround is a proxy for "inside a string literal", and it only fires when a quote
is immediately adjacent to the variable. A variable in the middle of a longer literal is still
substituted and still breaks parsing:

'${var}'              -> unchanged                      (fixed by this PR)
'${var}xyz'           -> unchanged                      (fixed by this PR)
'xyz${var}'           -> unchanged                      (fixed by this PR)
'prefix${var}suffix'  -> 'prefix'grafana_var'suffix'    (still broken, unchanged by this PR)

That last case produces byte-identical output before and after this change, so it is a pre-existing
gap rather than a regression, and it is exactly how _GRAFANA_SIMPLE_VAR_PATTERN already behaves.
Closing it properly needs string-literal-aware scanning rather than a wider regex, which is a
larger change than this fix warrants.

Testing

Three tests added, each confirmed failing before the change and passing after.

1. test_removes_all_grafana_variable_formats[braced_variable_in_quotes] — one new case in the
   existing parametrized list, mirroring the variable_in_quotes case that already asserts this
   contract for '$var'. A second case,
   [braced_variables_quoted_and_unquoted], puts a quoted and an unquoted braced variable in one
   query, pinning the fix as selective rather than "stopped substituting braced variables".
2. test_cleaned_query_remains_parseable_with_quoted_variables — feeds the cleaned query back
   through sqlglot and asserts the upstream table is still extractable. The existing suite only
   asserted string equality on the cleaner's output, never that the output parses, which is why an
   11-case suite passed throughout.
3. test_extract_panel_lineage_with_quoted_template_variable — end-to-end, asserts the emitted
   upstream URN is the real table and not the datasource UID. This is the user-visible regression;
   the existing test_extract_panel_lineage_postgres asserts only len(upstreams) == 1, which
   passes identically on both the correct and the fabricated-URN path.

Before:

FAILED test_grafana_lineage.py::test_extract_panel_lineage_with_quoted_template_variable
  - AssertionError: assert 'test_db.public.test_table' in
    'urn:li:dataset:(urn:li:dataPlatform:postgres,test_db.postgres_uid,PROD)'
FAILED test_grafana_query_extraction.py::...::test_removes_all_grafana_variable_formats[braced_variable_in_quotes]
  - assert "WHERE run_id = CAST(''grafana_var'' AS INTEGER)" == "WHERE run_id = CAST('${run_id}' AS INTEGER)"
FAILED test_grafana_query_extraction.py::...::test_cleaned_query_remains_parseable_with_quoted_variables
  - sqlglot.errors.ParseError: Expected AS after CAST. Line 1, Col: 78.
3 failed, 107 passed in 0.68s

After:

111 passed in 0.58s

No existing assertion was changed. ./gradlew :metadata-ingestion:lint is clean (ruff check, ruff
format, mypy).

Follow-up, not in this PR

Four further defects in the same function, each producing the same silent fallback, are
deliberately left out — each is independently arguable and would stall review of a one-line fix.
They are described in the linked issue:

• $__timeFrom() / $__timeTo() / $__timeGroup(...) are value-producing macros replaced with
  TRUE. An existing test asserts the current behaviour, so fixing it means changing that
  expectation.
• Macro argument matching uses [^)]*, which is not nesting-aware.
• _GRAFANA_SIMPLE_VAR_PATTERN treats $ as a sigil mid-identifier, mangling Oracle identifiers
  such as SOME$COL.
• _parse_sql discards parsed_sql.debug_info.table_error. Surfacing it as a source warning would
  make this whole class of failure discoverable rather than silent. Note also that
  GrafanaSourceReport.report_sql_parsing_{attempt,success,failure} are called only from
  field_utils.py and never from lineage.py, so sql_parsing_failures is structurally always 0
  for the lineage path, while report_lineage_extracted() counts the fabricated URN as a success.
• The quote guard is adjacency-based, so '%${var}%' — a very common Grafana LIKE idiom — is
  still substituted and still unparseable. Closing it properly means masking string literals before
  substituting, which would let both this pattern and its sibling drop their lookarounds entirely.
• The integration fixtures in tests/integration/grafana/ contain no Grafana template variables at
  all, so the whole cleaning path has no end-to-end coverage.

Whether _create_basic_lineage should emit a made-up dataset name at all is a design question for
maintainers, and is also left alone here.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly PR Title Format)
• Links to related issues
• Tests for the changes have been added/updated
• Docs related to the changes have been added/updated — n/a, no config or API surface change
• Entry in Updating DataHub — n/a, bugfix with no breaking change

🤖 Generated with Claude Code

---

Summary by cubic

Preserves quoted braced Grafana variables during cleaning so SQL keeps parsing. Previously '${var}' became ''grafana_var'', parsing failed, and lineage fell back to a datasource-UID dataset; now quoted ${...} is left as-is so upstreams resolve to the real table.

• Replace the braced-var regex with (?<!')\$\{[^}]+\}(?!') and align docs/comments; pass ordering continues to substitute unquoted variables.
• Tests: add quoted braced-var, mixed quoted/unquoted, a parseability check using sqlglot, and an end-to-end lineage test that asserts the real table URN with a real SchemaResolver.
• Limitation unchanged: the guard is quote-adjacent, not literal-aware; %${var}% can still be substituted and fail to parse. No config or migration changes.

^{Written for commit d2d8de4. Summary will update on new commits.}

[github-actions]

Linear: ING-3336

Thanks for your contribution! We have created an internal ticket to track this PR. A member of the core DataHub team will be assigned to review it within the next few business days - you will get a follow-up comment once a reviewer is assigned.

[cubic-dev-ai]

No issues found across 3 files

_{Re-trigger cubic}