Feature/runtime lineage

[swar00pduthks]

Problem

👋 Thanks for opening a pull request! Please include a brief summary of the problem your change is trying to solve, or bug fix. If your change fixes a bug or you'd like to provide context on why you're making the change, please link the issue as follows:

Closes: #3054 , #2772

Solution

New Feature: This PR introduces run-level graph visualization - enabling users to visualize relationships between dataset versions, job versions, and run nodes to simplify data troubleshooting and impact analysis.

Scalable Architecture: To ensure this feature can scale from day one, we implement CQRS (Command Query Responsibility Segregation) pattern with two denormalized tables optimized for different graph views:

• Parent-level run graph (run_parent_lineage_denormalized) - Business users see streamlined graphs with only top-level pipeline run nodes (~50 nodes for typical workloads)
• Full detailed run graph (run_lineage_denormalized) - Engineers access comprehensive graphs with all child run nodes for deep debugging (~13,500 nodes for detailed analysis)
• Monthly partitioning - Ensures performance remains fast as data grows, with automatic retention management

Performance: Designed for speed at scale - parent run graphs render in ~0.1s, detailed run graphs in ~0.8s, even with millions of historical runs. The denormalized design pre-computes 5-8 table joins, enabling sub-second graph traversal and rendering.

Note: All database schema changes require discussion. Please link the issue for context.

One-line summary: Introduces run-level graph visualization with denormalized tables and partitioning for scalable, sub-second lineage traversal supporting both parent-level (business) and detailed (technical) graph views.

Checklist

• [ X] You've signed-off your work
• Your changes are accompanied by tests (if relevant)
• [ X] Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• [ X] You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
• [ X] You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
• [ X] You've included a header in any source code files (if relevant)

[codecov]

Codecov Report

❌ Patch coverage is 86.10039% with 72 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.65%. Comparing base (1cfa6b7) to head (50b4b72).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
...ons/V77__backfill_denormalized_lineage_tables.java	69.38%	24 Missing and 6 partials ⚠️
...va/marquez/service/DenormalizedLineageService.java	83.96%	21 Missing ⚠️
...rc/main/java/marquez/db/mappers/RunDataMapper.java	88.50%	7 Missing and 3 partials ⚠️
...a/marquez/db/mappers/DatasetVersionDataMapper.java	90.90%	3 Missing and 1 partial ⚠️
...va/marquez/service/PartitionManagementService.java	94.23%	3 Missing ⚠️
api/src/main/java/marquez/MarquezApp.java	80.00%	1 Missing ⚠️
...c/main/java/marquez/api/ColumnLineageResource.java	97.36%	0 Missing and 1 partial ⚠️
api/src/main/java/marquez/db/LineageDao.java	0.00%	1 Missing ⚠️
...rc/main/java/marquez/tracing/TracingSQLLogger.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3084      +/-   ##
============================================
+ Coverage     81.18%   81.65%   +0.47%     
- Complexity     1506     1623     +117     
============================================
  Files           268      276       +8     
  Lines          7356     7850     +494     
  Branches        325      353      +28     
============================================
+ Hits           5972     6410     +438     
- Misses         1226     1272      +46     
- Partials        158      168      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[swar00pduthks]

@thijs-s can you check what do you think about this runlineage implementation , I have done on this dropwizard upgrade and if naming conventions for table please help to review

[thijs-s]

Definitely! I'm a little afraid of the size but will check it.

[swar00pduthks]

Definitely! I'm a little afraid of the size but will check it.

Thanks , the size is because of the dropwizard pull request is included in this (#3056) as it was not yet approved.. I can remove that .. and only raise pull request for runtime lineage .. let me know

[thijs-s]

no worries, I will try to handle it. if not then will let you know.