Skip to content

Add a snapshot-aware metadata and statistics catalog - #123

Open
yliang412 wants to merge 11 commits into
mainfrom
yuchen/catalog
Open

yliang412 wants to merge 11 commits into
mainfrom
yuchen/catalog

Conversation

@yliang412

Copy link
Copy Markdown
Member

Summary

  • Add a SQLite/PostgreSQL metadata catalog with one snapshot history, source namespaces, and non-destructive migrations.
  • Persist basic and additive advanced statistics; integrate with DataFusion and the CLI.
  • Preserve native DML and observe DML/DDL to invalidate statistics atomically, with regression coverage and PostgreSQL CI.

Validation

398 workspace tests, 237 no-default-feature core tests, 44 catalog tests/doctests with PostgreSQL enabled, and 17 feature + 45 TPC-H SLTs passed. Clippy, formatting, and actionlint passed.

The full JOB run timed out with explain mismatches; the focused JOB 1a mismatch reproduces identically on main. Other JOB failures remain unattributed. No golden files changed.

Query templates and instance history are deferred to a separate PR.

yliang412 and others added 10 commits October 20, 2026 09:09
Co-Authored-By: GPT-5.6 Sol <noreply@pi.dev>
Generated-By: pi 0.85.1
Co-Authored-By: GPT-5.6 Sol <noreply@pi.dev>
Generated-By: pi 0.85.1
Co-Authored-By: GPT-5.6 Sol <noreply@pi.dev>
Generated-By: pi 0.85.1
Co-Authored-By: GPT-5.6 Sol <noreply@pi.dev>
Generated-By: pi 0.85.1
Co-Authored-By: GPT-5.6 Sol <noreply@pi.dev>
Generated-By: pi 0.85.1
Co-Authored-By: GPT-5.6 Sol <noreply@pi.dev>
Co-Authored-By: GPT-6.1 Sol <noreply@pi.dev>
Generated-By: pi 1.3.0
Co-Authored-By: GPT-5.6 Sol <noreply@pi.dev>
Co-Authored-By: GPT-6.1 Sol <noreply@pi.dev>
Generated-By: pi 1.3.0
Co-Authored-By: GPT-5.6 Sol <noreply@pi.dev>
Co-Authored-By: GPT-6.1 Sol <noreply@pi.dev>
Generated-By: pi 1.3.0
Co-Authored-By: GPT-5.6 Sol <noreply@pi.dev>
Co-Authored-By: GPT-6.1 Sol <noreply@pi.dev>
Generated-By: pi 1.3.0
…validation

Version source-scoped metadata in one global snapshot history with non-destructive legacy migrations and additive advanced statistics.\n\nPreserve native DML capabilities, observe DML and DDL outcomes, and atomically invalidate statistics without replaying source writes. Harden namespace resolution, counter bounds, and demo file handling.\n\nAdd regression coverage, PostgreSQL CI, and catalog contracts and review evidence.

Co-Authored-By: GPT-6.1 Sol <noreply@pi.dev>
Generated-By: pi 1.2.0
@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.50165% with 296 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.6%. Comparing base (429db6b) to head (9aade9b).

Files with missing lines Patch % Lines
...td/connectors/datafusion/src/persistent_catalog.rs 93.9% 100 Missing ⚠️
optd/connectors/datafusion/src/statistics_udfs.rs 87.5% 48 Missing ⚠️
optd/catalog/src/index/create_index.rs 75.6% 29 Missing ⚠️
optd/connectors/datafusion/src/bin/optd-cli.rs 0.0% 18 Missing ⚠️
optd/catalog/src/entity/column.rs 31.5% 13 Missing ⚠️
optd/catalog/src/advanced_statistics.rs 95.3% 12 Missing ⚠️
...td/connectors/datafusion/src/observed_execution.rs 94.4% 12 Missing ⚠️
.../catalog/src/statistics/get_relation_statistics.rs 93.8% 8 Missing ⚠️
optd/catalog/src/index/get_indexes.rs 90.5% 7 Missing ⚠️
optd/catalog/src/statistics/mod.rs 83.7% 7 Missing ⚠️
... and 20 more
Additional details and impacted files
Files with missing lines Coverage Δ
optd/catalog/src/index/drop_index.rs 100.0% <100.0%> (ø)
optd/catalog/src/index/mod.rs 100.0% <100.0%> (ø)
optd/catalog/src/migration/mod.rs 100.0% <100.0%> (ø)
optd/catalog/src/relation/drop_relation.rs 100.0% <100.0%> (ø)
optd/catalog/src/relation/get_all_relations.rs 100.0% <100.0%> (ø)
optd/catalog/src/relation/get_relation.rs 100.0% <100.0%> (ø)
optd/catalog/src/relation/mod.rs 100.0% <100.0%> (ø)
optd/catalog/src/schema/create_schema.rs 100.0% <100.0%> (ø)
optd/catalog/src/schema/get_all_schemas.rs 100.0% <100.0%> (ø)
optd/catalog/src/schema/get_schema.rs 100.0% <100.0%> (ø)
... and 35 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Co-Authored-By: GPT-6.1 Sol <noreply@pi.dev>
Generated-By: pi 1.2.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants