Consolidate the metastore onto the default storage (overlay → migrate → contract)

[em3s]

Background

Actionbase's original design used MySQL as the metastore. Today the catalog (database, table, alias) lives across two tiers: H2 (bootstrap constant metadata) and MySQL (operational metadata). This consolidates it onto a single default storage — the same datastore that holds data — and moves the representation from the v2 HASH model to the v3 Edge type, simplifying the architecture.

PRs will be split per task.

Task

• Phase 1 — Overlay (HBase r/w -> MySQL read-only, override merge) — see breakdown below
• Phase 2 — Migration (backfill the frozen MySQL into HBase) — see breakdown below
• Phase 3 — Contract (remove overlay, MySQL, H2, JDBC) — see breakdown below

Phase 1 breakdown — Overlay (HBase r/w -> MySQL read-only, override merge)

• Overlay metastore label — PR: TBD

  • Background: We need a two-layer overlay label where HBase is the read/write layer and MySQL is a read-only base — a generalization of the existing LocalBackedJdbcHashLabel.
  • Task:
    • Implement the two-layer label: HBase (overlay) + MySQL (base)
    • Read = override merge (dedup on edge id encode(src, tgt), HBase wins)
    • Write/delete = HBase only; MySQL stays untouched
  • Done When: When a key exists in both layers, the HBase value is returned and MySQL is read only as a fallback.

• v3 Edge indexed schema — PR: TBD

  • Background: HBase HashEdge cannot be scanned, so the new metastore must be index-based (HBaseIndexedLabel = v3 Edge).
  • Task:
    • Define the v3 Edge schema for the operational metadata labels
    • Add indexes covering the scan patterns (full enumerate + source prefix)
    • Guarantee full property round-trip
  • Done When: Reload/inspector scans work through the index.

• Route operational metadata labels to the overlay — PR: TBD

  • Background: Route new operational metadata writes to HBase through the overlay. The H2 constant seed is left unchanged.
  • Task:
    • Route operational writes for service/storage/label/query/alias to the overlay
    • Update the LabelEntity.materialize branch and the Metadata.kt storage assignment
  • Done When: New writes go to HBase and existing reads fall back to MySQL.

• Rework MetastoreInspector — PR: TBD

  • Background: MetastoreInspector queries MySQL directly via SQL, bypassing the Label abstraction. Left as-is, new HBase metadata would be invisible.
  • Task:
    • Rework it to merge HBase index scan results
    • Preserve dump/count/prefix-filter behavior
  • Done When: /graph/v2/metastore/* shows both HBase and MySQL metadata.

• E2E parity tests — PR: TBD

  • Background: After introducing the overlay, behavior must match the current MySQL-backed metastore.
  • Task:
    • E2E comparing overlay results against current MySQL behavior
    • Verify new HBase write → override read
    • Regression tests for reload/inspector scan
  • Done When: The existing metastore E2E passes with identical results on the overlay.

Phase 2 breakdown — Migration

• Backfill MySQL into HBase — PR: TBD

  • Background: MySQL has been frozen read-only since Phase 1, so the backfill is a one-time copy of a stable snapshot.
  • Task:
    • Scan the MySQL metastore and write each key absent from HBase
    • Non-clobbering writes (insert-if-absent / setnx) so newer Phase-1 HBase versions are never overwritten
    • Make the job idempotent / re-runnable
  • Done When: every MySQL key exists equivalently in HBase.

• Reconciliation / shadow-compare — PR: TBD

  • Background: removing the overlay in Phase 3 is irreversible, so migration completeness must be proven first.
  • Task:
    • Scan MySQL vs HBase and compare key-by-key (equivalent SchemaEdge)
    • Report missing keys / mismatches
  • Done When: reconciliation reports zero discrepancies — the gate for Phase 3.

Phase 3 breakdown — Contract (point of no return)

• Collapse the overlay to a single HBase metastore — PR: TBD

  • Background: with migration complete, the overlay's MySQL base is no longer needed.
  • Task:
    • Route all metastore labels directly to HBaseIndexedLabel (drop the overlay wrapper)
    • Drop the MySQL fallback read path
  • Done When: the metastore reads/writes HBase only.

• Move bootstrap constant seeding to HBase — PR: TBD

  • Background: the H2 constant metadata seed is the last remaining non-HBase tier.
  • Task:
    • Seed the bootstrap system-label constants into HBase at startup
    • Remove the H2 local store
  • Done When: startup seeds constants into HBase with no H2.

• Remove MySQL/JDBC/H2 code, deps, and config — PR: TBD

  • Background: cleanup once nothing reads MySQL/H2.
  • Task:
    • Remove JdbcStorage, JdbcHashLabel, LocalBackedJdbcHashLabel, LocalStorage, MetadataTable, schema.sql, StorageType.LOCAL/JDBC
    • Drop deps (mysql-connector, HikariCP, Exposed, H2) and the metastore* config
    • Drop the MySQL-scan path from MetastoreInspector
  • Done When: no MySQL/H2/JDBC references remain and the build is green.

Done When

Operational metadata is stored in the default storage as v3 Edges, MySQL/H2/JDBC are removed, and existing behavior is preserved.

Notes

Phases 1–2 are reversible; only Phase 3 is irreversible. Related: #281, #173, #247, #234.