unmeasured

Most freshness agents are named for what they find. This one is named for what it refuses to fake.

A DataHub freshness agent that closes the loop — discover → measure → attribute → record → verify — built on four honest states: FRESH / STALE / UNMEASURED / UNCOVERED. The last two are the point: it treats "I could not tell, and here is why" as a first-class finding, writes it into the catalog, and then proves the write landed.

It reads two platforms through one set of honesty rules — sqlite via node:sqlite, postgres via docker exec + psql — and its lineage attribution crosses the platform border: in the captured runs, a stale postgres table's root cause is a sqlite table.

Built for the Build with DataHub hackathon, against the official NYC Taxi sample dataset. Zero npm dependencies: built-in node:sqlite plus a bare stdio JSON-RPC client for mcp-server-datahub.

Why

Everyone building a product idealizes the outcome. Almost nobody starts by writing down the worst failure. The cost of that grows with the system — worst of all during big architectural rework, where old and new paths coexist and what you read is no longer the true value. The frightening part: the results look the same either way.

And in the real world, the person running the check often can't read the problem. They can tell whether the number is right and whether the code is right — but whether the code still means what the business declared is exactly where errors hide.

Agents make this worse, not better: an agent infers from the nearest available result unless the structure forces it back to the source. That's not a prompt problem — the author of this repo made the same mistake while building it (see below). The constraint has to be structural.

So this agent is built around three rules, each paid for in production before this repo existed:

1. If an input cannot be measured, say UNMEASURED. Never a number, never a verdict.
2. A verdict may only exist when both numbers are in hand — what the catalog declares, and what the data measures — printed side by side, so the business side can check the code side and vice versa.
3. Zero results must report coverage. "Found nothing" is not "no problems."

The loop

Finding a problem and stopping there leaves the catalog knowing less than the agent does. Each stage below does real work — and each has a stated way of refusing, because a stage that cannot refuse is a stage that will eventually lie:

Stage	What it does	What it refuses to do
discover	Pages the catalog to exhaustion, then compares what it collected against the total the server declared	Trust one page. Measured here: search declared total: 15 and returned 10 — a first-page scanner reports on a subset while claiming "the pipeline"
measure	Puts the catalog's declared cadence and the data's actual MAX timestamp side by side	Emit any verdict with either number missing — that is UNMEASURED, with the reason attached
attribute	Walks lineage to collapse N stale symptoms into root causes vs. downstream echoes	Guess. Unreadable lineage → unattributed, stated as such. A stale table under a healthy parent stays a separate failure
record	Writes every finding — including UNMEASURED — into the graph as tags + structured properties	Invent vocabulary. Only the five declared terms are writable; an unknown state throws
verify	Reads every write back and compares	Trust the write response. A success-response with a disagreeing readback is reported as loudly as a failure — because it is one

What that looks like on the real pipeline — both platforms, one scan (full output in examples/, captured unedited from live runs). Captured 2026-07-30; gap is measured against the clock at run time, so it grows by one each day this sample dataset sits still:

=== discovery ===
  matched 6 dataset(s) under "nyc_taxi"  (walked 16 of 16 declared)

=== findings ===
  STALE        raw_trips              declared every 1d | data ends 2016-03-10 | gap 3795d
  STALE        raw_trips_pg           declared every 1d | data ends 2016-03-10 | gap 3795d   <- postgres
  STALE        mart_daily_summary     declared every 1d | data ends 2016-03-10 | gap 3795d
  STALE        staging_trips          declared every 1d | data ends 2016-03-10 | gap 3795d
  UNMEASURED   v_daily_from_staging   — the catalog never declared how often this should update
  UNMEASURED   v_staging_from_raw     — the catalog never declared how often this should update

=== attribution ===
  4 stale table(s) -> 1 root cause(s), 3 downstream echo(es), 0 unattributed
  ROOT  raw_trips  — fix this one first
  echo  raw_trips_pg         — stale because nyc_taxi.main.raw_trips is    <- cause on another platform
  echo  mart_daily_summary   — stale because staging_trips is
  echo  staging_trips        — stale because raw_trips is

=== write-back ===
  + raw_trips_pg          wrote freshness_stale + freshness_downstream_echo (readback confirmed; 1 pre-existing tag(s) untouched)
  + raw_trips             wrote freshness_stale + freshness_root_cause (readback confirmed; 4 pre-existing tag(s) untouched)
  + v_daily_from_staging  wrote freshness_unmeasured (readback confirmed)
  ...

=== summary ===
  found 4 STALE
  -> 1 table(s) to actually fix, not 4
  -> this run is NOT sufficient to claim "no problems".

Four stale alerts — spanning two platforms — collapse to one table to fix. Two views nobody declared a cadence for are now visibly unmeasured in the DataHub UI instead of silently absent from a report. And summarizeRun — the only place in the codebase allowed to say "no problems found" — requires full coverage and complete discovery and a real verdict for every target and every write verified, before it will say it.

The attribution also reads layered breaks. The datapack's second official instance plants one: raw ends 03-10, but staging ends 03-01 — the transform stopped before its source ran dry. An echo whose data ends earlier than its stale source's is hiding a second failure, and the agent says so instead of folding it into "stale because the parent is":

echo  staging_trips  — stale because nyc_taxi_pipeline.main.raw_trips is
      ! and its data ends 9d earlier than its source's — the transform itself also stopped

Built failure-first

Every stage was built failure-path first, one commit each, in the order a failure would poison the stage after it:

#	Path built first	Why this order
1	UNMEASURED — data side deliberately not connected	An agent that learns to say "stale" before it learns to say "I don't know" ends up with an output layer that always returns something plausible — which destroys the only signal telling you whether the input was any good.
2	UNCOVERED — unknown URN must report coverage	"No results" and "the query never worked" render identically unless you force them apart.
3	STALE — real data, both numbers printed together	The part every freshness checker has.
4	FRESH — an env=DEV control seeded with rows from now	Until a table that is fresh comes back FRESH, nothing proves the agent isn't reporting STALE unconditionally.
5	discovery that stops short must say "walked N of M"	Otherwise scan-by-search silently reintroduces the zero-result lie at fleet scale.
6	attribution that refuses when lineage is unreadable	A guessed root cause sends the on-call engineer to the wrong table with the agent's confidence behind it.
7	write-back readback, mismatch = failure	The write response is a claim about the graph, not evidence. See below for what filing that distinction upstream turned up.

Step 1 is enforced, not aspirational: with SLICE_DISCONNECT_DATA=1, any verdict makes the CLI exit non-zero. That firing is captured in examples/gate-fires.txt rather than merely asserted — a gate that has never been observed failing is itself an output layer that never fails. The same discipline applies to the test suite: every guard broken on purpose turned its tests red — including three conditions inside summarizeRun that did not, until this claim was checked instead of repeated.

What step 2 caught — in this repo, on day one

Step 2's acceptance test was written before the implementation: feed a URN that does not exist; the agent must report coverage, never "no problems". The first run reported COVERED.

get_entities on a nonexistent URN does not return an empty result. It returns:

[{"error": "Entity urn:li:dataset:(...,does_not_exist,PROD) not found",
  "urn":   "urn:li:dataset:(...,does_not_exist,PROD)"}]

— with isError: false at the MCP envelope level. The presence check matched on the echoed urn and ignored error: the server explicitly said not found, and the client read found. The nearest signal (the envelope) won over the source (the payload) — the exact failure mode this agent exists to prevent, committed by its own author.

A scanner built happy-path first would have shipped this. Every typo'd or renamed table in a scan list would count as covered-and-healthy, and the run would report full coverage with zero findings — the most convincing possible way to be wrong.

Fixed with three-valued existence, never merged: found / absent (catalog said not-found — uncovered, not "no problems") / unknown (couldn't ask, non-not-found error, or a bare urn with no substance — which resolves to unknown rather than defaulting to found). The real payload is a regression fixture.

Three clocks, none substitutable

• The data's clock — MAX(DATE(tpep_pickup_datetime)): when data actually last arrived.
• The metadata's clock — when DataHub last saw the metadata. Measured 2026-07-30: the read-only MCP surface doesn't expose it at all (all five tools scanned; the only "freshness" hits were the name and description of the freshness_sla glossary term — an expectation, not a timestamp). And even if it were available it would be the wrong clock: this dataset's metadata was ingested recently while its data ends in 2016. Using it yields a beautiful, wrong number. Recorded in code as a first-class UNMEASURED reason (measureDeclaredLastUpdate), not a comment and not a default.
• The host's clock — the reference "now". Taken from this machine, never from the system under test.

The catalog declares how often the table should update (daily_refresh); the data answers when it actually last did (2016-03-10); the verdict is the argument between them: gap_days = 3795.

Writing is where an agent does damage — so writing gets the constraints

The write-back layer (src/writeback.mjs) enforces three things rather than trusting them:

1. Fixed vocabulary. Only the five terms declared in scripts/setup_agent_metadata.py are writable (3 state tags, 2 attribution tags, plus 3 typed structured properties for the evidence). An unknown state throws in our code before it ever reaches the server.
2. Self-correcting. State tags are mutually exclusive. Every write removes the state tags that no longer apply before adding the one that does — a table that recovered sheds freshness_stale instead of accumulating a contradiction. Re-running is idempotent, and pre-existing human tags are never touched (the readback counts them, untouched, as proof).
3. Readback or it didn't happen. During this hackathon I filed datahub#18753 after finding three storage layers giving three different answers about the same write, hours after it landed. So every write is read back and compared; a mismatch is reported as loudly as a failure, because it is one.

UNMEASURED is written, not skipped. That is the whole point: the catalog gets to carry "we checked this and could not tell, here is why" as a queryable fact. Tools that only write findings leave "nobody knows" looking exactly like "nothing wrong".

Why tags and structured properties, and not Assertions

The obvious question for anyone who knows DataHub: it already models data-quality assertions, so why write findings as tags plus structured properties instead?

Because assertions answer pass or fail, and the state this agent exists to surface is neither. A dataset with no declared cadence has not failed a freshness check — there was no check to fail. A dataset whose source cannot be read has not passed one either. Modelled as an assertion, both collapse into the same place a missing assertion already lives: nowhere. That is precisely the collapse this repo is built to prevent, so encoding it in a pass/fail shape would give up the argument at the first step.

Tags plus structured properties were the vocabulary available to say a third thing — freshness_unmeasured, carrying unmeasured.detail and unmeasured.checked_at so the reason and the time of the attempt travel with it. Five terms, declared up front, and an unknown state throws rather than inventing a sixth.

This is a workaround, and it is worth naming as one. The thing I would actually want is a first-class unknown alongside pass and fail — queryable, visible on the entity page, and distinguishable from "no one has looked". Every quality tool in the ecosystem would stop treating absence as health. Until then, this is the closest honest encoding the platform allows, and the readback in the next section is what keeps it honest.

The platform boundary is real, and the second platform proves it

The data side lives behind two functions in src/datasource.mjs: resolveDataSource — which physical source a URN maps to, by explicit registration keyed <platform>:<container>, never guessed — and resolveTimestampColumn — which column is the data's clock, declared or inferred only when unambiguous. Everything above that boundary (status resolution, coverage, attribution, write-back) is platform-blind.

That claim is demonstrated, not asserted: src/pg.mjs puts postgres on the other side of the boundary — docker exec + psql, no npm dependency, no published port — answering to the identical failure vocabulary. Unreachable container, missing table, ambiguous column, empty table, NULL MAX: each is UNMEASURED with its own stated reason, never a fallback number. A BigQuery or Snowflake reader is the same exercise.

Two details found while crossing the boundary, both now regression-tested:

• Both platforms have a database named nyc_taxi, and a registry keyed on the name alone sent postgres URNs to the sqlite file. The resulting UNMEASURED was honest but its reason was wrong — which is its own kind of lie. The platform is in the URN; the registry must not throw it away.
• Identifiers reach psql -c by string (it has no bind parameters), so every interpolated name is validated first; a hostile name is refused before any SQL is sent — the tests prove it by counting the SQL that was sent.

Run it

Prerequisites:

• Node ≥ 22.5 (uses built-in node:sqlite; developed on Node 24)
• Docker with the DataHub quickstart running on localhost:8080
• The official hackathon NYC Taxi sample dataset ingested (its create_db.py writes ~/.datahub/datasets/nyc-taxi/nyc_taxi.db; override the path with NYC_TAXI_DB if yours lives elsewhere)
• uvx (the MCP server is fetched on demand via uvx mcp-server-datahub@latest; no DataHub token needed — quickstart metadata auth is off)
• Python with acryl-datahub (only for seeding the agent vocabulary and the FRESH control)

npm test                                  # 127 tests, no network needed
python scripts/setup_agent_metadata.py    # declare the 5 tags + 3 properties (once)
node bin/check.mjs                        # read-only check of the default table
node bin/check.mjs --scan nyc_taxi --write            # the full loop: discover -> ... -> verify
node bin/check.mjs --scan nyc_taxi_pipeline --write   # the second official instance (the 9-day break)
node bin/demo.mjs                         # the demo as a reproducible script

The recorded demo predates two output changes committed since: the agent now prints a line to stderr while the MCP server starts, and it no longer says "1 table to actually fix, not 1" when nothing actually collapsed. Same beats, same commands, same results — but the script in this repo is the newer one, and saying so is cheaper than letting a video quietly drift away from the code it claims to be.

The second --write line is not optional if you intend to run the demo: the demo's independent GraphQL check asks the graph for a finding on nyc_taxi_pipeline, and it refuses to continue when there isn't one. That refusal is the point — but from a clean catalog it fires because nothing has written there yet, not because the loop is broken.

If DataHub is not running, the agent says so and exits 2 rather than reporting anything:

starting the DataHub MCP server via uvx and connecting to http://localhost:8080

STOP: this run did not complete, so it reports nothing.
   MCP server exited with code 1 before answering
   [ ... the MCP server's own stderr, verbatim ... ]

   Target: http://localhost:8080. Is the DataHub quickstart running?
   No target was measured, so no target is FRESH.

That path is deliberate. An agent whose whole argument is that unmeasured things must not render as healthy does not get to greet an unreachable catalog with a stack trace — and it does not get to sit out a 90-second timeout when the server process already exited.

The postgres mirror (optional — the sqlite path works without it) needs a container with no published port; seeding and reading both go through docker exec:

docker run -d --name unmeasured-postgres \
  -e POSTGRES_PASSWORD=unmeasured -e POSTGRES_DB=nyc_taxi postgres:16-alpine
python scripts/create_postgres_mirror.py  # seed + emit metadata + cross-platform lineage

If docker is not on your PATH, point DOCKER_EXE at it — both the agent and the seed script honor it.

Sample outputs from real runs live in examples/ — captured for readers who want to see the loop without standing up the stack.

The FRESH control re-seeds with python scripts/create_dev_control.py. It goes stale on its own after a day — deliberately. If you forget to re-seed, the agent reports STALE on the control, which tells you the control is stale, not that the agent is broken. A control that silently stayed green would be the same never-failing output layer this repo exists to avoid.

Found along the way

Building against the sample dataset surfaced reproducible issues upstream, filed during the hackathon: static-assets#218, static-assets#219, static-assets PR#220 (script fixes), and datahub#18753 (platform-level graph-index ghost edges, 3/3 reproduced with a 0/10 control group).

The day the catalog actually held both official instances, this repo's own namePrefix filter silently swallowed the second one: nyc_taxi matched nyc_taxi_pipeline as well, so a scan reported on a set it never announced had changed under it. A filter bug is a coverage bug. It is fixed with the collision as a regression test, and it stays in this README for the same reason the day-one COVERED bug does: the failure mode this agent exists to prevent does not spare its author, which is exactly why the checks have to be structural.

Layout

src/discover.mjs    discovery — pages to exhaustion, reports "walked N of M declared"
src/measure.mjs     measurement layer — answers "what did I measure / why couldn't I"
src/status.mjs      status resolution — no defaults, no fallback verdicts
src/lineage.mjs     attribution — root causes vs echoes (with extra-lag detection), or an honest refusal
src/writeback.mjs   record + verify — fixed vocabulary, self-correcting tags, readback required
src/datasource.mjs  the truth layer's input contract — platform-keyed, explicit, never inferred
src/pg.mjs          the postgres read — docker exec + psql, same failure vocabulary as sqlite
src/mcp.mjs         bare stdio JSON-RPC client for mcp-server-datahub
bin/check.mjs       the agent CLI + self-check gates (violations exit non-zero)
bin/demo.mjs        the demo as a reproducible script — real runs only
scripts/            agent vocabulary; the env=DEV FRESH control; the postgres mirror + lineage
examples/           unedited output from live runs
test/               127 tests, including forged-input and real-payload regression guards

License

Copyright 2026 Ciki Zeng.

Licensed under the Apache License, Version 2.0. See LICENSE for the full text.