Skip to content

fix(nyc-taxi): make planted freshness defects deterministic - #226

Open
danielAsaboro wants to merge 1 commit into
datahub-project:mainfrom
danielAsaboro:agent/fix-nyc-taxi-scenario
Open

fix(nyc-taxi): make planted freshness defects deterministic#226
danielAsaboro wants to merge 1 commit into
datahub-project:mainfrom
danielAsaboro:agent/fix-nyc-taxi-scenario

Conversation

@danielAsaboro

Copy link
Copy Markdown

Summary

The committed nyc_taxi_pipeline.db does not currently contain the two
freshness signals described by its README: the observable stage lag is nine
calendar days rather than three, and no mart row has trip_count = 0.

This change makes the generator deterministic for the sparse 250k-row sample:

  • derive the cutoff from the raw pickup business timestamp instead of the
    staging table that is about to be truncated;
  • plant the empty load on an existing median mart date instead of assuming
    cutoff - 7 days exists;
  • report the actual post-cutoff lag, which can exceed the three-day target when
    sampled dates are sparse;
  • regenerate the committed pipeline database and update the README with its
    observed dates; and
  • add focused stdlib regression tests for the raw-date cutoff and guaranteed
    empty-load row.

The regenerated committed artifact has:

Signal Value
Raw max business date 2016-03-10
Staging max business date 2016-03-01
Mart max business date 2016-03-01
Observable stage lag 9 calendar days
Empty-load mart date 2015-01-21
Empty-load trip_count 0

The cutoff target is still raw max minus three calendar days (2016-03-07).
The observable lag is nine days because the committed sample has no rows from
March 2 through March 7. The README now makes that distinction explicit.

The approximately 86.5 MiB generated database is intentionally included in
this patch. This repository already tracks the generated NYC Taxi artifacts,
and its README directs demo users to the committed pipeline database. Updating
only the generator would leave the default download/demo path with the broken
signals reported in #219 and #222. The clean nyc_taxi.db source artifact is
unchanged.

Closes #219.
Closes #222.

Verification

python3 -m unittest -v tests/test_nyc_taxi_create_db.py
Ran 2 tests in 0.010s
OK
cd datasets/nyc-taxi
python3 create_db.py --pipeline-from-existing
Raw data through: 2016-03-10
Staging/mart cutoff target: 2016-03-07 (3 calendar days)
Observed staging max: 2016-03-01 (9 calendar days behind raw)
Empty load: 2015-01-21 shows 0 trips
SELECT MAX(DATE(tpep_pickup_datetime)) FROM raw_trips;
SELECT MAX(trip_date) FROM staging_trips;
SELECT MAX(trip_date) FROM mart_daily_summary;
SELECT trip_date, trip_count
FROM mart_daily_summary
WHERE trip_count = 0;
2016-03-10
2016-03-01
2016-03-01
2015-01-21|0

Scope

This changes only the NYC Taxi dataset generator, its focused regression test,
the generated planted database, and the corresponding README section. It does
not change the clean nyc_taxi.db source artifact or any DataHub service.

@danielAsaboro
danielAsaboro marked this pull request as ready for review August 10, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant