Add reservations timeline mirror table - #9864
Conversation
This comment has been minimized.
This comment has been minimized.
| # containing (run time - 2 hours) instead of the default (run time - 1 hour) | ||
| # so each day partition gets a final rewrite at 01:00 the next day. | ||
| date_partition_parameter: null | ||
| destination_table: reservations_timeline_v1${{ (logical_date - macros.timedelta(hours=1)).strftime("%Y%m%d") }} |
There was a problem hiding this comment.
The scheduling comment says the 01:00 run must target logical_date - 2 hours to rewrite the preceding day’s final partition, but this still subtracts one hour. At 01:00 that selects the new day. Please change both this expression and submission_date below to two hours.
There was a problem hiding this comment.
The target is (run time - 2 hours). logical_date is (run time - 1 hour) since it's the start of the data interval, therefore (logical_date - 1 hour) = (run time - 2 hour)
hamzahamidi
left a comment
hamzahamidi
left a comment
There was a problem hiding this comment.
The scheduling expressions use logical_date - 1 hour, even though the intended 01:00 repair needs the preceding day from logical_date - 2 hours. The inline comment identifies the affected line and paired parameter.
4578108 to
60ed9f2
Compare
60ed9f2 to
fb7aeac
Compare
This comment has been minimized.
This comment has been minimized.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
This adds monitoring_derived.reservations_timeline_v1, an hourly mirror of moz-fx-bigquery-reserv-global.region-us.INFORMATION_SCHEMA.RESERVATIONS_TIMELINE, plus its schema and a dry_run.skip entry because the dry run service account can't read the admin project.
I checked the scheduling arithmetic and it holds up: for an hourly DAG logical_date is the start of the data interval, so logical_date - 1h is run time - 2h, and the 01:00 run does land on the previous day's partition after the last minute's per_second_details has filled in. The column list and order in query.sql match schema.yaml exactly, including the nested autoscale, scheduling_policy, labels, and per_second_details records. Partitioning on period_start with the $YYYYMMDD decorator is consistent with the DATE(period_start) = @submission_date filter, and monitoring_derived's dataset_metadata.yaml already grants mozilla-confidential/data-viewers, so no table-level access config is needed.
Worth noting for reviewers: because the query is on the dry run skip list, CI will not catch a mismatch between schema.yaml and the source view's actual columns — the first failure would surface as a failed Airflow task rather than a failed check. Also, with date_partition_parameter: null this can't be repaired with bqetl query backfill, which the description does call out. Only one minor comment inline.
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Integration report
|
3 participants
Description
Copy of
moz-fx-bigquery-reserv-global.region-us.INFORMATION_SCHEMA.RESERVATIONS_TIMELINEso it can used by moz-confidential viewers without being given permissions to the admin project.Airflow should have access to the information schema view through https://github.com/mozilla/global-platform-admin/blob/f4e504d9643c01002c3deef75fa83b64bb99a0ed/bigquery-reservations/tf/main.tf#L56
Reviewer, please follow this checklist