fix: remove --date argument from nimbus_sizing_clients_v1 metadata template - #9856
Closed
yashikakhurana wants to merge 13 commits into
Closed
yashikakhurana wants to merge 13 commits into
yashikakhurana wants to merge 13 commits into
Conversation
…obile experiments Add Fenix and iOS targeting tables, group experiments by appName, use hash-based 10% sampling for mobile (no sample_id column in derived tables), and include platform in results written to GCS and BigQuery.
- Add submission_date partition filter to dry-run probe so mobile tables with require_partition_filter=true pass validation - Wrap per-platform loop body in try/except so one platform failure does not discard results already accumulated for other platforms - Use MOD(..., 100) < SAMPLE_ID_MAX for mobile sampling to mirror desktop sample_id range and keep both branches linked to the constant - Replace table+is_mobile params with app_name in build_query and _dry_run_sql; derive config from _APP_CONFIG to make mismatches unrepresentable
…gation
Create nimbus_sizing_clients_{desktop,fenix,ios}_v1 tables that pre-materialise
a 7-day rolling window of deduplicated, 10%-sampled clients per app. The
experiment_population_estimates_v1 query.py now just runs COUNTIF queries
against these pre-built tables and ships results to GCS and BigQuery, with
no inline deduplication, date arithmetic, or is_mobile branching needed.
…ient tables
Replaces three manually-maintained SQL files with a single Jinja2 template
in sql_generators/nimbus_population_sizing/. The generator renders
nimbus_sizing_clients_{desktop,fenix,ios}_v1 from one template with
conditional blocks for desktop vs mobile differences (client ID field,
sampling method, date column). Running `bqetl generate
nimbus_population_sizing` regenerates all three tables from the single
source of truth.
…toring generator
Moves the per-app nimbus sizing client pool tables into the existing
experiment_monitoring sql_generator following Mike's suggestion, using
the same pattern as experiment_enrollment_aggregates_v1 with {% if %}
conditional blocks for desktop vs Glean mobile apps.
Changes:
- Extend experiment_monitoring/__init__.py to support destination_project
per query in per_app: true mode
- Add nimbus_sizing_clients_v1 template to experiment_monitoring/templates/
with desktop/mobile conditional blocks
- Register nimbus_sizing_clients_v1 in templating.yaml targeting
moz-fx-data-experiments, skipping all non-sizing apps
- Move generated tables from monitoring.nimbus_sizing_clients_{app}_v1
to {app}_derived.nimbus_sizing_clients_v1 to match the generator naming
- Remove standalone nimbus_population_sizing generator
- Update query.py table references to new paths
…ference new sizing table paths
… lines in generated output
Generated output from sql_generators is not committed to sql/ — the template in sql_generators/experiment_monitoring/templates/ is the source of truth and bqetl generate produces the files at deploy time.
…ents Intermediate client pool tables belong in moz-fx-data-shared-prod alongside their source data. Only the final GCS/BQ output (experiment_population_estimates_v1) stays in moz-fx-data-experiments.
…itoring generator No query currently uses destination_project; revert the __init__.py change to keep the diff minimal.
…mplate --date is not a valid bq query flag; only valid for Python scripts with argparse. bqetl auto-adds --parameter=submission_date:DATE from the SQL.
Contributor
Author
|
Superseded by a clean branch from main |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--dateis not a validbq queryCLI flag — it is only valid for Python scripts with argparse--parameter=submission_date:DATEfrom@submission_datein the SQL, so theargumentsline is not needed and was causing all threenimbus_sizing_clients_v1tasks to fail withUnknown command line flag 'date'Jira: EXP-7366