ci(docsbase): versioned connector docs + config schemas image, auto-refreshed on release - #402
ci(docsbase): versioned connector docs + config schemas image, auto-refreshed on release#402shirshanka wants to merge 1 commit into
Conversation
0e58838 to
6a0463e
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
6a0463e to
45af722
Compare
| - name: Log in to Docker Hub | ||
| if: ${{ (github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && inputs.dry_run == false }} | ||
| uses: docker/login-action@v3 | ||
| with: | ||
| username: acryldata | ||
| password: ${{ secrets.DOCSBASE_DH_TOKEN }} | ||
|
|
||
| - name: Push | ||
| if: ${{ (github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && inputs.dry_run == false }} |
There was a problem hiding this comment.
Blocker: github.event_name is never workflow_call here, so nothing is ever pushed.
In a called workflow the github context describes the caller's event. publish-pypi-release.yml triggers on release: published, so at this point github.event_name == 'release' — and this guard, the identical one on Push (L205), and the one on Dry-run notice (L223) are all false on the only automated path.
Net effect: a full docGen runs per version, the image is built, nothing is pushed, and the job reports success with no log line explaining the skip.
Second issue in the same block: DOCSBASE_DH_TOKEN is declared required: false (L50-52) but the caller hardcodes dry_run: false, so this login always runs on a release with no emptiness guard. docker-unified.yml:120-125 doesn't put secrets.X != '' in an if: (the secrets context isn't available there) — it evaluates it in env: and exports an output. This matters for the datahub-project/datahub mirror too, which receives this file via sync-upstream.yml and will never have the secret.
The suggestion drops the event sniffing, adds the repo's guard pattern, and pins the action. Note it also moves docker/login-action v3 → the v4.0.0 SHA the rest of the repo pins.
| - name: Log in to Docker Hub | |
| if: ${{ (github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && inputs.dry_run == false }} | |
| uses: docker/login-action@v3 | |
| with: | |
| username: acryldata | |
| password: ${{ secrets.DOCSBASE_DH_TOKEN }} | |
| - name: Push | |
| if: ${{ (github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && inputs.dry_run == false }} | |
| - name: Check whether docker login is possible | |
| id: docker-login | |
| env: | |
| ENABLE_DOCKER_LOGIN: ${{ secrets.DOCSBASE_DH_TOKEN != '' }} | |
| run: echo "docker-login=$ENABLE_DOCKER_LOGIN" >> "$GITHUB_OUTPUT" | |
| - name: Log in to Docker Hub | |
| if: ${{ inputs.dry_run == false && steps.docker-login.outputs.docker-login == 'true' }} | |
| uses: docker/login-action@b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # v4.0.0 | |
| with: | |
| username: acryldata | |
| password: ${{ secrets.DOCSBASE_DH_TOKEN }} | |
| - name: Push | |
| if: ${{ inputs.dry_run == false && steps.docker-login.outputs.docker-login == 'true' }} |
There was a problem hiding this comment.
Fixed in 583d04e: dropped the github.event_name check entirely and gated on inputs.dry_run alone, plus a new "Check whether docker login is possible" step that verifies DOCSBASE_DH_TOKEN is actually set before attempting login/push.
| } >> "$GITHUB_STEP_SUMMARY" | ||
|
|
||
| - name: Dry-run notice | ||
| if: ${{ (github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && inputs.dry_run == true }} |
There was a problem hiding this comment.
Same root cause as the login/push guards above — github.event_name is the caller's event, so this never fires either and a dry run prints nothing.
| if: ${{ (github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && inputs.dry_run == true }} | |
| if: ${{ inputs.dry_run == true }} |
There was a problem hiding this comment.
Same fix as above — this step now just checks inputs.dry_run == true.
|
|
||
| assemble-and-publish: | ||
| needs: generate | ||
| if: ${{ !cancelled() }} # publish whatever versions succeeded, even if some legs failed |
There was a problem hiding this comment.
!cancelled() ships a corpus with holes under a tag that claims a contiguous range.
With fail-fast: false (L78) a failed leg uploads no artifact but this job still runs, and floor/ceiling are computed from only the surviving trees. So if 1.6.0.5–1.6.0.8 fail in a 12-version backfill, the image still publishes as 1.6.0.1-1.6.0.10 and overwrites :latest, with org.acryl.docsbase.floor/ceiling labels asserting the whole span. A consumer resolving the range and reading /1.6.0.6/schemas/ hits a missing directory at runtime — and the previous good :latest is already gone.
Simplest fix is to not publish an incomplete corpus:
| if: ${{ !cancelled() }} # publish whatever versions succeeded, even if some legs failed | |
| if: ${{ success() }} # a failed leg means a hole in the range — don't publish a partial corpus |
If you do want partial publishes, then the tag and labels shouldn't imply density — carry the explicit version list instead of <floor>-<ceiling>.
There was a problem hiding this comment.
Fixed — changed to success() per your suggestion.
| cp docs/generated/ingestion/config_schemas/*_config.json "out/$V/schemas/" 2>/dev/null || \ | ||
| echo "::warning::no config_schemas produced for $V" | ||
| cp -R metadata-ingestion/docs/sources/. "out/$V/docs/" 2>/dev/null || \ | ||
| echo "::warning::no docs/sources for $V" |
There was a problem hiding this comment.
These two copies fail soft, so a version can be counted as covered with no schemas/ at all.
2>/dev/null || echo "::warning::..." defeats set -euo pipefail on L112. If docGen runs but emits no *_config.json, the leg still uploads an artifact containing only docs/, and Compute coverage counts that version in versions_count and in the floor-ceiling tag — so the image advertises coverage for a version whose /<version>/schemas/ doesn't exist, found only at consumption time.
Worse case: both copies fail, and if-no-files-found: warn (L125) uploads nothing, so the version silently disappears from the corpus while the run stays green.
| cp docs/generated/ingestion/config_schemas/*_config.json "out/$V/schemas/" 2>/dev/null || \ | |
| echo "::warning::no config_schemas produced for $V" | |
| cp -R metadata-ingestion/docs/sources/. "out/$V/docs/" 2>/dev/null || \ | |
| echo "::warning::no docs/sources for $V" | |
| shopt -s nullglob | |
| schemas=(docs/generated/ingestion/config_schemas/*_config.json) | |
| if [ ${#schemas[@]} -eq 0 ]; then | |
| echo "::error::no config_schemas produced for $V" | |
| exit 1 | |
| fi | |
| cp "${schemas[@]}" "out/$V/schemas/" | |
| cp -R metadata-ingestion/docs/sources/. "out/$V/docs/" |
Worth flipping L125 to if-no-files-found: error as well, so a leg that produces nothing fails instead of vanishing.
There was a problem hiding this comment.
Fixed — the corpus assembly step now fails the leg (exit 1) if config_schemas/ wasn't produced, and if-no-files-found is now error instead of warn.
| run: | | ||
| set -euo pipefail | ||
| if [ -n "${{ inputs.versions }}" ]; then | ||
| M='${{ inputs.versions }}' | ||
| else | ||
| M="$(python3 metadata-ingestion/scripts/docsbase/list_versions.py --days ${{ inputs.days }})" | ||
| fi | ||
| echo "matrix=$M" >> "$GITHUB_OUTPUT" | ||
| echo "Selected versions: $M" |
There was a problem hiding this comment.
inputs.* interpolated straight into shell — injectable, in a job that holds the push token.
${{ }} substitutes into the script source before bash parses it, so input text becomes shell syntax. A workflow_dispatch with days set to 90; curl -X POST -d "$DOCSBASE_DH_TOKEN" https://… expands on L70 into a second command, in a workflow that secrets: inherit supplies the Docker Hub token to. Less dramatically, a single quote or newline in versions breaks the M='...' assignment on L68 and the $GITHUB_OUTPUT write on L72, giving an opaque fromJson failure downstream.
Passing through the environment means the value can never be re-parsed as code:
| run: | | |
| set -euo pipefail | |
| if [ -n "${{ inputs.versions }}" ]; then | |
| M='${{ inputs.versions }}' | |
| else | |
| M="$(python3 metadata-ingestion/scripts/docsbase/list_versions.py --days ${{ inputs.days }})" | |
| fi | |
| echo "matrix=$M" >> "$GITHUB_OUTPUT" | |
| echo "Selected versions: $M" | |
| env: | |
| VERSIONS: ${{ inputs.versions }} | |
| DAYS: ${{ inputs.days }} | |
| run: | | |
| set -euo pipefail | |
| if [ -n "$VERSIONS" ]; then | |
| M="$VERSIONS" | |
| else | |
| M="$(python3 metadata-ingestion/scripts/docsbase/list_versions.py --days "$DAYS")" | |
| fi | |
| echo "matrix=$M" >> "$GITHUB_OUTPUT" | |
| echo "Selected versions: $M" |
inputs.image_tag at L170-171 needs the same treatment.
There was a problem hiding this comment.
Fixed — days/versions/image_tag are now passed via env: instead of interpolated into the script body. Applied the same treatment to matrix.version, which flows from the same versions input and had the same exposure.
| # The `push:` trigger on the dev branch is for validation and should be removed | ||
| # before merge — steady state is workflow_dispatch (manual/backfill) plus | ||
| # workflow_call from publish-pypi-release.yml (auto-refresh on each final release). | ||
| name: connector-docsbase |
There was a problem hiding this comment.
No concurrency group on a workflow that writes a mutable :latest tag.
This repo has published three four-part releases in one day (v1.6.0.12/13/14 around 2026-07-09). Two release: published events start two runs whose generate legs take very different times — each does a full docGen including a fresh uv pip install .[docs] — and both Push steps write …docsbase:latest. Last to finish wins, so :latest can end up serving 1.6.0.12 while 1.6.0.14 is already on PyPI, with no error anywhere.
cancel-in-progress: false rather than true, since each run is a real corpus build you don't want to lose:
| name: connector-docsbase | |
| name: connector-docsbase | |
| concurrency: | |
| # Runs all write the mutable :latest tag — serialize them. | |
| group: connector-docsbase | |
| cancel-in-progress: false |
There was a problem hiding this comment.
Added a concurrency group (cancel-in-progress: false) exactly as suggested.
| # NOTE (scaffold): action refs use version tags for readability; pin to SHAs to | ||
| # match repo policy before merge. JDK/Python versions mirror documentation.yml. | ||
| # The `push:` trigger on the dev branch is for validation and should be removed | ||
| # before merge — steady state is workflow_dispatch (manual/backfill) plus | ||
| # workflow_call from publish-pypi-release.yml (auto-refresh on each final release). |
There was a problem hiding this comment.
This is the "pin before merge" note, and the refs are still floating — flagging so it doesn't ride along into master. A retagged upstream release would run unreviewed code in a job holding the Docker Hub token and every inherited secret.
The SHAs documentation.yml (which the header says this mirrors) currently pins, if useful:
| action | pinned ref |
|---|---|
actions/checkout |
de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 |
actions/setup-java |
be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5 |
actions/setup-python |
a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6 |
actions/upload-artifact |
bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0 |
actions/download-artifact |
3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 |
docker/login-action |
b45d80f862d83dbcd57f89517bcf500b2ab88fb2 # v4.0.0 |
Note those are newer majors than the @v4/@v5/@v3 used here, so pinning is also a version bump — worth a quick check that upload-artifact v7 / download-artifact v8 behave the same for this matrix pattern.
The second half of this note (push: trigger is for validation, remove before merge) is also still outstanding.
There was a problem hiding this comment.
Pinned all action refs to the SHAs you listed, including bumping docker/login-action to v4.0.0.
| # docsbase only wants final X.Y.Z.W releases, not rc/post/dev suffixed tags. | ||
| if [[ "$TAG" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then |
There was a problem hiding this comment.
This regex excludes DataHub's real three-segment releases.
v1.7.0, v1.6.0, v1.5.0 and v1.0.0 all exist as release tags — minor and major bumps ship as X.Y.Z by design. For tag v1.7.0, TAG="1.7.0" fails this match, is_final_release=false, and refresh_docsbase is skipped entirely.
The same four-part pattern is duplicated in metadata-ingestion/scripts/docsbase/list_versions.py:16 (_FOUR_PART), so those versions are dropped from manual/backfill matrices too — they can't enter the corpus by any route, even though :latest's range label spans across them (e.g. 1.6.0.17-1.7.0.3).
The comment says the intent is "final releases, not rc/post/dev suffixed tags", which is really "three or four segments, no suffix":
| # docsbase only wants final X.Y.Z.W releases, not rc/post/dev suffixed tags. | |
| if [[ "$TAG" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then | |
| # docsbase wants final X.Y.Z and X.Y.Z.W releases, not rc/post/dev suffixed tags. | |
| if [[ "$TAG" =~ ^[0-9]+(\.[0-9]+){2,3}$ ]]; then |
list_versions.py:16 needs the matching change: re.compile(r"^\d+(\.\d+){2,3}$").
There was a problem hiding this comment.
Fixed everywhere this pattern appeared: list_versions.py's _FINAL_RELEASE, the is_final_release check here, and the coverage/prune regex in connector-docsbase.yml (deduped there into one DOCSBASE_VERSION_RE workflow-level env var) — all now ^[0-9]+(\.[0-9]+){2,3}$.
| needs: [setup, push_to_pypi] | ||
| if: ${{ needs.setup.outputs.is_final_release == 'true' }} | ||
| uses: ./.github/workflows/connector-docsbase.yml | ||
| secrets: inherit |
There was a problem hiding this comment.
secrets: inherit is broader than this needs.
The callee declares only DOCSBASE_DH_TOKEN, but inherit hands it the whole set — including TWINE_PASSWORD — while it checks out arbitrary release tags and runs ./gradlew :metadata-ingestion:docGen, i.e. executes that tag's build scripts.
| secrets: inherit | |
| secrets: | |
| DOCSBASE_DH_TOKEN: ${{ secrets.DOCSBASE_DH_TOKEN }} |
There was a problem hiding this comment.
Narrowed to just DOCSBASE_DH_TOKEN.
| uses: ./.github/workflows/connector-docsbase.yml | ||
| secrets: inherit | ||
| with: | ||
| versions: '["${{ needs.setup.outputs.tag }}"]' |
There was a problem hiding this comment.
Passing one version rebuilds :latest from just that version — is that intended?
The callee builds the image FROM scratch out of only the artifacts this run produced, and nothing merges in the previously published corpus. So on v1.7.0.4 the matrix has one entry, floor == ceiling, and it publishes :1.7.0.4-1.7.0.4 plus :latest containing exactly one version. Anything resolving :latest for 1.7.0.3 or earlier then gets a missing path, and the prior multi-version :latest is only recoverable by re-running a backfill.
I might be misreading the intended lifecycle. If :latest is meant to be single-version, then the <floor>-<ceiling> range tag and the sliding-window framing read as promising otherwise. If it's meant to accumulate, this needs to either pull the previous corpus forward before assembling, or pass the full window (days:) instead of the one new tag.
No suggestion here since it depends which way you want it.
There was a problem hiding this comment.
Went with accumulate: assemble-and-publish now pulls :latest forward, merges in this run's freshly generated versions (which win over stale copies), then prunes anything outside the days retention window — unioned with this run's own versions, so an explicit backfill of an old version isn't immediately pruned in the same run that generated it. Verified live, not just in theory: a single-version dry run correctly showed the merge+prune behavior, then I ran a real (non-dry) full backfill that republished :latest with clean coverage 1.6.0..1.7.0.4 (21 versions, digest sha256:c6b390c54e5f4583ec2f5f295708e74b73ba3fdba44a13800215949f5b0551f5).
…efreshed on release
For each acryl-datahub final release (X.Y.Z or X.Y.Z.W), checks out the tag,
runs docGen (config schemas) and captures metadata-ingestion/docs/sources,
then assembles a single versioned corpus image (/<version>/{docs,schemas}
layout) published as acryldata/datahub-ingestion-docsbase, tagged by the
<floor>-<ceiling> version range it covers plus a moving :latest. Coverage is
recorded both in an in-image CORPUS_MANIFEST.json and as OCI labels so a
consumer can inspect what a given digest covers before pinning it.
Adds metadata-ingestion/scripts/docsbase/list_versions.py, which queries
PyPI for final acryl-datahub releases within a lookback window (used to
build the version matrix for workflow_dispatch backfills).
Wires connector-docsbase.yml up as a workflow_call target from
publish-pypi-release.yml: once a release's push_to_pypi job succeeds, and
only for a genuine final X.Y.Z / X.Y.Z.W tag, it re-runs the docsbase build
for just that version and publishes it (non-dry-run) so the image stays
current without a manual trigger. workflow_dispatch remains available for
manual/backfill runs, with a dry_run input that builds without pushing.
Addresses review from treff7es on PR #402:
- Fix: github.event_name is the CALLER's event inside a workflow_call, never
'workflow_call' — the login/push/dry-run-notice guards never fired on the
automated path. Drop the event_name check; gate on inputs.dry_run alone,
plus a new step that checks DOCSBASE_DH_TOKEN is actually set before
attempting login/push.
- Fix: the four-part-only version regex (in both list_versions.py and
publish-pypi-release.yml's is_final_release check, plus the coverage
computation in this file) excluded real three-part DataHub releases
(v1.7.0, v1.6.0, ...) from ever entering the corpus. Now accepts X.Y.Z or
X.Y.Z.W.
- Fix: assemble-and-publish ran on !cancelled(), so a failed matrix leg could
still publish a corpus with a hole in it under a range tag that claims
contiguous coverage. Changed to success().
- Fix: the per-version corpus assembly step swallowed a missing
config_schemas/ or docs/sources with `2>/dev/null || warning`, so a version
could be counted as covered with nothing in it. Now fails the leg loudly.
- Fix: inputs.days/versions/image_tag were interpolated directly into shell
script source instead of passed via env, in a job that holds the Docker
Hub token. Same treatment applied to matrix.version, which flows from the
same versions input.
- Fix: no concurrency group meant two releases landing close together could
race on the mutable :latest tag. Added one (cancel-in-progress: false, so
real corpus builds aren't lost).
- Fix: secrets: inherit handed the reusable workflow the whole secret set
(including TWINE_PASSWORD) when it only needs DOCSBASE_DH_TOKEN. Narrowed
to just that secret.
- Pinned all action refs to the SHAs already used elsewhere in this repo
(documentation.yml), including a docker/login-action v3 -> v4.0.0 bump.
Not addressed here (open design question, tracked separately): whether
:latest should accumulate versions across runs or reflect only the versions
passed to a given run — currently a single-version workflow_call run
rebuilds :latest from just that one version.
45af722 to
abefd05
Compare
…efreshed on release
For each acryl-datahub final release (X.Y.Z or X.Y.Z.W), checks out the tag,
runs docGen (config schemas) and captures metadata-ingestion/docs/sources,
then assembles a single versioned corpus image (/<version>/{docs,schemas}
layout) published as acryldata/datahub-ingestion-docsbase, tagged by the
<floor>-<ceiling> version range it covers plus a moving :latest. Coverage is
recorded both in an in-image CORPUS_MANIFEST.json and as OCI labels so a
consumer can inspect what a given digest covers before pinning it.
:latest is a sliding window, not a per-run snapshot: assemble-and-publish
pulls it forward, merges in this run's freshly generated versions (which
win over stale copies), then prunes anything outside the `days` retention
window (unioned with this run's own versions, so a one-off backfill of an
old version isn't immediately pruned in the same run that generated it).
This is what makes a single-version auto-refresh safe to publish as :latest
without regressing previously-published versions.
Adds metadata-ingestion/scripts/docsbase/list_versions.py, which queries
PyPI for final acryl-datahub releases within a lookback window (used both
to build the version matrix for workflow_dispatch backfills and to compute
the retention window every run).
Wires connector-docsbase.yml up as a workflow_call target from
publish-pypi-release.yml: once a release's push_to_pypi job succeeds, and
only for a genuine final X.Y.Z / X.Y.Z.W tag, it re-runs the docsbase build
for just that version and publishes it (non-dry-run) so the image stays
current without a manual trigger. workflow_dispatch remains available for
manual/backfill runs, with a dry_run input that builds without pushing.
Addresses review from treff7es on PR #402:
- Fix: github.event_name is the CALLER's event inside a workflow_call, never
'workflow_call' — the login/push/dry-run-notice guards never fired on the
automated path. Drop the event_name check; gate on inputs.dry_run alone,
plus a new step that checks DOCSBASE_DH_TOKEN is actually set before
attempting login/push.
- Fix: the four-part-only version regex (in list_versions.py,
publish-pypi-release.yml's is_final_release check, and the coverage
computation in this file) excluded real three-part DataHub releases
(v1.7.0, v1.6.0, ...) from ever entering the corpus. Now accepts X.Y.Z or
X.Y.Z.W everywhere, via one shared DOCSBASE_VERSION_RE within this file.
- Fix: assemble-and-publish ran on !cancelled(), so a failed matrix leg could
still publish a corpus with a hole in it under a range tag that claims
contiguous coverage. Changed to success().
- Fix: the per-version corpus assembly step swallowed a missing
config_schemas/ or docs/sources with `2>/dev/null || warning`, so a version
could be counted as covered with nothing in it. Now fails the leg loudly.
- Fix: inputs.days/versions/image_tag were interpolated directly into shell
script source instead of passed via env, in a job that holds the Docker
Hub token. Same treatment applied to matrix.version, which flows from the
same versions input.
- Fix: no concurrency group meant two releases landing close together could
race on the mutable :latest tag. Added one (cancel-in-progress: false, so
real corpus builds aren't lost).
- Fix: secrets: inherit handed the reusable workflow the whole secret set
(including TWINE_PASSWORD) when it only needs DOCSBASE_DH_TOKEN. Narrowed
to just that secret.
- Fixed (this round): :latest getting rebuilt from only the versions passed
to a given run instead of accumulating — see the accumulate/prune
description above.
- Pinned all action refs to the SHAs already used elsewhere in this repo
(documentation.yml), including a docker/login-action v3 -> v4.0.0 bump.
abefd05 to
a1e13af
Compare
…efreshed on release
For each acryl-datahub final release (X.Y.Z or X.Y.Z.W), checks out the tag,
runs docGen (config schemas) and captures metadata-ingestion/docs/sources,
then assembles a single versioned corpus image (/<version>/{docs,schemas}
layout) published as acryldata/datahub-ingestion-docsbase, tagged by the
<floor>-<ceiling> version range it covers plus a moving :latest. Coverage is
recorded both in an in-image CORPUS_MANIFEST.json and as OCI labels so a
consumer can inspect what a given digest covers before pinning it.
:latest is a sliding window, not a per-run snapshot: assemble-and-publish
pulls it forward, merges in this run's freshly generated versions (which
win over stale copies), then prunes anything outside the `days` retention
window (unioned with this run's own versions, so a one-off backfill of an
old version isn't immediately pruned in the same run that generated it).
This is what makes a single-version auto-refresh safe to publish as :latest
without regressing previously-published versions.
Adds metadata-ingestion/scripts/docsbase/list_versions.py, which queries
PyPI for final acryl-datahub releases within a lookback window (used both
to build the version matrix for workflow_dispatch backfills and to compute
the retention window every run).
Wires connector-docsbase.yml up as a workflow_call target from
publish-pypi-release.yml: once a release's push_to_pypi job succeeds, and
only for a genuine final X.Y.Z / X.Y.Z.W tag, it re-runs the docsbase build
for just that version and publishes it (non-dry-run) so the image stays
current without a manual trigger. workflow_dispatch remains available for
manual/backfill runs, with a dry_run input that builds without pushing.
Addresses review from treff7es on PR #402:
- Fix: github.event_name is the CALLER's event inside a workflow_call, never
'workflow_call' — the login/push/dry-run-notice guards never fired on the
automated path. Drop the event_name check; gate on inputs.dry_run alone,
plus a new step that checks DOCSBASE_DH_TOKEN is actually set before
attempting login/push.
- Fix: the four-part-only version regex (in list_versions.py,
publish-pypi-release.yml's is_final_release check, and the coverage
computation in this file) excluded real three-part DataHub releases
(v1.7.0, v1.6.0, ...) from ever entering the corpus. Now accepts X.Y.Z or
X.Y.Z.W everywhere, via one shared DOCSBASE_VERSION_RE within this file.
- Fix: assemble-and-publish ran on !cancelled(), so a failed matrix leg could
still publish a corpus with a hole in it under a range tag that claims
contiguous coverage. Changed to success().
- Fix: the per-version corpus assembly step swallowed a missing
config_schemas/ or docs/sources with `2>/dev/null || warning`, so a version
could be counted as covered with nothing in it. Now fails the leg loudly.
- Fix: inputs.days/versions/image_tag were interpolated directly into shell
script source instead of passed via env, in a job that holds the Docker
Hub token. Same treatment applied to matrix.version, which flows from the
same versions input.
- Fix: no concurrency group meant two releases landing close together could
race on the mutable :latest tag. Added one (cancel-in-progress: false, so
real corpus builds aren't lost).
- Fix: secrets: inherit handed the reusable workflow the whole secret set
(including TWINE_PASSWORD) when it only needs DOCSBASE_DH_TOKEN. Narrowed
to just that secret.
- Fixed (this round): :latest getting rebuilt from only the versions passed
to a given run instead of accumulating — see the accumulate/prune
description above.
- Pinned all action refs to the SHAs already used elsewhere in this repo
(documentation.yml), including a docker/login-action v3 -> v4.0.0 bump.
a1e13af to
583d04e
Compare
|
@treff7es thanks for the thorough review — all findings addressed (replied inline on each), summary here:
Also resolved the accumulate/prune design question: Ready for another look whenever you have a chance. |
treff7es
left a comment
treff7es
left a comment
There was a problem hiding this comment.
Approving — round 1 all looks properly addressed, and I verified the result rather than taking it on trust: the published :latest really does carry 21 versions, 1.6.0 → 1.7.0.4, built from 583d04eb, with the layout and labels the manifest claims.
Everything below is about the new accumulate/prune code that came in as the fix for my :latest-gets-clobbered comment. The design is right, and the commenting is genuinely good — the github.event_name note in particular is the kind of thing that saves the next person an afternoon. My findings all cluster on one theme: the read-modify-write cycle over :latest has several paths that fail green and silently shrink the published corpus.
Approving rather than blocking because the blast radius is contained: refresh_docsbase runs after push_to_pypi, so it can't break a release; the corpus is fully regenerable with one workflow_dispatch; and nothing consumes the image yet (release-lock.yml / DOCSBASE_DIGEST don't exist in this repo). Worst case is a red job on a successful release, or a stale image.
Three I'd land before merge
- Move
docker loginabove the accumulate step, and narrow the "no image yet" classifier (L201). Docker Hub returns the same message for missing and unauthorized — I reproduced it:pull access denied for …, repository does not exist or may require 'docker login'. That matchesrepository does not exist, so an auth failure reads as "fresh start" and republishes:latestwith only this run's versions. The repo is public today so it's latent, but it's the one failure that destroys accumulated state. Moving the login also kills the anonymous-pull rate-limit flake on shared runner IPs. - Fail when
dry_run == falseandDOCSBASE_DH_TOKENis empty (L293). Right now login and push both skip, the dry-run notice also skips (dry_runis false), and the job goes green having built an image and thrown it away. Your own test plan still has "confirm the secret is configured" unchecked, which makes this the most likely thing to actually happen on the first release after merge. - Drop the two
|| trues (L181, L192). Together they're what turn "no artifacts downloaded" into a confident### docsbase publishedstep summary.
Answering your own open question from round 1
You asked whether upload-artifact v7 / download-artifact v8 behave the same for this matrix pattern. They don't, in one way that matters — see the comment on L161.
Not blocking, but worth deciding before anyone pins this image
The schema-count floor (L139). It's the only failure that produces a corpus which looks complete — right versions, right manifest, quietly missing connectors — and a consumer can't tell "this connector didn't exist at 1.6.0" from "the build dropped it."
One process note: because uses: ./.github/workflows/… resolves at the caller's ref, and the caller runs at the release tag, every fix here only reaches the automated path on releases cut afterwards. So "fix it in a follow-up" has a real delay attached — worth knowing when deciding what rides in this PR.
Verified clean, so no comments on them: list_versions.py passes ruff check, ruff format --check and mypy under the repo's config (it is in mypy scope — setup.cfg excludes only venv|build|dist|examples); sort -V orders 1.6.0.9 < 1.6.0.10 correctly; the bare [ -n "$IMAGE_TAG" ] && … at L261 does not trip set -e; and DOCSBASE_VERSION_RE correctly filters out the dev/etc/proc/sys scaffolding that docker cp drags in — I confirmed the published layer contains only version dirs plus the manifest.
| concurrency: | ||
| # Every run writes the mutable :latest tag — serialize so the last one to | ||
| # finish is also the last one started, not just the fastest. | ||
| group: connector-docsbase | ||
| cancel-in-progress: false |
There was a problem hiding this comment.
cancel-in-progress: false alone doesn't serialize — it drops runs. (Refining my own round-1 suggestion, which was incomplete.)
GitHub's docs: "only one run can be pending in a concurrency group—any additional pending runs cancel the previous one." So with A in progress and B pending, C arriving cancels B. Because each auto-refresh only generates its own version and pulls the rest forward, B's release is then permanently absent until someone backfills by hand. This repo has shipped three four-part releases in a day, and a generate leg is slow, so the window is wide.
There's now a queue key for exactly this:
| concurrency: | |
| # Every run writes the mutable :latest tag — serialize so the last one to | |
| # finish is also the last one started, not just the fastest. | |
| group: connector-docsbase | |
| cancel-in-progress: false | |
| concurrency: | |
| # Every run writes the mutable :latest tag — serialize so the last one to | |
| # finish is also the last one started, not just the fastest. | |
| # queue: max is required for true serialization — with the default (single) | |
| # only ONE run may be pending, and a third arrival cancels the queued one, | |
| # permanently dropping that release from the corpus. | |
| group: connector-docsbase | |
| cancel-in-progress: false | |
| queue: max |
| # Connector extras build C-extensions (gssapi->krb5, python-ldap->sasl/ldap); | ||
| # the bare runner lacks these headers. Mirrors install_deps.sh used by docs CI. | ||
| - name: Install system build deps for connector extras | ||
| run: | | ||
| sudo apt-get update | ||
| sudo apt-get install -y --no-install-recommends \ | ||
| libkrb5-dev libsasl2-dev libldap2-dev |
There was a problem hiding this comment.
This doesn't actually mirror install_deps.sh.
The script installs python3-ldap, ldap-utils and unixodbc-dev on top of these three. unixodbc-dev is pyodbc's build dependency, reachable through [docs] → base_dev_requirements. I diffed the script at master and at v1.6.0 (the oldest tag in the current corpus) — byte-identical, and it ships in every tag you check out, so calling it stays correct per-version instead of drifting from a hand-copied list. documentation.yml, which the header says this mirrors, just calls it.
| # Connector extras build C-extensions (gssapi->krb5, python-ldap->sasl/ldap); | |
| # the bare runner lacks these headers. Mirrors install_deps.sh used by docs CI. | |
| - name: Install system build deps for connector extras | |
| run: | | |
| sudo apt-get update | |
| sudo apt-get install -y --no-install-recommends \ | |
| libkrb5-dev libsasl2-dev libldap2-dev | |
| # Connector extras build C-extensions (gssapi->krb5, python-ldap->sasl/ldap); | |
| # the bare runner lacks these headers. install_deps.sh ships in every tag we | |
| # check out, so it tracks that version's needs instead of drifting from a | |
| # hand-maintained copy here. | |
| - name: Install system build deps for connector extras | |
| run: ./metadata-ingestion/scripts/install_deps.sh |
| if [ ${#schemas[@]} -eq 0 ]; then | ||
| echo "::error::no config_schemas produced for $V" | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
-eq 0 is far too weak — docGen exits 0 with a partial schema set.
docgen.py wraps per-plugin schema generation in except Exception as e: logger.warning(...) and returns the plugin anyway, so the caller's loaded += 1 branch runs, plugin_metrics.failed is never incremented, and the failed > 0 → sys.exit(1) guard never fires. A connector whose extra failed to install just vanishes with a warning buried in Gradle output.
To calibrate how weak the guard is, I extracted the live image — schemas per version run 100 (1.6.0) → 111 (1.7.0.4), monotonically. A systemic failure yielding 40 sails straight through and overwrites :latest, and the manifest still records that version as fully covered.
| if [ ${#schemas[@]} -eq 0 ]; then | |
| echo "::error::no config_schemas produced for $V" | |
| exit 1 | |
| fi | |
| # docGen exits 0 even when plugins fail: docgen.py swallows per-plugin | |
| # errors into logger.warning and still counts them as loaded, so a | |
| # systemic problem (missing header, dep rot on an old tag) yields a | |
| # handful of schemas rather than none. Live corpus runs 100 -> 111, so | |
| # this floor only trips on a real collapse. | |
| if [ ${#schemas[@]} -lt "${DOCSBASE_MIN_SCHEMAS:-80}" ]; then | |
| echo "::error::only ${#schemas[@]} config_schemas produced for $V (expected >= ${DOCSBASE_MIN_SCHEMAS:-80})" | |
| echo "::error::check the docGen log for 'Failed to load additional metadata for'" | |
| exit 1 | |
| fi |
Comparing against the previous version's count would be even better, since the series is monotonic. Narrowing that except Exception in docgen.py is the real fix, but it's outside this diff.
| cp -R metadata-ingestion/docs/sources/. "out/$V/docs/" | ||
| echo "schemas: $(find "out/$V/schemas" -mindepth 1 -maxdepth 1 | wc -l) | docs dirs: $(find "out/$V/docs" -mindepth 1 -maxdepth 1 | wc -l)" |
There was a problem hiding this comment.
Minor asymmetry: zero schemas is now fatal (good), but zero docs isn't. cp -R src/. succeeds on an empty-but-present source dir, the count is printed and never checked, and if-no-files-found: error is satisfied by the schemas alone — so a docs-less version can enter the corpus.
| cp -R metadata-ingestion/docs/sources/. "out/$V/docs/" | |
| echo "schemas: $(find "out/$V/schemas" -mindepth 1 -maxdepth 1 | wc -l) | docs dirs: $(find "out/$V/docs" -mindepth 1 -maxdepth 1 | wc -l)" | |
| cp -R metadata-ingestion/docs/sources/. "out/$V/docs/" | |
| docs_n="$(find "out/$V/docs" -mindepth 1 -maxdepth 1 | wc -l)" | |
| [ "$docs_n" -gt 0 ] || { echo "::error::no docs/sources captured for $V"; exit 1; } | |
| echo "schemas: ${#schemas[@]} | docs dirs: $docs_n" |
| - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 | ||
| with: | ||
| pattern: docsbase-* | ||
| path: corpus | ||
| merge-multiple: true |
There was a problem hiding this comment.
This answers your round-1 question about v8 behaviour — and the answer is "not the same".
download-artifact has asymmetric failure semantics across its input modes. I checked the pinned source (3e5f45b, v8.0.1):
name:→throw new Error("Artifact '<name>' not found")artifact-ids:→throw new Error("None of the provided artifact IDs were found")pattern:→ no throw. It filters the run's artifact list, logsFiltered from X to Y artifacts, and the download block is guarded byif (artifacts.length). It then reportsTotal of 0 artifact(s) downloadedand exits 0.
This PR uses pattern:. Combined with the || true on L181, zero artifacts becomes a green run that pulls :latest forward, prunes, rebuilds and pushes — byte-identical content, fresh generated_at, and a step summary announcing a successful publish for a release whose docs were never added. It's the most convincing green-but-wrong path in the workflow, because it emits positive evidence of success.
No change needed here; the fix is on L181.
| cat > corpus/CORPUS_MANIFEST.json <<EOF | ||
| {"floor":"$floor","ceiling":"$ceiling","versions_count":$versions_count,"versions":$versions_json,"source_sha":"$GITHUB_SHA","generated_at":"$generated_at"} | ||
| EOF | ||
| # Tags: range = true identity (primary), latest = moving pointer. |
There was a problem hiding this comment.
Small accuracy point: the range tag isn't an identity. Nothing makes <floor>-<ceiling> unique — a re-run, or a backfill of a middle version, reproduces the same floor/ceiling with different content and silently overwrites it. Live tags already show the scheme in use (1.5.0.7-1.6.0.13, 1.6.0-1.7.0.4), and :1.6.0-1.7.0.4 currently resolves to the same digest as :latest.
The "pin the digest" guidance in the step summary is the right mitigation, so this is just the comment overselling it:
| # Tags: range = true identity (primary), latest = moving pointer. | |
| # Tags: range is the primary pointer, latest is the moving one. Note the | |
| # range tag is NOT content-addressed — a re-run, or a backfill of a middle | |
| # version, reproduces the same floor-ceiling with different content and | |
| # overwrites it. The digest emitted after push is the only real identity. |
Related: nothing verifies the set is gap-free, so a corpus of {1.6.0.9, 1.7.0.3} still publishes as :1.6.0.9-1.7.0.3, which reads as "every release in that interval". CORPUS_MANIFEST.json carries the honest list, but the OCI labels expose only floor/ceiling/count. A contiguous=true|false label would let a consumer tell a range from a range-with-holes.
| - name: Build image | ||
| run: | | ||
| set -euo pipefail | ||
| printf 'FROM scratch\nCOPY corpus/ /\n' > Dockerfile.docsbase | ||
| tag_args=() | ||
| while IFS= read -r t; do [ -n "$t" ] && tag_args+=(-t "$t"); done <<< "${{ steps.coverage.outputs.tags }}" | ||
| docker build -f Dockerfile.docsbase \ | ||
| --label "org.acryl.docsbase.floor=${{ steps.coverage.outputs.floor }}" \ | ||
| --label "org.acryl.docsbase.ceiling=${{ steps.coverage.outputs.ceiling }}" \ | ||
| --label "org.acryl.docsbase.versions_count=${{ steps.coverage.outputs.versions_count }}" \ | ||
| --label "org.acryl.docsbase.source_sha=$GITHUB_SHA" \ | ||
| --label "org.acryl.docsbase.generated_at=${{ steps.coverage.outputs.generated_at }}" \ | ||
| "${tag_args[@]}" . |
There was a problem hiding this comment.
Two things in this step.
1. inputs.image_tag reaches shell source via ${{ }}. It's unvalidated free text, flows into steps.coverage.outputs.tags (L261-262), and is expanded inside the herestring below. A value containing " terminates it and the rest parses as shell — in a job holding the Docker Hub push token. You already fixed exactly this class in round 1 for days/versions/image_tag; these four sites (L279, plus L310/L313/L328 in Push/Dry-run notice) were missed because the value now arrives via a step output rather than an input.
2. The build context is the whole monorepo. docker build … . tars the entire checkout — plus previous/, a full duplicate of corpus/ — to produce a FROM scratch image that only needs corpus/. It also silently inherits the repo-root .dockerignore, which is maintained for the product images: none of its current patterns match under corpus/, but a future **/docs/-style entry would gut this image with a clean, green build. Worth moving Dockerfile.docsbase into a dedicated dir next to corpus/ and building that as the context.
The env: fix for (1):
| - name: Build image | |
| run: | | |
| set -euo pipefail | |
| printf 'FROM scratch\nCOPY corpus/ /\n' > Dockerfile.docsbase | |
| tag_args=() | |
| while IFS= read -r t; do [ -n "$t" ] && tag_args+=(-t "$t"); done <<< "${{ steps.coverage.outputs.tags }}" | |
| docker build -f Dockerfile.docsbase \ | |
| --label "org.acryl.docsbase.floor=${{ steps.coverage.outputs.floor }}" \ | |
| --label "org.acryl.docsbase.ceiling=${{ steps.coverage.outputs.ceiling }}" \ | |
| --label "org.acryl.docsbase.versions_count=${{ steps.coverage.outputs.versions_count }}" \ | |
| --label "org.acryl.docsbase.source_sha=$GITHUB_SHA" \ | |
| --label "org.acryl.docsbase.generated_at=${{ steps.coverage.outputs.generated_at }}" \ | |
| "${tag_args[@]}" . | |
| - name: Build image | |
| env: | |
| TAGS: ${{ steps.coverage.outputs.tags }} | |
| FLOOR: ${{ steps.coverage.outputs.floor }} | |
| CEILING: ${{ steps.coverage.outputs.ceiling }} | |
| VERSIONS_COUNT: ${{ steps.coverage.outputs.versions_count }} | |
| GENERATED_AT: ${{ steps.coverage.outputs.generated_at }} | |
| run: | | |
| set -euo pipefail | |
| printf 'FROM scratch\nCOPY corpus/ /\n' > Dockerfile.docsbase | |
| tag_args=() | |
| while IFS= read -r t; do [ -n "$t" ] && tag_args+=(-t "$t"); done <<< "$TAGS" | |
| docker build -f Dockerfile.docsbase \ | |
| --label "org.acryl.docsbase.floor=$FLOOR" \ | |
| --label "org.acryl.docsbase.ceiling=$CEILING" \ | |
| --label "org.acryl.docsbase.versions_count=$VERSIONS_COUNT" \ | |
| --label "org.acryl.docsbase.source_sha=$GITHUB_SHA" \ | |
| --label "org.acryl.docsbase.generated_at=$GENERATED_AT" \ | |
| "${tag_args[@]}" . |
| - name: Check whether docker login is possible | ||
| id: docker-login | ||
| env: | ||
| ENABLE_DOCKER_LOGIN: ${{ secrets.DOCSBASE_DH_TOKEN != '' }} | ||
| run: echo "docker-login=$ENABLE_DOCKER_LOGIN" >> "$GITHUB_OUTPUT" |
There was a problem hiding this comment.
Blocker: with dry_run: false and no secret, this job goes green having published nothing.
The three terminal steps are mutually exclusive and non-exhaustive. When docker-login is 'false': login skips, push skips, and Dry-run notice also skips because dry_run is false. $GITHUB_STEP_SUMMARY is only written inside Push, so it stays empty. The last executed step is Build image, which succeeds — job conclusion: success.
The operator sees a green Refresh connector docsbase image on a workflow whose headline job already passed, and has to notice two grey "skipped" icons to spot it. This can persist across every release indefinitely, with the image simply never moving. Your test plan still lists configuring DOCSBASE_DH_TOKEN as unchecked, so this is the first-release path.
The round-1 guard I suggested was right as a login guard; it just needs to not be silent when publishing was actually requested:
| - name: Check whether docker login is possible | |
| id: docker-login | |
| env: | |
| ENABLE_DOCKER_LOGIN: ${{ secrets.DOCSBASE_DH_TOKEN != '' }} | |
| run: echo "docker-login=$ENABLE_DOCKER_LOGIN" >> "$GITHUB_OUTPUT" | |
| - name: Check whether docker login is possible | |
| id: docker-login | |
| env: | |
| ENABLE_DOCKER_LOGIN: ${{ secrets.DOCSBASE_DH_TOKEN != '' }} | |
| DRY_RUN: ${{ inputs.dry_run }} | |
| run: | | |
| set -euo pipefail | |
| if [ "$DRY_RUN" != "true" ] && [ "$ENABLE_DOCKER_LOGIN" != "true" ]; then | |
| echo "::error::dry_run=false but DOCSBASE_DH_TOKEN is unset/empty — refusing to" | |
| echo "::error::build and discard silently. Configure the secret, or pass dry_run=true." | |
| exit 1 | |
| fi | |
| echo "docker-login=$ENABLE_DOCKER_LOGIN" >> "$GITHUB_OUTPUT" |
Worth flipping the workflow_call secret declaration (L61-63) to required: true at the same time — required: false tells GitHub explicitly not to complain.
Separately: workflow_call's dry_run defaults to true. The current caller passes false correctly, but for a reusable workflow whose entire purpose is publishing, a future caller that omits it gets a silent build-and-discard. The workflow_dispatch default of true is right and should stay.
| echo "- range: \`${{ steps.coverage.outputs.range }}\`" | ||
| echo "- digest: \`$digest\`" | ||
| echo "" | ||
| echo "Pin the digest in the consumer (\`release-lock.yml\` → \`DOCSBASE_DIGEST\`)." |
There was a problem hiding this comment.
release-lock.yml and DOCSBASE_DIGEST don't exist anywhere in this repo — I grepped the whole tree at this ref. If the consumer lives in another repo, worth naming it here so the operator reading this summary knows where to go.
| with urllib.request.urlopen(_PYPI, timeout=30) as r: | ||
| data = json.load(r) | ||
|
|
||
| cutoff = dt.datetime.now(dt.timezone.utc) - dt.timedelta(days=args.days) | ||
| picked: list[tuple[dt.datetime, str]] = [] | ||
| for version, files in data.get("releases", {}).items(): | ||
| if not _FINAL_RELEASE.match(version) or not files: | ||
| continue |
There was a problem hiding this comment.
Two robustness gaps, both of which end in the corpus being pruned to nothing (see my comment on connector-docsbase.yml L209).
1. [] is a valid success. .get("releases", {}) turns a PyPI shape change into an empty result, printed on stdout with exit 0. The caller unions that into its retention set and deletes everything outside it. PyPI has repeatedly signalled intent to drop the releases key from /pypi/<project>/json, so this isn't hypothetical.
2. Yanked releases are included. f.get("yanked") is never checked. There are currently 13 yanked acryl-datahub releases; all the non-rc ones are old enough to fall outside any window, so it's latent — but a yanked final release inside the window would be built into the corpus and counted in the retention set.
| with urllib.request.urlopen(_PYPI, timeout=30) as r: | |
| data = json.load(r) | |
| cutoff = dt.datetime.now(dt.timezone.utc) - dt.timedelta(days=args.days) | |
| picked: list[tuple[dt.datetime, str]] = [] | |
| for version, files in data.get("releases", {}).items(): | |
| if not _FINAL_RELEASE.match(version) or not files: | |
| continue | |
| with urllib.request.urlopen(_PYPI, timeout=30) as r: | |
| data = json.load(r) | |
| releases = data.get("releases") | |
| if not isinstance(releases, dict) or not releases: | |
| # Callers union this list into their retention set and prune everything | |
| # outside it, so returning [] silently wipes the published corpus. A | |
| # shape change in PyPI's JSON must fail loudly rather than look empty. | |
| sys.exit("PyPI JSON for acryl-datahub has no usable 'releases' map") | |
| cutoff = dt.datetime.now(dt.timezone.utc) - dt.timedelta(days=args.days) | |
| picked: list[tuple[dt.datetime, str]] = [] | |
| for version, files in releases.items(): | |
| if not _FINAL_RELEASE.match(version) or not files: | |
| continue | |
| if any(f.get("yanked") for f in files): | |
| continue |
Also worth considering, lower priority:
- Exit non-zero when
versionsends up empty (behind an--allow-emptyescape hatch), so the caller's matrix can't silently become[]. - A retry around
urlopen. This runs twice per workflow — the second call is inassemble-and-publish, after every generate leg has finished, so a transient PyPI blip there discards tens of job-minutes and ends the run without refreshing:latest. - There's a
metadata-ingestion/scripts/tests/directory (test_docgen.py,test_utils.py) and atestScriptsGradle task. The regex and window arithmetic here are load-bearing for the prune step, so a ~30-line test over a canned PyPI payload would be worth it — that's a genuine regression risk, not coverage-chasing.
2 participants
Summary
connector-docsbase.yml: for each acryl-datahub four-part release (X.Y.Z.W), checks out the tag, runsdocGen(config schemas) and capturesmetadata-ingestion/docs/sources, then assembles a single versioned corpus image (/opt/<version>/{docs,schemas}layout) published asacryldata/datahub-ingestion-docsbase, tagged by the<floor>-<ceiling>version range it covers plus a moving:latest.CORPUS_MANIFEST.jsonand as OCI labels (org.acryl.docsbase.floor/ceiling/versions_count/source_sha/generated_at) so a consumer can inspect what a given digest covers before pinning it.metadata-ingestion/scripts/docsbase/list_versions.py, which queries PyPI for final four-partacryl-datahubreleases within a lookback window (used to build the version matrix forworkflow_dispatchbackfills).connector-docsbase.ymlup as aworkflow_calltarget frompublish-pypi-release.yml: once a release'spush_to_pypijob succeeds, and only for a genuine finalX.Y.Z.Wtag (rc/dev-suffixed tags are skipped), it re-runs the docsbase build for just that version and publishes it (non-dry-run) — so the image keeps itself current without anyone needing to remember to trigger it by hand.workflow_dispatchremains available for manual/backfill runs (explicitversionslist or adayslookback), with adry_runinput that builds without pushing.Test plan
workflow_dispatchdry run on this branch (single version, no push) completed successfully end-to-end: version selection → docGen → corpus assembly → image build. https://github.com/acryldata/datahub/actions/runs/30608823305DOCSBASE_DH_TOKENis configured as a repo secret before merge (needed for the real, non-dry-run push path).refresh_docsbaseinpublish-pypi-release.ymland successfully pushes an updated image.