From 769e2f106a3679cdff6a7d25b0bbca2b315e5195 Mon Sep 17 00:00:00 2001 From: comphead Date: Wed, 7 Oct 2026 09:21:38 -0700 Subject: [PATCH 1/7] docs: Update release README examples to 55.2.0 and add SVN download steps - Use 55.2.0 (and branch-55, 55.1.0) in all examples. - Replace the manual version edit and `cargo check -p datafusion` with sed commands and `cargo update --workspace`, plus a check that no reference to the old version is left. - Add `svn ls` checks and commands to `svn export`, verify and unpack the RC and release tarballs. --- dev/release/README.md | 146 ++++++++++++++++++++++++++++++------------ 1 file changed, 105 insertions(+), 41 deletions(-) diff --git a/dev/release/README.md b/dev/release/README.md index d49c1c9125235..a3c1a6c6d5043 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -69,6 +69,18 @@ Follow the instructions in the header of the KEYS file to append your key. Here svn commit KEYS -m "Add key for John Doe" ``` +### Check SVN access + +Check that you can read the `dev` and `release` areas of the Apache distribution SVN server: + +```shell +svn ls https://dist.apache.org/repos/dist/dev/datafusion +svn ls https://dist.apache.org/repos/dist/release/datafusion +``` + +The commands list the existing release candidates and releases. The scripts used below commit +to these areas and will ask for your ASF credentials. + ## Release Process: Step by Step As part of the Apache governance model, official releases consist of signed @@ -79,34 +91,47 @@ We then publish the code in the approved artifacts to crates.io. First create a new release branch from `main` in the `apache` repository. -For example, to create the `branch-50` branch for the `50.x.y` release series: +For example, to create the `branch-55` branch for the `55.x.y` release series: ```shell git fetch apache # make sure we are up to date git checkout apache/main # checkout current latest development branch -git checkout -b branch-50 # create local branch -git push -u apache branch-50 # push branch to apache remote +git checkout -b branch-55 # create local branch +git push -u apache branch-55 # push branch to apache remote ``` ### 2. Prepare PR to Update the Release Version -Manually update the DataFusion version in the root `Cargo.toml` to -reflect the new release version. Ensure `Cargo.lock` is updated accordingly by -running: - -```shell -cargo check -p datafusion -``` - -Within the user documentation there are references to the current version number. -Update these to the current version. At the time of this writing we need to manually -update the following files +Update the DataFusion version in the root `Cargo.toml` (the workspace version and the +versions of the internal crates) and in `Cargo.lock` to reflect the new release version. +Within the user documentation there are also references to the current version number. +At the time of this writing we need to update the following files: - `docs/source/download.md` - `docs/source/user-guide/configs.md` - `docs/source/user-guide/crate-configuration.md` - `docs/source/user-guide/example-usage.md` +For example, to update the version from `55.1.0` to `55.2.0`: + +```shell +# version and lockfile +sed -i '' 's/"55\.1\.0"/"55.2.0"/g' Cargo.toml +sed -i '' 's/55\.1\.0/55.2.0/' docs/source/download.md docs/source/user-guide/{configs,crate-configuration,example-usage}.md +cargo update --workspace +``` + +`sed -i ''` is the macOS (BSD) syntax, use `sed -i` with GNU `sed`. `cargo update --workspace` +only refreshes the workspace crates in `Cargo.lock`, so it does not compile anything. + +Review the diff of `Cargo.toml`: only the workspace `version` and the internal `datafusion*` +entries should change. Then check that no reference to the old version is left. The following +command should print nothing: + +```shell +git grep -nF '55.1.0' -- . ':!dev/changelog' +``` + Then commit the changes and create a PR targeting the release branch `branch-N`. ```shell @@ -118,20 +143,20 @@ git commit -a -m 'Update version' To protect a release candidate branch from accidental merges, create PR against `main`: ```shell -git fetch apache && git checkout -b protect_branch_50 -./dev/release/add-branch-protection.sh 50 +git fetch apache && git checkout -b protect_branch_55 +./dev/release/add-branch-protection.sh 55 ``` The script will modify `.asf.yaml` and add following block: ```yaml -branch-50: +branch-55: required_pull_request_reviews: required_approving_review_count: 1 ``` - Commit changes -- Push to `origin/protect_branch_50` +- Push to `origin/protect_branch_55` - Create a PR against `main`. - Merge to `main`. - Notify community in Discord/Slack that release branch is created @@ -146,7 +171,7 @@ Backports are important and sometimes unexpected, so please proceed to next rele ### 5. Prepare PR to Update Changelog Update the changelog in `dev/changelog/`. Each release has its -own file, such as `dev/changelog/50.0.0.md`, which should include all changes +own file, such as `dev/changelog/55.2.0.md`, which should include all changes since the previous release. The changelog is generated using a Python script, which requires a GitHub @@ -160,7 +185,7 @@ uv sync To generate the changelog, set the `GITHUB_TOKEN` environment variable and run `./dev/release/generate-changelog.py` with two commit IDs or tags followed by the release version. For example, to generate a changelog of all changes -between the `50.3.0` tag and `branch-51` for release `51.0.0`: +between the `55.1.0` tag and `branch-55` for release `55.2.0`: > [!NOTE] > @@ -173,7 +198,7 @@ between the `50.3.0` tag and `branch-51` for release `51.0.0`: ```shell export GITHUB_TOKEN= -uv run ./dev/release/generate-changelog.py 50.3.0 branch-51 51.0.0 > dev/changelog/51.0.0.md +uv run ./dev/release/generate-changelog.py 55.1.0 branch-55 55.2.0 > dev/changelog/55.2.0.md ``` This script creates a changelog from GitHub PRs based on the labels associated with them as well as looking for @@ -182,7 +207,7 @@ titles starting with `feat:`, `fix:`, or `docs:`. Once the changelog is generated, run `prettier` to format the document: ```shell -prettier -w dev/changelog/51.0.0.md +prettier -w dev/changelog/55.2.0.md ``` Then commit the changes and create a PR targeting the release branch. @@ -209,17 +234,17 @@ Pick numbers in sequential order, with `1` for `rc1`, `2` for `rc2`, etc. While the official release artifacts are signed tarballs and zip files, we also tag the commit it was created from for convenience and code archaeology. Release tags -look like `50.3.0`, and release candidate tags look like `50.3.0-rc1`. See [the list of existing +look like `55.2.0`, and release candidate tags look like `55.2.0-rc1`. See [the list of existing tags]. [the list of existing tags]: https://github.com/apache/datafusion/tags -Create and push the RC tag, for example, to create the `50.3.0-rc1` tag from `branch-50`, use: +Create and push the RC tag, for example, to create the `55.2.0-rc1` tag from `branch-55`, use: ```shell git fetch apache -git tag 50.3.0-rc1 apache/branch-50 -git push apache 50.3.0-rc1 +git tag 55.2.0-rc1 apache/branch-55 +git push apache 55.2.0-rc1 ``` Please make sure the format is correct, tools like Homebrew listens for tags and in case of malformed tags users would be notified for non-existent version @@ -228,10 +253,10 @@ Please make sure the format is correct, tools like Homebrew listens for tags and Run the `create-tarball.sh` script with the `` tag and `` number you determined in previous steps: -For example, to create the `50.3.0-rc1` artifacts: +For example, to create the `55.2.0-rc1` artifacts: ```shell -GITHUB_TOKEN= ./dev/release/create-tarball.sh 50.3.0 1 +GITHUB_TOKEN= ./dev/release/create-tarball.sh 55.2.0 1 ``` The `create-tarball.sh` script @@ -243,6 +268,14 @@ The `create-tarball.sh` script 2. Provides you an email template to send to `dev@datafusion.apache.org` for release voting. +Check that the artifacts were uploaded to SVN: + +```shell +svn ls https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-55.2.0-rc1/ +``` + +It should list `apache-datafusion-55.2.0.tar.gz` together with its `.asc`, `.sha256` and `.sha512` files. + ### 7. Vote on Release Candidate Artifacts Send the email output from the script to `dev@datafusion.apache.org`. @@ -257,7 +290,21 @@ review the release candidate. `dev/release/verify-release-candidate.sh` is a script in this repository that can assist in the verification process. Run it like this: ```shell -./dev/release/verify-release-candidate.sh 50.3.0 1 +./dev/release/verify-release-candidate.sh 55.2.0 1 +``` + +To download and inspect the artifacts of a release candidate yourself, export them from SVN, +check the signature and the checksums, and unpack the tarball: + +```shell +svn export https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-55.2.0-rc1 +cd apache-datafusion-55.2.0-rc1 +svn export https://dist.apache.org/repos/dist/dev/datafusion/KEYS +gpg --import KEYS +gpg --verify apache-datafusion-55.2.0.tar.gz.asc apache-datafusion-55.2.0.tar.gz +shasum -a 256 -c apache-datafusion-55.2.0.tar.gz.sha256 +shasum -a 512 -c apache-datafusion-55.2.0.tar.gz.sha512 +tar xzf apache-datafusion-55.2.0.tar.gz ``` #### If Changes Are Requested @@ -282,11 +329,17 @@ with the release verification. NOTE: steps in this section can only be done by PMC members after release is approved. Move artifacts to the release location in SVN, e.g. -https://dist.apache.org/repos/dist/release/datafusion/datafusion-50.3.0/, using +https://dist.apache.org/repos/dist/release/datafusion/datafusion-55.2.0/, using the `release-tarball.sh` script: ```shell -./dev/release/release-tarball.sh 50.3.0 1 +./dev/release/release-tarball.sh 55.2.0 1 +``` + +Check that the artifacts are in the release location: + +```shell +svn ls https://dist.apache.org/repos/dist/release/datafusion/datafusion-55.2.0/ ``` Congratulations! The release is now official! @@ -296,9 +349,9 @@ Congratulations! The release is now official! Tag the same release candidate commit with the final release tag ```shell -git checkout 50.3.0-rc1 -git tag 50.3.0 -git push apache 50.3.0 +git checkout 55.2.0-rc1 +git tag 55.2.0 +git push apache 55.2.0 ``` ### 10. Publish on Crates.io @@ -315,10 +368,21 @@ instructions](https://doc.rust-lang.org/cargo/reference/publishing.html) to create an account and login to crates.io before asking to be added as an owner to all DataFusion crates. -Download and unpack the official release tarball +Download the official release tarball from SVN, check its signature and checksum, and unpack it: + +```shell +svn export https://dist.apache.org/repos/dist/release/datafusion/datafusion-55.2.0 +cd datafusion-55.2.0 +svn export https://dist.apache.org/repos/dist/release/datafusion/KEYS +gpg --import KEYS +gpg --verify apache-datafusion-55.2.0.tar.gz.asc apache-datafusion-55.2.0.tar.gz +shasum -a 512 -c apache-datafusion-55.2.0.tar.gz.sha512 +tar xzf apache-datafusion-55.2.0.tar.gz +cd apache-datafusion-55.2.0 +``` Verify that the Cargo.toml in the tarball contains the correct version -(e.g. `version = "50.3.0"`) and then publish the crates by running the following commands +(e.g. `version = "55.2.0"`) and then publish the crates by running the following commands ```shell (cd datafusion/common && cargo publish) @@ -370,10 +434,10 @@ If it happens crates.io fails with wrong dependency message like below, just rer error: failed to prepare local package for uploading Caused by: - failed to select a version for the requirement `datafusion-proto = "^53.1.0"` - candidate versions found which didn't match: 53.0.0, 52.5.0, 52.4.0, ... + failed to select a version for the requirement `datafusion-proto = "^55.2.0"` + candidate versions found which didn't match: 55.1.0, 55.0.0, 54.1.0, ... location searched: crates.io index - required by package `datafusion-ffi v53.1.0 (/private/tmp/apache-datafusion-53.1.0/datafusion/ffi)` + required by package `datafusion-ffi v55.2.0 (/private/tmp/apache-datafusion-55.2.0/datafusion/ffi)` ``` ### Publish datafusion-cli on Homebrew @@ -423,7 +487,7 @@ svn ls https://dist.apache.org/repos/dist/dev/datafusion To delete a release candidate: ```shell -svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-50.0.0-rc1/ +svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-55.2.0-rc1/ ``` #### Delete old releases from `release` SVN @@ -440,5 +504,5 @@ svn ls https://dist.apache.org/repos/dist/release/datafusion To delete a release: ```shell -svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-50.0.0 +svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-55.1.0 ``` From fa43c44663aba9f98cd6aa73e74aa48ccd4c34a0 Mon Sep 17 00:00:00 2001 From: comphead Date: Wed, 7 Oct 2026 09:26:53 -0700 Subject: [PATCH 2/7] docs: Remove gpg, checksum and extra notes from release README additions --- dev/release/README.md | 45 ++++--------------------------------------- 1 file changed, 4 insertions(+), 41 deletions(-) diff --git a/dev/release/README.md b/dev/release/README.md index a3c1a6c6d5043..c752cd24495e3 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -69,18 +69,6 @@ Follow the instructions in the header of the KEYS file to append your key. Here svn commit KEYS -m "Add key for John Doe" ``` -### Check SVN access - -Check that you can read the `dev` and `release` areas of the Apache distribution SVN server: - -```shell -svn ls https://dist.apache.org/repos/dist/dev/datafusion -svn ls https://dist.apache.org/repos/dist/release/datafusion -``` - -The commands list the existing release candidates and releases. The scripts used below commit -to these areas and will ask for your ASF credentials. - ## Release Process: Step by Step As part of the Apache governance model, official releases consist of signed @@ -102,10 +90,8 @@ git push -u apache branch-55 # push branch to apache remote ### 2. Prepare PR to Update the Release Version -Update the DataFusion version in the root `Cargo.toml` (the workspace version and the -versions of the internal crates) and in `Cargo.lock` to reflect the new release version. -Within the user documentation there are also references to the current version number. -At the time of this writing we need to update the following files: +Update the DataFusion version in the root `Cargo.toml`, in `Cargo.lock` and in the user +documentation. At the time of this writing the documentation files are - `docs/source/download.md` - `docs/source/user-guide/configs.md` @@ -121,17 +107,6 @@ sed -i '' 's/55\.1\.0/55.2.0/' docs/source/download.md docs/source/user-guide/{c cargo update --workspace ``` -`sed -i ''` is the macOS (BSD) syntax, use `sed -i` with GNU `sed`. `cargo update --workspace` -only refreshes the workspace crates in `Cargo.lock`, so it does not compile anything. - -Review the diff of `Cargo.toml`: only the workspace `version` and the internal `datafusion*` -entries should change. Then check that no reference to the old version is left. The following -command should print nothing: - -```shell -git grep -nF '55.1.0' -- . ':!dev/changelog' -``` - Then commit the changes and create a PR targeting the release branch `branch-N`. ```shell @@ -274,8 +249,6 @@ Check that the artifacts were uploaded to SVN: svn ls https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-55.2.0-rc1/ ``` -It should list `apache-datafusion-55.2.0.tar.gz` together with its `.asc`, `.sha256` and `.sha512` files. - ### 7. Vote on Release Candidate Artifacts Send the email output from the script to `dev@datafusion.apache.org`. @@ -293,17 +266,11 @@ review the release candidate. ./dev/release/verify-release-candidate.sh 55.2.0 1 ``` -To download and inspect the artifacts of a release candidate yourself, export them from SVN, -check the signature and the checksums, and unpack the tarball: +To download and unpack the artifacts of a release candidate from SVN: ```shell svn export https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-55.2.0-rc1 cd apache-datafusion-55.2.0-rc1 -svn export https://dist.apache.org/repos/dist/dev/datafusion/KEYS -gpg --import KEYS -gpg --verify apache-datafusion-55.2.0.tar.gz.asc apache-datafusion-55.2.0.tar.gz -shasum -a 256 -c apache-datafusion-55.2.0.tar.gz.sha256 -shasum -a 512 -c apache-datafusion-55.2.0.tar.gz.sha512 tar xzf apache-datafusion-55.2.0.tar.gz ``` @@ -368,15 +335,11 @@ instructions](https://doc.rust-lang.org/cargo/reference/publishing.html) to create an account and login to crates.io before asking to be added as an owner to all DataFusion crates. -Download the official release tarball from SVN, check its signature and checksum, and unpack it: +Download and unpack the official release tarball: ```shell svn export https://dist.apache.org/repos/dist/release/datafusion/datafusion-55.2.0 cd datafusion-55.2.0 -svn export https://dist.apache.org/repos/dist/release/datafusion/KEYS -gpg --import KEYS -gpg --verify apache-datafusion-55.2.0.tar.gz.asc apache-datafusion-55.2.0.tar.gz -shasum -a 512 -c apache-datafusion-55.2.0.tar.gz.sha512 tar xzf apache-datafusion-55.2.0.tar.gz cd apache-datafusion-55.2.0 ``` From 32b59e1398c622b952cced9f702af3c0dbd6957f Mon Sep 17 00:00:00 2001 From: comphead Date: Wed, 7 Oct 2026 09:43:16 -0700 Subject: [PATCH 3/7] docs: Remove RC download from release README --- dev/release/README.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/dev/release/README.md b/dev/release/README.md index c752cd24495e3..f3ecd614a7e17 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -266,14 +266,6 @@ review the release candidate. ./dev/release/verify-release-candidate.sh 55.2.0 1 ``` -To download and unpack the artifacts of a release candidate from SVN: - -```shell -svn export https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-55.2.0-rc1 -cd apache-datafusion-55.2.0-rc1 -tar xzf apache-datafusion-55.2.0.tar.gz -``` - #### If Changes Are Requested If the release is not approved or urgent backports requested, please start over from [here](#4-backporting-urgent-changes) From 7bb0181467258dfc69df8cd33825035601619b46 Mon Sep 17 00:00:00 2001 From: comphead Date: Wed, 7 Oct 2026 09:48:14 -0700 Subject: [PATCH 4/7] docs: Keep the last major release in the release SVN --- dev/release/README.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/dev/release/README.md b/dev/release/README.md index f3ecd614a7e17..987965f4e7f2d 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -447,8 +447,9 @@ svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/ #### Delete old releases from `release` SVN -Only the latest release should be available. Delete old releases after -publishing the new release. +Keep the latest release and the last major release (`X.0.0`). Delete all other old releases +after publishing the new release. For example, after publishing 55.2.0 keep `datafusion-55.2.0` +and `datafusion-55.0.0`, and delete everything else, such as 55.1.0 and every 54.x release. To get a list of DataFusion releases: @@ -456,8 +457,11 @@ To get a list of DataFusion releases: svn ls https://dist.apache.org/repos/dist/release/datafusion ``` -To delete a release: +To delete releases: ```shell -svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-55.1.0 +svn delete -m "delete old DataFusion release" \ + https://dist.apache.org/repos/dist/release/datafusion/datafusion-54.0.0 \ + https://dist.apache.org/repos/dist/release/datafusion/datafusion-54.1.0 \ + https://dist.apache.org/repos/dist/release/datafusion/datafusion-55.1.0 ``` From d32ba8975df47d2fb11fb95adedc60fdddc3e1d3 Mon Sep 17 00:00:00 2001 From: comphead Date: Wed, 7 Oct 2026 09:53:13 -0700 Subject: [PATCH 5/7] docs: Keep all releases of the last major version in the release SVN --- dev/release/README.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/dev/release/README.md b/dev/release/README.md index 987965f4e7f2d..74b3dbfcab231 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -447,9 +447,9 @@ svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/ #### Delete old releases from `release` SVN -Keep the latest release and the last major release (`X.0.0`). Delete all other old releases -after publishing the new release. For example, after publishing 55.2.0 keep `datafusion-55.2.0` -and `datafusion-55.0.0`, and delete everything else, such as 55.1.0 and every 54.x release. +Keep the last major release (`X.0.0`) and the releases after it. Delete older releases +after publishing the new release. For example, after publishing 55.2.0 keep `datafusion-55.0.0`, +`datafusion-55.1.0` and `datafusion-55.2.0`, and delete every 54.x release. To get a list of DataFusion releases: @@ -462,6 +462,5 @@ To delete releases: ```shell svn delete -m "delete old DataFusion release" \ https://dist.apache.org/repos/dist/release/datafusion/datafusion-54.0.0 \ - https://dist.apache.org/repos/dist/release/datafusion/datafusion-54.1.0 \ - https://dist.apache.org/repos/dist/release/datafusion/datafusion-55.1.0 + https://dist.apache.org/repos/dist/release/datafusion/datafusion-54.1.0 ``` From 6fa0157b3157841263dd2759b88a2dd8682790ca Mon Sep 17 00:00:00 2001 From: comphead Date: Wed, 7 Oct 2026 15:35:28 -0700 Subject: [PATCH 6/7] docs: Add script that prints svn deletes for old RCs and releases --- dev/release/README.md | 36 ++-------- dev/release/print-svn-deletes.sh | 116 +++++++++++++++++++++++++++++++ 2 files changed, 123 insertions(+), 29 deletions(-) create mode 100755 dev/release/print-svn-deletes.sh diff --git a/dev/release/README.md b/dev/release/README.md index 74b3dbfcab231..6d54d79553518 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -431,36 +431,14 @@ The release information is used to generate a template for a board report (see e See the ASF documentation on [when to archive](https://www.apache.org/legal/release-policy.html#when-to-archive) for more information. -Release candidates should be deleted once the release is published. +Release candidates should be deleted once the release is published. In the `release` SVN keep the +last major release (`X.0.0`) and the releases after it, and delete older releases. For example, after +publishing 55.2.0 keep `datafusion-55.0.0`, `datafusion-55.1.0` and `datafusion-55.2.0`, and delete +every 54.x release. -To get a list of DataFusion release candidates: +The `print-svn-deletes.sh` script prints the `svn delete` commands for a released version. It does not +run them, so check the printed commands and then run them: ```shell -svn ls https://dist.apache.org/repos/dist/dev/datafusion -``` - -To delete a release candidate: - -```shell -svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-55.2.0-rc1/ -``` - -#### Delete old releases from `release` SVN - -Keep the last major release (`X.0.0`) and the releases after it. Delete older releases -after publishing the new release. For example, after publishing 55.2.0 keep `datafusion-55.0.0`, -`datafusion-55.1.0` and `datafusion-55.2.0`, and delete every 54.x release. - -To get a list of DataFusion releases: - -```shell -svn ls https://dist.apache.org/repos/dist/release/datafusion -``` - -To delete releases: - -```shell -svn delete -m "delete old DataFusion release" \ - https://dist.apache.org/repos/dist/release/datafusion/datafusion-54.0.0 \ - https://dist.apache.org/repos/dist/release/datafusion/datafusion-54.1.0 +./dev/release/print-svn-deletes.sh 55.2.0 ``` diff --git a/dev/release/print-svn-deletes.sh b/dev/release/print-svn-deletes.sh new file mode 100755 index 0000000000000..053090880cdcb --- /dev/null +++ b/dev/release/print-svn-deletes.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Prints the `svn delete` commands that clean up the Apache distribution SVN +# server after a release has been published. It only lists the SVN directories +# and never deletes anything. Check the printed commands and then run them. +# +# For the released version it prints deletes for +# - the release candidates of that version and older versions in the `dev` area +# - the releases older than the last major release (.0.0) in the +# `release` area. The last major release and the releases after it are kept. +# +# Usage: ./dev/release/print-svn-deletes.sh +# Example: ./dev/release/print-svn-deletes.sh 55.2.0 +# +# DIST_URL can point to another SVN repository, for example to test the script. + +set -euo pipefail + +usage() { + echo "Usage: $0 , for example: $0 55.2.0" >&2 + exit 1 +} + +# Turns X.Y.Z into a number that can be compared, for example 55.2.0 -> 55002000 +version_number() { + local maj min pat + IFS=. read -r maj min pat <<<"$1" + echo $((10#$maj * 1000000 + 10#$min * 1000 + 10#$pat)) +} + +# Prints an svn delete command with one URL per line +print_delete() { + local message=$1 url + shift + printf 'svn delete -m "%s"' "$message" + for url in "$@"; do + printf ' \\\n %s' "$url" + done + printf '\n' +} + +version_re='^([0-9]+)\.[0-9]+\.[0-9]+$' +rc_re='^apache-datafusion-([0-9]+\.[0-9]+\.[0-9]+)-rc[0-9]+/$' +release_re='^datafusion-([0-9]+)\.[0-9]+\.[0-9]+/$' + +[ $# -eq 1 ] && [[ $1 =~ $version_re ]] || usage +version=$1 +major=${BASH_REMATCH[1]} +version_num=$(version_number "$version") + +dist_url=${DIST_URL:-https://dist.apache.org/repos/dist} +dev_url=$dist_url/dev/datafusion +release_url=$dist_url/release/datafusion + +dev_listing=$(svn ls "$dev_url") +release_listing=$(svn ls "$release_url") + +# Old files are deleted after the new release is published +if ! grep -qxF "datafusion-$version/" <<<"$release_listing"; then + echo "datafusion-$version/ is not in $release_url yet, run release-tarball.sh first" >&2 + exit 1 +fi + +rc_deletes=() +while read -r name; do + if [[ $name =~ $rc_re ]] && [ "$(version_number "${BASH_REMATCH[1]}")" -le "$version_num" ]; then + rc_deletes+=("$dev_url/${name%/}") + fi +done <<<"$dev_listing" + +release_deletes=() +kept=() +while read -r name; do + if [[ $name =~ $release_re ]]; then + if [ "${BASH_REMATCH[1]}" -lt "$major" ]; then + release_deletes+=("$release_url/${name%/}") + else + kept+=("${name%/}") + fi + fi +done <<<"$release_listing" + +echo "# Released version: $version" +echo +echo "# dev: release candidates of $version and older" +if [ ${#rc_deletes[@]} -gt 0 ]; then + print_delete "delete old DataFusion RCs" "${rc_deletes[@]}" +else + echo "# nothing to delete" +fi +echo +echo "# release: keep $major.0.0 and newer, delete older releases" +if [ ${#kept[@]} -gt 0 ]; then + echo "# keeping: ${kept[*]}" +fi +if [ ${#release_deletes[@]} -gt 0 ]; then + print_delete "delete old DataFusion releases" "${release_deletes[@]}" +else + echo "# nothing to delete" +fi From a1101e1c2b8559e57db5b9af7e0e9ec86593c00a Mon Sep 17 00:00:00 2001 From: comphead Date: Wed, 7 Oct 2026 15:59:53 -0700 Subject: [PATCH 7/7] docs: Replace the svn delete script with a short loop in the README --- dev/release/README.md | 32 +++++++-- dev/release/print-svn-deletes.sh | 116 ------------------------------- 2 files changed, 25 insertions(+), 123 deletions(-) delete mode 100755 dev/release/print-svn-deletes.sh diff --git a/dev/release/README.md b/dev/release/README.md index 6d54d79553518..42af909549d10 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -431,14 +431,32 @@ The release information is used to generate a template for a board report (see e See the ASF documentation on [when to archive](https://www.apache.org/legal/release-policy.html#when-to-archive) for more information. -Release candidates should be deleted once the release is published. In the `release` SVN keep the -last major release (`X.0.0`) and the releases after it, and delete older releases. For example, after -publishing 55.2.0 keep `datafusion-55.0.0`, `datafusion-55.1.0` and `datafusion-55.2.0`, and delete -every 54.x release. +Release candidates should be deleted once the release is published. -The `print-svn-deletes.sh` script prints the `svn delete` commands for a released version. It does not -run them, so check the printed commands and then run them: +To get a list of DataFusion release candidates: ```shell -./dev/release/print-svn-deletes.sh 55.2.0 +svn ls https://dist.apache.org/repos/dist/dev/datafusion +``` + +To delete a release candidate: + +```shell +svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-55.2.0-rc1/ +``` + +#### Delete old releases from `release` SVN + +Keep the last major release (`X.0.0`) and the releases after it. Delete older releases after +publishing the new release. For example, after publishing 55.2.0 keep `datafusion-55.0.0`, +`datafusion-55.1.0` and `datafusion-55.2.0`, and delete every 54.x release. + +The following prints, but does not run, the `svn delete` commands for all releases that do not +belong to the `major` version. Check the printed commands and then run them: + +```shell +major=55 +for release in $(svn ls https://dist.apache.org/repos/dist/release/datafusion | grep -E "^datafusion-[0-9]+\." | grep -v "^datafusion-$major\."); do + echo "svn delete -m 'delete old DataFusion release' https://dist.apache.org/repos/dist/release/datafusion/$release" +done ``` diff --git a/dev/release/print-svn-deletes.sh b/dev/release/print-svn-deletes.sh deleted file mode 100755 index 053090880cdcb..0000000000000 --- a/dev/release/print-svn-deletes.sh +++ /dev/null @@ -1,116 +0,0 @@ -#!/usr/bin/env bash -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# Prints the `svn delete` commands that clean up the Apache distribution SVN -# server after a release has been published. It only lists the SVN directories -# and never deletes anything. Check the printed commands and then run them. -# -# For the released version it prints deletes for -# - the release candidates of that version and older versions in the `dev` area -# - the releases older than the last major release (.0.0) in the -# `release` area. The last major release and the releases after it are kept. -# -# Usage: ./dev/release/print-svn-deletes.sh -# Example: ./dev/release/print-svn-deletes.sh 55.2.0 -# -# DIST_URL can point to another SVN repository, for example to test the script. - -set -euo pipefail - -usage() { - echo "Usage: $0 , for example: $0 55.2.0" >&2 - exit 1 -} - -# Turns X.Y.Z into a number that can be compared, for example 55.2.0 -> 55002000 -version_number() { - local maj min pat - IFS=. read -r maj min pat <<<"$1" - echo $((10#$maj * 1000000 + 10#$min * 1000 + 10#$pat)) -} - -# Prints an svn delete command with one URL per line -print_delete() { - local message=$1 url - shift - printf 'svn delete -m "%s"' "$message" - for url in "$@"; do - printf ' \\\n %s' "$url" - done - printf '\n' -} - -version_re='^([0-9]+)\.[0-9]+\.[0-9]+$' -rc_re='^apache-datafusion-([0-9]+\.[0-9]+\.[0-9]+)-rc[0-9]+/$' -release_re='^datafusion-([0-9]+)\.[0-9]+\.[0-9]+/$' - -[ $# -eq 1 ] && [[ $1 =~ $version_re ]] || usage -version=$1 -major=${BASH_REMATCH[1]} -version_num=$(version_number "$version") - -dist_url=${DIST_URL:-https://dist.apache.org/repos/dist} -dev_url=$dist_url/dev/datafusion -release_url=$dist_url/release/datafusion - -dev_listing=$(svn ls "$dev_url") -release_listing=$(svn ls "$release_url") - -# Old files are deleted after the new release is published -if ! grep -qxF "datafusion-$version/" <<<"$release_listing"; then - echo "datafusion-$version/ is not in $release_url yet, run release-tarball.sh first" >&2 - exit 1 -fi - -rc_deletes=() -while read -r name; do - if [[ $name =~ $rc_re ]] && [ "$(version_number "${BASH_REMATCH[1]}")" -le "$version_num" ]; then - rc_deletes+=("$dev_url/${name%/}") - fi -done <<<"$dev_listing" - -release_deletes=() -kept=() -while read -r name; do - if [[ $name =~ $release_re ]]; then - if [ "${BASH_REMATCH[1]}" -lt "$major" ]; then - release_deletes+=("$release_url/${name%/}") - else - kept+=("${name%/}") - fi - fi -done <<<"$release_listing" - -echo "# Released version: $version" -echo -echo "# dev: release candidates of $version and older" -if [ ${#rc_deletes[@]} -gt 0 ]; then - print_delete "delete old DataFusion RCs" "${rc_deletes[@]}" -else - echo "# nothing to delete" -fi -echo -echo "# release: keep $major.0.0 and newer, delete older releases" -if [ ${#kept[@]} -gt 0 ]; then - echo "# keeping: ${kept[*]}" -fi -if [ ${#release_deletes[@]} -gt 0 ]; then - print_delete "delete old DataFusion releases" "${release_deletes[@]}" -else - echo "# nothing to delete" -fi