Skip to content

Parallelize CI #932

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
levkk merged 16 commits into from
Apr 25, 2026
Merged

Parallelize CI #932

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
3b94acb
chore: base images for everything
levkk Apr 24, 2026
6cb7198
ci: trigger base-image builds on levkk-base-package-images
levkk Apr 24, 2026
f6c3e77
fix(ci-base): rustup components need repeated -c flags
levkk Apr 24, 2026
9b21595
ok lets try this
levkk Apr 24, 2026
fca6795
save
levkk Apr 24, 2026
4ef1b68
hmm
levkk Apr 24, 2026
1c9e11c
moar jobs
levkk Apr 24, 2026
76b8e93
letsgoo
levkk Apr 24, 2026
5667e42
playing around
levkk Apr 24, 2026
0331f99
better cache key
levkk Apr 24, 2026
8b1f36f
remove ci image
levkk Apr 24, 2026
4629579
rebase
levkk Apr 25, 2026
0aeed4c
add missing deps
levkk Apr 25, 2026
3ba762d
use same binary for plugin tests
levkk Apr 25, 2026
a40ddeb
remove rwx
levkk Apr 25, 2026
375d297
clippy all targets
levkk Apr 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
302 changes: 73 additions & 229 deletions .github/workflows/ci.yml
View file
Open in desktop
Comment thread
levkk marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -7,248 +7,92 @@ on:
types: [opened, synchronize, reopened]

jobs:
fmt:
runs-on: blacksmith-4vcpu-ubuntu-2404
steps:
- uses: actions/checkout@v4
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
override: true
- name: Install CMake 3.31
run: |
sudo apt update && sudo apt install mold -y
sudo apt remove cmake
sudo pip3 install cmake==3.31.6
cmake --version
- name: Format
run: cargo fmt --all -- --check
- name: Clippy
working-directory: pgdog
run: cargo clippy
build:
runs-on: blacksmith-4vcpu-ubuntu-2404
steps:
- uses: actions/checkout@v4
- uses: actions-rs/toolchain@v1
- name: Install CI deps
run: bash integration/ci/install-deps.sh
# Binary cache runs first: its key must be computed against a clean
# tree so it matches what downstream jobs compute. rust-cache runs
# after — it restores target/ which contains generated *.rs under
# target/*/build/*/out/ that would otherwise destabilize the hash.
- name: Compute cache key
id: cache-key
run: echo "key=pgdog-bin-${{ runner.os }}-$(bash integration/ci/cache-key.sh)" >> "$GITHUB_OUTPUT"
- name: Cache pgdog binaries
id: pgdog-bin
uses: useblacksmith/cache@v5
with:
toolchain: stable
override: true
- name: Install CMake 3.31
run: |
sudo apt update && sudo apt install mold -y
sudo apt remove cmake
sudo pip3 install cmake==3.31.6
cmake --version
- name: Build
run: cargo build
- name: Check release
run: cargo check --release
tests:
runs-on: blacksmith-4vcpu-ubuntu-2404
steps:
- uses: actions/checkout@v4
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
override: true
components: llvm-tools-preview
path: |
target/debug/pgdog
target/release/pgdog
key: ${{ steps.cache-key.outputs.key }}
- uses: useblacksmith/rust-cache@v3
if: steps.pgdog-bin.outputs.cache-hit != 'true'
with:
prefix-key: "v1" # Change this when updating tooling
- name: Setup PostgreSQL
run: |
sudo service postgresql start
sudo -u postgres createuser --superuser --login $USER
sudo -u postgres createdb $USER
bash integration/setup.sh || (sudo service postgresql restart && bash integration/setup.sh)
sudo apt update && sudo apt install -y python3-virtualenv mold
sudo gem install bundler
sudo apt remove -y cmake
sudo pip3 install cmake==3.31.6
cmake --version
cargo install cargo-nextest --version "0.9.78" --locked
bash integration/toxi/setup.sh
- name: Install test dependencies
run: cargo install cargo-nextest --version "0.9.78" --locked
- name: Install coverage tooling
run: cargo install cargo-llvm-cov --locked --version "0.6.10"
- name: Run tests with coverage
env:
RUSTFLAGS: "-C link-dead-code"
run: |
cargo llvm-cov clean --workspace
cargo llvm-cov nextest --lcov --output-path lcov.info --no-fail-fast --test-threads=1 --filter-expr "package(pgdog) | package(pgdog-config) | package(pgdog-vector) | package(pgdog-stats) | package(pgdog-postgres-types)"
- name: Run documentation tests
run: cargo test --doc
# Requires CODECOV_TOKEN secret for upload
- uses: codecov/codecov-action@v4
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
with:
files: lcov.info
flags: unit
fail_ci_if_error: true
integration:
prefix-key: build-v1
- name: Build (debug)
if: steps.pgdog-bin.outputs.cache-hit != 'true'
run: cargo build --bin pgdog
- name: Build (release)
if: steps.pgdog-bin.outputs.cache-hit != 'true'
run: cargo build --release --bin pgdog

ci:
runs-on: blacksmith-4vcpu-ubuntu-2404
needs: [build]
timeout-minutes: 30
continue-on-error: ${{ matrix.continue_on_error == true }}
strategy:
fail-fast: false
matrix:
Comment thread
levkk marked this conversation as resolved.
include:
- { name: pgbench, script: integration/pgbench/run.sh }
- { name: schema-sync, script: integration/schema_sync/run.sh }
- { name: go, script: integration/go/run.sh }
- { name: js, script: integration/js/pg_tests/run.sh }
- { name: ruby, script: integration/ruby/run.sh }
- { name: java, script: integration/java/run.sh }
- { name: mirror, script: integration/mirror/run.sh }
- { name: sql, script: integration/sql/run.sh }
- { name: toxi, script: integration/toxi/run.sh }
- { name: rust, script: integration/rust/run.sh }
- { name: python, script: integration/python/run.sh }
- { name: complex, script: integration/complex/run.sh }
- { name: dry-run, script: integration/dry_run/run.sh }
- { name: copy-data, script: integration/copy_data/run.sh }
- { name: load-balancer, script: integration/load_balancer/run.sh }
Comment thread
levkk marked this conversation as resolved.
# plugins/run.sh builds 4 plugin crates with cargo, so it needs
# the workspace target/ cache; the other entries just run the
# cached pgdog binary.
- { name: plugins, script: integration/plugins/run.sh, needs_rust_cache: true, continue_on_error: true }
env:
LLVM_PROFILE_FILE: ${{ github.workspace }}/target/llvm-cov-target/profiles/pgdog-%p-%m.profraw
PGDOG_BIN: ${{ github.workspace }}/target/release/pgdog
steps:
- uses: actions/checkout@v4
- uses: actions-rs/toolchain@v1
- name: Install CI deps
run: bash integration/ci/install-deps.sh
- name: Compute cache key
id: cache-key
run: echo "key=pgdog-bin-${{ runner.os }}-$(bash integration/ci/cache-key.sh)" >> "$GITHUB_OUTPUT"
- name: Restore pgdog binaries
uses: useblacksmith/cache/restore@v5
with:
toolchain: stable
override: true
components: llvm-tools-preview
- uses: useblacksmith/rust-cache@v3
path: |
target/debug/pgdog
target/release/pgdog
key: ${{ steps.cache-key.outputs.key }}
fail-on-cache-miss: true
- name: Restore Rust cache for plugin builds
if: matrix.needs_rust_cache
uses: useblacksmith/rust-cache@v3
with:
prefix-key: release-1
prefix-key: build-v1
- name: Setup dependencies
run: |
sudo service postgresql start
sudo -u postgres createuser --superuser --login $USER
sudo -u postgres createdb $USER
bash integration/setup.sh || (sudo service postgresql restart && bash integration/setup.sh)
sudo apt update && sudo apt install -y python3-virtualenv mold php-cli php-pgsql curl ca-certificates
# Use a versioned PostgreSQL 18 client so the protocol-version smoke
# test does not depend on whatever libpq the runner image ships.
sudo install -d /usr/share/postgresql-common/pgdg
sudo curl -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc --fail https://www.postgresql.org/media/keys/ACCC4CF8.asc
. /etc/os-release
sudo sh -c "echo 'deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $VERSION_CODENAME-pgdg main' > /etc/apt/sources.list.d/pgdg.list"
sudo apt update && sudo apt install -y postgresql-client-18
echo "PSQL_BIN=/usr/lib/postgresql/18/bin/psql" >> "$GITHUB_ENV"
sudo gem install bundler
sudo apt remove -y cmake
sudo pip3 install cmake==3.31.6
cmake --version
cargo install cargo-nextest --version "0.9.78" --locked
bash integration/toxi/setup.sh
sudo curl -SL https://github.com/docker/compose/releases/download/v2.36.1/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
- name: Install coverage tooling
run: cargo install cargo-llvm-cov --locked --version "0.6.10"
- name: Prepare instrumented PgDog build
env:
RUSTFLAGS: "-C link-dead-code"
run: |
cargo llvm-cov clean --workspace
mkdir -p target/llvm-cov-target/profiles
cargo llvm-cov run --no-report --release --package pgdog --bin pgdog -- --help
rm -f target/llvm-cov-target/profiles/*.profraw
rm -f target/llvm-cov-target/profiles/.last_snapshot
rm -rf target/llvm-cov-target/reports
BIN_PATH=$(find target/llvm-cov-target -type f -path '*/release/pgdog' | head -n 1)
if [ -z "$BIN_PATH" ]; then
echo "Instrumented PgDog binary not found" >&2
exit 1
fi
echo "Using instrumented binary at $BIN_PATH"
echo "PGDOG_BIN=$(realpath "$BIN_PATH")" >> "$GITHUB_ENV"
- name: pgbench
run: bash integration/pgbench/run.sh
- name: schema-sync
run: bash integration/schema_sync/run.sh
- name: Go
run: bash integration/go/run.sh
- name: JavaScript
run: bash integration/js/pg_tests/run.sh
- name: Ruby
run: bash integration/ruby/run.sh
- name: Java
run: bash integration/java/run.sh
- name: Mirror
run: bash integration/mirror/run.sh
- name: SQL
run: bash integration/sql/run.sh
- name: Toxi
run: bash integration/toxi/run.sh
- name: Rust
run: bash integration/rust/run.sh
- name: Stop shared PgDog
run: bash -lc 'source integration/common.sh; stop_pgdog'
- name: Data sync
run: bash integration/copy_data/run.sh
- name: Python
run: bash integration/python/run.sh
- name: Load balancer
run: bash integration/load_balancer/run.sh
- name: More complex stuff
run: bash integration/complex/run.sh
- name: Dry run
run: bash integration/dry_run/run.sh
run: bash integration/ci/setup.sh --with-toxi
- name: Run ${{ matrix.name }}
run: bash ${{ matrix.script }}
- name: Ensure PgDog stopped
run: |
if pgrep -x pgdog > /dev/null; then
killall -TERM pgdog
sleep 5
fi
- name: Generate integration coverage report
run: cargo llvm-cov report --release --package pgdog --lcov --output-path integration.lcov
# Requires CODECOV_TOKEN secret for upload
- uses: codecov/codecov-action@v4
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
with:
files: integration.lcov
flags: integration
fail_ci_if_error: true
plugin-unit-tests:
runs-on: blacksmith-4vcpu-ubuntu-2404
continue-on-error: true
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
override: true
- uses: useblacksmith/rust-cache@v3
with:
prefix-key: "plugin-unit-v1"
- name: Install CMake 3.31
run: |
sudo apt-get install -y mold
sudo apt remove cmake
sudo pip3 install cmake==3.31.6
cmake --version
- name: Install test dependencies
run: cargo install cargo-nextest --version "0.9.78" --locked
- name: Run plugin unit tests
run: cargo nextest run -E 'package(pgdog-example-plugin)' --no-fail-fast
plugin-integration-tests:
runs-on: blacksmith-4vcpu-ubuntu-2404
continue-on-error: true
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
override: true
- uses: useblacksmith/rust-cache@v3
with:
prefix-key: "plugin-integration-v1"
- name: Setup PostgreSQL
run: |
sudo service postgresql start
sudo -u postgres createuser --superuser --login $USER
sudo -u postgres createdb $USER
bash integration/setup.sh || (sudo service postgresql restart && bash integration/setup.sh)
- name: Install dependencies
run: |
sudo apt update && sudo apt install -y python3-virtualenv mold
sudo gem install bundler
sudo apt remove -y cmake
sudo pip3 install cmake==3.31.6
cmake --version
cargo install cargo-nextest --version "0.9.78" --locked
- name: Build plugin
run: |
cargo build --release
pushd plugins/pgdog-example-plugin
cargo build --release
popd
- name: Run plugin integration tests
run: bash integration/plugins/run.sh
if: always()
run: bash integration/ci/ensure-pgdog-stopped.sh
19 changes: 19 additions & 0 deletions .github/workflows/fmt.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
name: fmt
on:
push:
branches:
- main
pull_request:
types: [opened, synchronize, reopened]

jobs:
fmt:
runs-on: blacksmith-4vcpu-ubuntu-2404
steps:
- uses: actions/checkout@v4
- name: Install CI deps
run: bash integration/ci/install-deps.sh
- name: Format
run: cargo fmt --all -- --check
- name: Clippy
run: cargo clippy --all-targets
Loading
Loading