Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .github/workflows/ci-macos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,13 @@ jobs:
- '.github/workflows/**'
- 'Cargo.toml'
- 'Cargo.lock'
- 'Justfile'
- 'README.md'
- '_typos.toml'
- '.rustfmt.toml'
- 'benches/**'
- 'docs/**'
- 'rustfmt.toml'
- 'src/**'
- 'tests/**'
- 'examples/**'
Expand Down
7 changes: 7 additions & 0 deletions .github/workflows/ci-ubuntu.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,13 @@ jobs:
- '.github/workflows/**'
- 'Cargo.toml'
- 'Cargo.lock'
- 'Justfile'
- 'README.md'
- '_typos.toml'
- '.rustfmt.toml'
- 'benches/**'
- 'docs/**'
- 'rustfmt.toml'
- 'src/**'
- 'tests/**'
- 'examples/**'
Expand Down
45 changes: 45 additions & 0 deletions .github/workflows/ci-windows.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
name: CI / Windows

on:
workflow_dispatch:
push:
branches: ["master"]
pull_request:
branches: ["master"]

permissions:
contents: read

jobs:
quality:
name: Quality
uses: ./.github/workflows/quality.yml
with:
runner: windows-latest

unit:
name: Unit
needs: [quality]
uses: ./.github/workflows/unit.yml
with:
runner: windows-latest

integration:
name: Disk writer integration
needs: [quality]
runs-on: windows-latest

steps:
- uses: actions/checkout@v5

- uses: dtolnay/rust-toolchain@stable

- uses: Swatinem/rust-cache@v2
with:
shared-key: cargo-windows-latest

- name: Windows disk writer contracts
run: cargo test --test contracts --features testing,disk-writer disk_writer_contract::

- name: API benchmark smoke test
run: cargo bench --bench api_workloads --features memory-metadata,disk-writer,opendal-reader-hf -- --test
18 changes: 16 additions & 2 deletions .github/workflows/integration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@ permissions:
contents: read

env:
API_BENCHMARK_FEATURES: memory-metadata,disk-writer,opendal-reader-hf
CARGO_TERM_COLOR: always
CORE_FEATURES: testing,memory-metadata,disk-writer,opendal-reader-fs,validators

jobs:
integration:
Expand All @@ -31,5 +33,17 @@ jobs:
with:
shared-key: cargo-${{ inputs.runner }}

# - name: Integration tests
# run: cargo nextest run --tests it
- name: Integration tests
run: cargo nextest run --tests --features "${{ env.CORE_FEATURES }}"

- name: API benchmark smoke test
run: cargo bench --bench api_workloads --features "${{ env.API_BENCHMARK_FEATURES }}" -- --test

- uses: taiki-e/install-action@v2
with:
tool: cargo-llvm-cov

- name: Coverage
run: |
cargo llvm-cov --workspace --features "${{ env.CORE_FEATURES }}" --no-report
cargo llvm-cov report --ignore-filename-regex 'src/utils/testing/' --fail-uncovered-lines 0 --fail-uncovered-functions 0 --fail-uncovered-regions 0
35 changes: 26 additions & 9 deletions .github/workflows/quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ permissions:

env:
CARGO_TERM_COLOR: always
CORE_FEATURES: testing,memory-metadata,disk-writer,opendal-reader-fs,validators
API_BENCHMARK_FEATURES: memory-metadata,disk-writer,opendal-reader-hf
COMPONENT_BENCHMARK_FEATURES: memory-metadata,disk-writer,opendal-reader-fs

jobs:
code-quality:
Expand Down Expand Up @@ -40,21 +43,35 @@ jobs:
run: rustup toolchain install nightly --profile minimal --component rustfmt

- name: Rustfmt
run: |
cargo +nightly fmt --all
if ! git diff --quiet; then
git status --short
git diff --stat
exit 1
fi
run: cargo +nightly fmt --all -- --check

- name: Clippy
run: cargo clippy --all-features --all-targets -- -D warnings
run: |
cargo clippy --all-targets --features "${{ env.CORE_FEATURES }}" -- -D warnings
cargo clippy --bench api_workloads --features "${{ env.API_BENCHMARK_FEATURES }}" -- -D warnings

- name: Feature compile checks
run: |
cargo check
cargo check --tests
cargo check --features opendal-reader-memory
cargo check --features opendal-reader-http
cargo check --features opendal-reader-hf
cargo check --features opendal-reader-s3
cargo check --features opendal-reader-sftp

- name: Install MSRV toolchain
run: rustup toolchain install 1.91.0 --profile minimal

- name: MSRV
run: cargo +1.91.0 check --all-targets --features "${{ env.CORE_FEATURES }}"

- name: Benchmarks compile
run: |
cargo bench --benches --features "${{ env.COMPONENT_BENCHMARK_FEATURES }}" --no-run
cargo bench --bench api_workloads --features "${{ env.API_BENCHMARK_FEATURES }}" --no-run

- name: Docs.rs feature set
run: RUSTDOCFLAGS='--cfg docsrs' cargo doc --all-features --no-deps
env:
RUSTDOCFLAGS: --cfg docsrs -D warnings -D missing-docs
run: cargo doc --features "${{ env.CORE_FEATURES }}" --no-deps
7 changes: 4 additions & 3 deletions .github/workflows/unit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ permissions:

env:
CARGO_TERM_COLOR: always
CORE_FEATURES: testing,memory-metadata,disk-writer,opendal-reader-fs,validators

jobs:
unit:
Expand All @@ -31,8 +32,8 @@ jobs:
with:
shared-key: cargo-${{ inputs.runner }}

- name: Unit tests (no features)
run: cargo nextest run --lib
- name: Unit tests
run: cargo nextest run --lib --features "${{ env.CORE_FEATURES }}"

- name: Doc tests
run: cargo test --doc --all-features
run: cargo test --doc --features "${{ env.CORE_FEATURES }}"
101 changes: 28 additions & 73 deletions docs/CI.md
Original file line number Diff line number Diff line change
@@ -1,75 +1,30 @@
# CI Pipelines

## Pipeline Graph

```mermaid
flowchart TD
U["CI / Ubuntu"]
M["CI / macOS"]
Q(["Quality"])
T(["Unit"])
I(["Integration"])

U --> Q
M --> Q
Q -.-> T
Q -.-> I
```

## Workflow Layout

- [`.github/workflows/ci-ubuntu.yml`](../.github/workflows/ci-ubuntu.yml) is the Ubuntu entry workflow.
- [`.github/workflows/ci-macos.yml`](../.github/workflows/ci-macos.yml) is the macOS entry workflow.
- [`.github/workflows/quality.yml`](../.github/workflows/quality.yml) defines reusable quality and compilation checks.
- [`.github/workflows/unit.yml`](../.github/workflows/unit.yml) defines reusable unit and doc test execution.
- [`.github/workflows/integration.yml`](../.github/workflows/integration.yml) defines
reusable integration test execution.

## Job Order

`CI / Ubuntu` and `CI / macOS` both route into the same logical pipeline shape.
`quality` runs first, then `unit` and `integration` both depend on `quality`, so
formatting, linting, and compile validation must pass before the test fan-out begins.

## Quality Workflow

The quality workflow performs static validation and build verification before tests run.

### [Typos][typos]

- `typos` checks source code and documentation for spelling mistakes.
- Version `1.49.0` is pinned so dictionary updates do not unexpectedly break the pipeline.

### [rustfmt][rustfmt]

- `cargo +nightly fmt --all` checks formatting consistency by reformatting the workspace.
- The workflow fails if formatting would change tracked files.

### [Clippy][clippy]

- `cargo clippy --all-features --all-targets -- -D warnings` runs lints across all targets and features.
- `-D warnings` promotes warnings to errors so the job fails on any lint finding.

### Feature Compile Checks

This catches missing imports, `cfg` mistakes, and feature-gating regressions without
requiring full test execution for every combination.

### [docs.rs][docs-rs] Feature Set

- `RUSTDOCFLAGS='--cfg docsrs' cargo doc --all-features --no-deps` validates that
documentation builds under a docs.rs-like configuration.
- This helps catch documentation-only compilation issues and cfg-gated API doc failures.

## Unit Workflow

The unit workflow focuses on fast correctness checks that do not require the broader integration feature matrix.

## Integration Workflow

The integration workflow exercises feature-backed behavior and the broader end-to-end test matrix.

[clippy]: https://doc.rust-lang.org/clippy/
[docs-rs]: https://docs.rs/
[rustfmt]: https://github.com/rust-lang/rustfmt
[typos]: https://github.com/crate-ci/typos
Ubuntu and macOS entry workflows call the same quality, unit, and integration
workflows. Windows runs the shared quality and unit jobs plus the disk writer contracts
that exercise its native atomic replacement path. Tests begin only after quality checks
pass.

The shared core feature set enables test support, the in-memory metadata backend, disk
writer, OpenDAL filesystem reader, and validators. Quality checks also compile default,
test-only, and representative OpenDAL memory, HTTP, S3, and SFTP combinations.

## Quality

The quality workflow runs pinned typo checking, nightly rustfmt in check mode, Clippy
with warnings denied, feature compile checks, the Rust 1.91 MSRV check, benchmark
compilation, and rustdoc with warnings and missing public documentation denied. OpenDAL service combinations are sampled
instead of combining every mutually independent integration into one oversized build.

## Tests and Coverage

Unit and documentation tests run separately from feature-backed integration tests.
Nextest supplies concise execution and failure output. The integration workflow also
runs the controlled API benchmark in smoke mode. `cargo-llvm-cov` permits zero uncovered production lines, functions, or
regions. Feature-gated test implementations under `src/utils/testing/` are excluded
from the production coverage denominator.

Run the complete local workflow with `just ci`. Individual build and verification
tasks are available in the `Justfile`. Use `just bench-components` for Divan
microbenchmarks, `just bench-api` for controlled workloads, and `just bench-smoke` for
benchmark correctness checks.
Loading
Loading