Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
344 lines (239 loc) · 12.4 KB

CONTRIBUTING.md

File metadata and controls

344 lines (239 loc) · 12.4 KB
Raw

Introduction

We welcome and encourage contributions of all kinds, such as:

  1. Tickets with issue reports of feature requests
  2. Documentation improvements
  3. Code (PR or PR Review)

In addition to submitting new PRs, we have a healthy tradition of community members helping review each other's PRs. Doing so is a great way to help the community as well as get more familiar with Rust and the relevant codebases.

Finding and Creating Issues to Work On

You can find a curated good-first-issue list to help you get started.

Arrow-rs is an open contribution project, and thus there is no particular project imposed deadline for completing any issue or any restriction on who can work on an issue, nor how many people can work on an issue at the same time.

Contributors drive the project forward based on their own priorities and interests and thus you are free to work on any issue that interests you.

If someone is already working on an issue that you want or need but hasn't been able to finish it yet, you should feel free to work on it as well. In general it is both polite and will help avoid unnecessary duplication of work if you leave a note on an issue when you start working on it.

If you want to work on an issue which is not already assigned to someone else and there are no comment indicating that someone is already working on that issue then you can assign the issue to yourself by submitting a single word comment take. This will assign the issue to yourself. However, if you are unable to make progress you should unassign the issue so that someone else can get involved in the work. You can do this either by using the unassign me link at the top of the issue page or by leaving a single word comment untake (and ask for help if you are stuck).

If you plan to work on a new feature that doesn't have an existing ticket, it is a good idea to open a ticket to discuss the feature. Advanced discussion often helps avoid wasted effort by determining early if the feature is a good fit for Arrow-rs before too much time is invested. It also often helps to discuss your ideas with the community to get feedback on implementation.

Creating Pull Requests

With coding agents, the number of open PRs now far exceeds our review capacity. To encourage submitters to curate and prepare a small number of high quality PRs, and to help reviewers focus on them, users without write access are limited to 3 open, non-draft PRs at a time.

When possible, we recommend splitting your contributions into multiple smaller focused PRs rather than large PRs, because they are more likely to be reviewed quickly and are easier to understand and review.

Developer's guide to Arrow Rust

Setting Up Your Build Environment

Install the Rust tool chain:

https://www.rust-lang.org/tools/install

Also, make sure your Rust tool chain is up-to-date, because we always use the latest stable version of Rust to test this project.

rustup update stable

How to compile

This is a standard cargo project with workspaces. To build it, you need to have rust and cargo:

cargo build

You can also use rust's official docker image:

docker run --rm -v $(pwd):/arrow-rs -it rust /bin/bash -c "cd /arrow-rs && rustup component add rustfmt && cargo build"

The command above assumes that are in the root directory of the project.

You can also compile specific workspaces:

cd arrow && cargo build

Git Submodules

Before running tests and examples, it is necessary to set up the local development environment.

The tests rely on test data that is contained in git submodules.

To pull down this data run the following:

git submodule update --init

This populates data in two git submodules:

By default, cargo test will look for these directories at their standard location. The following environment variables can be used to override the location:

# Optionally specify a different location for test data
export PARQUET_TEST_DATA=$(cd ./parquet-testing/data; pwd)
export ARROW_TEST_DATA=$(cd ./testing/data; pwd)

From here on, this is a pure Rust project and cargo can be used to run tests, benchmarks, docs and examples as usual.

Running the tests

Run tests using the Rust standard cargo test command:

# run all unit and integration tests
cargo test

# run tests for the arrow crate
cargo test -p arrow

For some changes, you may want to run additional tests. You can find up-to-date information on the current CI tests in .github/workflows. Here are some examples of additional tests you may want to run:

# run tests for the parquet crate
cargo test -p parquet

# run arrow tests with all features enabled
cargo test -p arrow --all-features

# run the doc tests
cargo test --doc

Code Formatting

Our CI uses rustfmt to check code formatting. Before submitting a PR be sure to run the following and check for lint issues:

cargo +stable fmt --all -- --check

Miri

Run tests under Miri like so, assuming cargo-nextest is available:

# Run all tests
MIRIFLAGS="-Zmiri-disable-isolation -Zmiri-no-extra-rounding-error" cargo +nightly miri nextest run
# Run specific tests
MIRIFLAGS="-Zmiri-disable-isolation -Zmiri-no-extra-rounding-error" cargo +nightly miri nextest run -p arrow-buffer --lib bigint
# Run specific crate
MIRIFLAGS="-Zmiri-disable-isolation -Zmiri-no-extra-rounding-error" cargo +nightly miri nextest run -p arrow-buffer

The whole suite will take a long time to run so it's suggested to run individual suites/tests when required.

Add a cfg_attr to tests in cases where the test should be ignored by Miri:

#[test]
#[cfg_attr(miri, ignore)] // Takes too long
fn test123() {}

Please ensure you include a comment why the test is being ignored. Common cases are:

  • Takes too long
    • Threshold is if it takes longer than 1 minute
  • Zstd code unsupported by Miri
  • Inline assembly unsupported by Miri
  • Any other operation thats unsupported by Miri

Breaking Changes

Our release schedule allows breaking API changes only in major releases. This means that if your PR has a breaking API change, it should be marked as api-change and it will not be merged until development opens for the next major release. See this ticket for details.

Clippy Lints

We use clippy for checking lints during development, and CI runs clippy checks.

Run the following to check for clippy lints:

# run clippy with default settings
cargo clippy --workspace --all-targets --all-features -- -D warnings

If you use Visual Studio Code with the rust-analyzer plugin, you can enable clippy to run each time you save a file. See https://users.rust-lang.org/t/how-to-use-clippy-in-vs-code-with-rust-analyzer/41881.

In addition to the lints that clippy enables by default, we enable a few extra ones in [workspace.lints] in the root Cargo.toml. Every crate in the workspace opts in to those with:

[lints]
workspace = true

New crates should include that section, and new lints should be added to [workspace.lints] rather than to individual crates, so that they apply everywhere.

One of the concerns with clippy is that it often produces a lot of false positives, or that some recommendations may hurt readability. We do not have a policy of which lints are ignored, but if you disagree with a clippy lint, you may disable the lint and briefly justify it.

Search for expect(clippy:: in the codebase to identify lints that are intentionally suppressed. We currently prefer suppressing lints on the lowest unit possible.

  • If you are introducing a line that returns a lint warning or error, you may disable the lint on that line.
  • If you have several lints on a function or module, you may disable the lint on the function or module.
  • If a lint is pervasive across multiple modules, you may disable it at the crate level.

Spell Checking

We use typos to catch spelling mistakes in source code and comments. CI runs a spell check on every pull request.

Running locally

Install typos and run it from the repo root:

cargo install typos-cli
typos --config typos.toml

Adding a pre-commit hook

To catch typos before they reach CI, add typos as a Git pre-commit hook:

cat > .git/hooks/pre-commit << 'EOF'
#!/bin/sh
typos --config typos.toml
EOF
chmod +x .git/hooks/pre-commit

After that, typos will run automatically on every git commit. If it finds an issue, the commit is blocked until you fix or allowlist the word.

Allowlisting false positives

If typos flags a word that is intentionally spelled that way (a test value, a domain-specific term, a crate name, etc.), add it to [default.extend-words] in typos.toml at the repo root with a short comment explaining why:

[default.extend-words]
flate = "flate"  # flate2 is the crate name for the deflate/inflate compression library

Prefer renaming variables over adding allowlist entries when the flagged word is just a local identifier that can be changed without affecting test semantics.

Performance Improvements

Pull requests that improve performance, especially those that add non-trivial complexity or use unsafe, should include evidence of the improvement, such as benchmarks.

As benchmarks usually take a long time to run, we recommend running targeted tests instead of the full suite.

# run all benchmarks
cargo bench

# run arrow benchmarks
cargo bench -p arrow

# run benchmark for the parse_time function within the arrow-cast crate
cargo bench -p arrow-cast --bench parse_time

To set the baseline for your benchmarks, use the --save-baseline flag:

git checkout main

cargo bench --bench parse_time -- --save-baseline main

git checkout feature

cargo bench --bench parse_time -- --baseline main

If your PR proposes a performance improvement, include a summary of the benchmark results (for example, from cargo-criterion or critcmp) in the PR description. If you need to add new benchmarks to cover your change, make a separate PR first (for example, #9729) so we can run the benchmarks on an automated runner.

AI Generated Submissions

This project follows the guidance for AI generated submissions used by the Arrow Project. As such, it is expected that you will:

  • Only submit a PR if you are able to debug and own the changes yourself - review all generated code to understand every detail
  • Match the style and conventions used in the rest of the codebase, including PR titles and descriptions
  • Be upfront about AI usage and summarise what was AI-generated
  • If there are parts you don’t fully understand, leave comments on your own PR explaining what steps you took to verify correctness
  • Watch for AI’s tendency to generate overly verbose comments, unnecessary test cases, and incorrect fixes
  • Break down large PRs into smaller ones to make review easier

It is also important for submitters to be aware of potential copyright issues. See the ASF's guidance on AI-generated code for further information on licensing considerations.