fuzzing.md

Fuzzing npro

Everything in npro that reads bytes from outside is fuzzed, and each fuzz target checks the code under test against an oracle: another way of getting the same answer, or a property the answer must have. A target that only waits for a panic finds out little about code that cannot panic; one with an oracle finds wrong answers too.

Fuzzing runs in three places:

where	what	when
cargo test	the smoke tests: every target over its seeds and 4000 inputs from a fixed seed	every build, every builder, Miri included
sai fuzz	the known bugs replayed, then libFuzzer for a minute on each target	every push
sai idle tasks	libFuzzer, the targets taking turns	builders' idle time, on the latest completed push

The targets

target	the code under test	the oracle
utf8	npro_core::utf8::Utf8Validator, which checks ws text as it arrives	core::str::from_utf8 over the whole text. Fed a byte at a time, the validator must refuse at exactly the first byte no well-formed text could have next; fed in pieces of 0 to 16 bytes, it must refuse in the piece holding that byte, and go on refusing
sha1	npro_core::sha1::Sha1, for the ws accept	the digest of the message fed in pieces equals the digest of it in one go
base64	npro_core::base64::encode	a strict decoder in the harness gets the input back; exactly encoded_len bytes are used and no more written; a buffer one byte short, or empty, is refused with nothing written
transcript	npro-test's transcript reader	whatever it accepts keeps its limits, and step times never go backwards
h1-request	npro_h1::head as a server	the head in one piece and in pieces comes to the same verdict and leaves the same table; a refused head stays refused; no value holds a CR, LF or NUL; a complete request's path from / has no //, /./ or /../ step and no /. or /.. at its end. The top bit of the first byte picks a 256 byte table with token limits, so limits are within reach
h1-response	npro_h1::head as a client	as h1-request
chunked	npro_h1::chunked::Dechunk	a second reading of RFC 9112 7.1 with C's bounds, written apart from the decoder: the same data, ending at the same byte, or refused, in one piece and in pieces
uri	npro_h1::uri, a URL parsed or, with the first byte's top bit, a reference resolved against one of three bases	what is kept has a host of unreserved characters or an IPv6 literal, a port that is not 0, and a target from / with no dot segments in its path and only RFC 3986's characters, each % followed by two hex digits; written out again it parses to itself; resolving the empty reference against it gives it back; storage one byte short of it is refused
json	npro_json::Parser; the first byte's top bit picks 24 bytes for a token, 48 for names and a depth of 4, so the limits are within reach	in one piece and in pieces, the same events and the same verdict, refused at the same byte, and every call takes or gives something. An accepted document is checked apart from the parser: one value, every container closed, no deeper than allowed, no object with a name twice, no string with U+0000 or a noncharacter, every number one f64 reads; written out again compactly, it parses to the same events
ws-server	npro_ws::conn::Ws as a server, after the upgrade, as C's fuzz-ws	in one piece and in pieces, the same messages, pongs and peer's close handed to the application, the same frames written and the same close; where text turns out not to be UTF-8, how much of its good start was handed over may differ with the split, the one only starting the other. Every call takes something; a message's pieces say where it starts; nothing follows the peer's close; a whole text message is UTF-8 by core::str; what is written is whole, final, unmasked frames, a close last
ws-client	npro_ws::conn::Ws as a client, after the upgrade	as ws-server, from the client's side: what it writes is masked, with a fixed mask from the harness
ws-pmd	npro_ws::conn::Ws as a server with permessage-deflate, as C's fuzz-ws-pmd	as ws-server, and while it says it has more to give without input (rx_pending), it gives it. Messages that end must agree exactly; how far the inflater gets with the input it has depends on where that stops, so a message left unfinished need only start the other run's, and where a message does not inflate, text in it that is not UTF-8 may be refused first, or not, both runs failing. A message may inflate to 1MiB, so the zip bomb seed reaches the limit
ss-server	npro_ss::ss::Ss's server streams, on an h1 listener with two vhosts (one serving everything with a route under it, one serving a single exact path) and a ws listener whose route takes ws with permessage-deflate, in three slots with 96 byte buffers; the input is a script of operations: accepts, streams connecting out and their connects made or failed, a peer's bytes, the application's answers and calls, short writes, hangups, FINs and time passing	a model of the streams beside it: no look at them goes on without end; a stream is told nothing once destroyed or told it is gone, and its id is refused from then; a released connection's id is refused; an accept is refused only when the slots may all be in use; a request is routed to a route whose path leads its own, with the rest of the path given, and on the vhost serving everything none goes unrouted; every request told is followed by one transaction end, before the stream is told it is gone, and no second request before that; the access log line made from each request is one whole line, its six quotes and no control byte, however long or strange the peer's fields. At the end, every connection hung up and released, each stream accepted was destroyed or told it is gone, once, and every slot takes a connection again
files	npro_files::serve on a stream with three routes and a 404 document: two mounts served from a tree in memory, and the application's own; the input is the peer's bytes, requests or not	every path the service opens is under its mount's root, with no .. component and no \, as it promises its file system; every answer written; nothing panics
proxy	npro-proxy between a client and an origin the harness plays, on streams with 256 byte buffers: the input is the client's bytes then the origin's, split where the control byte says, the origin's connect failed when it says so, the client's output written a few bytes at a time so the holds come	a request taken on is answered; the buffers' bounds hold (their types see to it); nothing panics
cgi-head	npro_cgi::Head, a script's answer's head: the input, behind a control byte choosing the pieces, is what the script wrote, fed whole and in pieces	both come to the same: taken or refused alike, the head's end at the same byte, no more than 2K taken, the same status, length, type and headers; the payload's start left exactly where the blank line ended
certdist-update	npro_certdist::Update, the distribution server's message: the input, behind a control byte choosing the pieces, fed whole and in pieces into small storage, so the caps are reached	both come to the same: taken or refused alike, the message's end at the same byte, the same chain, key and name held, each plausible PEM, a pair with neither side empty; reset, nothing is kept

A target that splits its input to feed it in pieces takes the split from the input's first byte, so libFuzzer explores the split like the rest of the input.

What the h1 targets cannot see, a verdict that is wrong the same way whole and in pieces, is what crates/npro-test/tests/h1_c.rs checks: npro's parser against C's over thousands of heads (its README).

Where things are

• crates/npro-fuzz: the harnesses, one safe fn(&[u8]) per target, and Target, which names them. Part of the main workspace, with no dependencies outside it. Its tests/smoke.rs is the smoke tests.
• fuzz/: the libFuzzer targets, one line each, which hand libFuzzer's input to a harness. A workspace of its own, with its own Cargo.lock and deny.toml, so what fuzzing needs to build (libfuzzer-sys and the crates that compile libFuzzer, see dependencies.md) never enters the main workspace.
• fuzz/seeds/<target>/: the hand-made seeds each corpus starts from (its README). transcript also starts from the vendored transcripts.
• scripts/fuzz.sh: builds the targets and runs libFuzzer over them, by hand or under sai. It follows the C tree's fuzz/run.sh.

Running it by hand

It needs nightly, cargo-fuzz, cargo-deny, a C++ compiler for libFuzzer's runtime, and ideally llvm-symbolizer for readable reports:

rustup toolchain install nightly --profile minimal
cargo install --locked cargo-fuzz cargo-deny

Then:

scripts/fuzz.sh                 # 60s on every target
scripts/fuzz.sh 600 utf8        # 10 minutes on utf8
CORPUS=~/npro-corpus scripts/fuzz.sh 3600

It checks the fuzz workspace with cargo deny before building anything, and builds with AddressSanitizer and debug assertions. FUZZ_OPTS passes more flags to libFuzzer: FUZZ_OPTS=-verbosity=0 leaves out its line for every new unit and its periodic status, keeping the setup, the final stats and any report. Builds and corpora go under target/fuzz/ (or $CARGO_TARGET_DIR/fuzz/) unless BUILD and CORPUS say otherwise.

When it finds something

A finding is an input that made a target panic: the code under test panicked, or disagreed with the oracle. libFuzzer keeps it as target/fuzz/artifacts/crash-<sha1>, and the run ends listing it. To replay it under the fuzz build:

target/fuzz/x86_64-unknown-linux-gnu/release/utf8 target/fuzz/artifacts/crash-...

The panic message says what disagreed, eg, utf8: fed a byte at a time it ends Complete, core says InvalidAt(1). The same input reproduces it in a plain test too, which is where the fix's regression test belongs: npro_fuzz::Target::Utf8.run(&input).

Under sai

The fuzz configuration in .sai.json runs scripts/sai.sh fuzz 60 on the C tree's fuzz builder, fuzz-linux-debian13/x86_64-amd/gcc, with "idle": 2 and "pool": "fuzz". As in C's fuzz job, libFuzzer runs there with -verbosity=0, so the log stays a few lines per target; a FUZZ_OPTS set on the builder replaces it. sai.md says what that builder needs. sai gives the script:

• SAI_POOL_DIR, the fuzz pool, which sai keeps synced between every builder fuzzing npro. The corpora live there as corpus-<target>/, named by SHA-1 as sai expects, so coverage keeps advancing across jobs and builders. Pools belong to a repo: npro's fuzz pool is not the C tree's.
• SAI_POOL_KNOWN, the reproducers of every bug found so far. The CI run replays them before fuzzing, and fails while any still crashes. For one that no longer does, it tells sai-server, which notes at which commit.
• SAI_POOL_FINDINGS, where findings go: the input and libFuzzer's report, which sai-server groups into bugs and shows only to admins. A finding can be an unfixed security bug, and job logs are public, so under sai the log gets only how each run went.
• SAI_IDLE_SECS, in an idle task: the slice's length, build included. The time after the build is shared by as many targets as can each have two minutes, taking turns across slices. An idle slice reports what it finds, but does not fail.

Adding a target

1. A harness in crates/npro-fuzz/src/targets.rs: what it drives, and the oracle it checks against, reporting a disagreement with finding(). It must not panic for any other reason.
2. A Target variant, its name, and its arm in Target::run.
3. A smoke test in crates/npro-fuzz/tests/smoke.rs, drawing inputs that reach the code, which random bytes often do not.
4. Its seeds in fuzz/seeds/<name>/: for a protocol parser, C's corpus for it, copied with where it came from.
5. A [[bin]] in fuzz/Cargo.toml and its one-line fuzz/fuzz_targets/<name>.rs.

scripts/fuzz.sh and sai pick it up from cargo fuzz list. A target that needs a new crate is a dependency decision like any other (dependencies.md).

Not done yet

• sai-server's grouping of Rust findings. sai-server tells bugs apart by the first three frames of the report's stack that are in the code under test. It skips C's runtime and libFuzzer's frames, but not Rust's: every panic's stack starts with pthread_kill, std::sys::pal::unix::abort_internal and std::process::abort, so every finding in a target is grouped as one bug. The frames that tell findings apart come after core::panicking::panic_fmt. sai-server needs to skip frames in std::, core::, alloc:: (and <std::, <core::, <alloc:: impls), libfuzzer_sys::, pthread_kill, and npro_fuzz::targets::finding, and to stop at rust_fuzzer_test_input.
• Coverage reports (cargo fuzz coverage) are not wired up.
• Corpus minimizing (-merge=1, and sai's pool replace) is not done; the corpora only grow.