feat: implement RustCrypto traits for signatures and KEMs

[0rlych1kk4]

Summary

This PR continues the work started by @SebastianFaller in #295 and updates it for the current main branch.

It adds RustCrypto trait support for:

• KEM encapsulation and decapsulation
• Signature signing and verification

The original implementation commit was preserved, so the authorship remains attributed to @SebastianFaller.

Changes

• add Encapsulator and Decapsulator implementations for the RustCrypto KEM traits;
• add Signer and Verifier implementations for the RustCrypto signature traits;
• keep the usage examples as doctests;
• apply formatting fixes;
• resolve Clippy warnings in the added adapters.

Validation

Tested with both stable and beta Rust:

• cargo fmt --all -- --check
• cargo check --workspace --all-features
• cargo test -p oqs --all-features
• cargo test -p oqs --doc --all-features
• cargo clippy -p oqs --all-targets --all-features --no-deps -- -D warnings
• cargo +beta check --workspace --all-features
• cargo +beta test -p oqs --all-features
• cargo +beta test -p oqs --doc --all-features

Results:

• 615 unit tests passed
• 7 doctests passed

The full workspace Clippy run is still blocked by the existing
clippy::unnecessary_map_or warning in oqs-sys/build.rs, which is unrelated
to this change.

Continues #295.
Closes #137.

[0rlych1kk4]

@dstebila , this continues #295 with @SebastianFaller permission. I preserved the original authorship, rebased the work onto the current main branch, and verified it on stable and beta Rust. Whenever convenient, I would appreciate your review.

[0rlych1kk4]

@dstebila ,I reproduced the macOS-latest, nightly, true failure on the current upstream/main branch using a clean worktree.
After running:
git submodule update --init --recursive
git submodule update --remote
cargo build
the build fails with the same 15 missing HQC and SPHINCS identifier errors.
The true matrix variant updates liboqs from the pinned revision 21b3f8b0 to 5a1a854b, and the generated bindings from that newer revision are no longer compatible with the identifiers currently used by liboqs-rust.
This appears unrelated to the RustCrypto trait changes in this PR. The builds using the pinned submodule revision pass.

[dstebila]

@dstebila ,I reproduced the macOS-latest, nightly, true failure on the current upstream/main branch using a clean worktree. After running: git submodule update --init --recursive git submodule update --remote cargo build the build fails with the same 15 missing HQC and SPHINCS identifier errors. The true matrix variant updates liboqs from the pinned revision 21b3f8b0 to 5a1a854b, and the generated bindings from that newer revision are no longer compatible with the identifiers currently used by liboqs-rust. This appears unrelated to the RustCrypto trait changes in this PR. The builds using the pinned submodule revision pass.

Hi @0rlych1kk4 thanks for working on an update. There have been algorithm identifier changes since the Rust wrapper was last updated (removal of SPHINCS+, changes involving HQC, plus some others). So those will need to be updated as well, which I think would probably be best done as a separate PR.

[0rlych1kk4]

@dstebila Thanks for confirming. The algorithm identifier updates are being handled separately in #309 , as discussed. This keeps the liboqs compatibility changes independent from the RustCrypto trait implementation in #310.

[0rlych1kk4]

@dstebila @baentsch Just to keep the two PRs cleanly separated, I’d prefer to get #309 merged first before rebasing and continuing work on #310. #309 is already approved and all 21 checks are passing. Once #309 lands, I can rebase #310 onto the updated main branch and address any remaining CI issues there.

[0rlych1kk4]

@dstebila @baentsch #310 is now rebased on current main with #309 included, and all 21 CI checks are passing. The previous macOS nightly failure is no longer reproducible. Happy to address any remaining review comments.

[baentsch]

One comment that sprung into my face glancing over the PR. I then asked a friendly AI to review more fully: Here's the verdict (remember, I'm a Rust noob, so please forgive any slop I'm unable to catch):

1. New dependencies are unconditional — they should be behind a feature flag.
   kem, signature, and rand_core are added as mandatory [dependencies]. Every downstream consumer of oqs now pays this compile/dependency cost and inherits version-conflict risk, even though RustCrypto interop is a niche convenience. This should be gated, e.g. a rustcrypto feature that turns on optional deps (kem = { ..., optional = true }, etc.) and #[cfg(feature = "rustcrypto")] on the impls.

2. no_std regression.
   The crate is #![cfg_attr(not(feature = "std"), no_std)]. rand_core is pulled in with the getrandom feature unconditionally, which breaks builds on no_std targets without getrandom support. Note the library itself doesn't need getrandom at all — it's only used by the doctest's OsRng. The library only needs rand_core for the CryptoRngCore bound in the trait signature. So getrandom should be a dev-dependency-only feature (or the doctest should avoid OsRng), and the whole thing gated per (1).

3. The provided RNG is silently ignored in encapsulate — footgun.
   liboqs uses its own internal/global RNG, so _csprng is discarded. This is noted only in the doctest comment, not on the struct/impl docs. A caller passing a seeded RNG (a common reason to reach for the trait) gets no determinism and no warning. This deserves a prominent doc note on Encapsulator/the impl, since it's a genuine semantic surprise relative to what the Encapsulate trait implies.

Minor

• sig.rs maps errors with Err(_) => Err(signature::Error::new()), discarding the source. signature::Error::from_source(e) (std) would preserve context. The KEM side keeps type Error = crate::Error, which is nicer — the asymmetry is unavoidable though, since Signer/Verifier fix the error type.
• The two KEM doctests (on Encapsulator and Decapsulator) are identical copy-paste; fine, just redundant.
• impl<'a> ... for Encapsulator<'a> etc. may trip clippy::needless_lifetimes on newer toolchains (they report clippy clean, so OK for now).

[baentsch]

Reference to pre-release looks suspicious

[0rlych1kk4]

I checked the stable kem 0.3 API. It has changed substantially from 0.3.0-pre.0; the current adapter would need to be reworked around the new Kem associated type, TryKeyInit/KeyExport, fixed-size array types, and the updated encapsulation/decapsulation traits. For this PR, I’m keeping the pre-release version but gating the RustCrypto dependencies behind a rustcrypto feature. I can handle the stable kem 0.3 migration separately if preferred.

[baentsch]

Understood. I still wonder whether this reference makes the whole (KEM) work in this PR useless: Is 0.3.0-pre.0 used by anyone?? If my guess is right that no-one uses it, then the point of this PR is pretty moot (regarding KEMs).

So what about retaining the sig part in this PR, dropping the KEM part and working for that (on 0.3.0) in a separate PR?