[Store] Add multi-level parallel read via shared ThreadPool

[NUABO]

Description

• Add MC_BATCH_LOAD_THREADS env var for bucket-level parallelism. When > 1 and multiple buckets, dispatch each bucket to a thread.

• Add MC_BUCKET_READ_THREADS env var for intra-bucket POSIX read parallelism. When > 1, split read plans into chunks and dispatch to threads sharing the same fd (pread is thread-safe).

  MC_BATCH_LOAD_THREADS — bucket-level pool (independent files)
  MC_BUCKET_READ_THREADS — intra-bucket pool (POSIX pread chunking)
  • Both 0 (default): original sequential path, zero change.
  • Only batch > 1: buckets dispatched to bucket pool; URing uses
  batch_read() (one io_uring submission, queue
  depth 32), POSIX stays sequential per-bucket.
  • Only bucket > 1: buckets sequential, POSIX plans split across
  intra-bucket pool. URing uses batch_read().
  • Both > 1: bucket pool dispatches buckets, within each
  POSIX plans further split across intra-bucket
  pool — true 2-level concurrency.
  • URing batch_read() is enabled when either env var > 0

• Extract UringBatchReadBucket (URing alignment + batch_read + zero-copy) and IntraBucketReadChunked (POSIX chunk-dispatch) to eliminate duplication

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Test commands:

# Example: bash scripts/run_ci_test.sh

Test results:

• Unit tests pass
• Integration tests pass (if applicable)
• Manual testing done (describe below)

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit run --all-files and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)

[LujhCoconut]

Thanks for the contribution. Before diving into the design, two things need to be addressed first.

1. This PR does not compile. BucketStorageBackend(const FileStorageConfig&, const BucketBackendConfig&) ends up with two identical out-of-line definitions in storage_backend.cpp (the pre-existing one and the one added by this PR), which is a redefinition error. Note the new one also drops the initialization of storage_path_ and the aligned_io_buffer_ allocation — the thread-count/pool setup should be merged into the existing constructor instead.

A ready-for-review PR should at least build locally — please make sure the code compiles and passes basic tests (e.g. a BatchLoad smoke test with the new env vars enabled) before submitting, and also run the format checks (./scripts/code_format.sh) before (re)submitting.

2. The intra-bucket path is incompatible with io_uring + O_DIRECT. With use_uring enabled, bucket files are opened with O_DIRECT, but the intra-bucket chunked path issues vector_read() at plan.offset + plan.key_size, which is generally not 4096-aligned, so every read fails with EINVAL. And even setting correctness aside, I don't think this path is necessary on the uring backend: N threads each issuing blocking per-key reads only reach aggregate queue depth N at N times the syscall/thread cost, whereas a single batch_read() submission reaches queue depth 32 from one thread — threads are the wrong tool for parallelism within one file when io_uring is available.

The design would hold up much better as: within a bucket, use batch_read() on the uring path and fall back to threaded pread only for the non-uring (buffered POSIX) backend; bucket-level threading across independent files can stay as-is. This combination is worth thinking through carefully — happy to discuss before you invest more time in the current direction.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 63.82979% with 85 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
mooncake-store/src/storage_backend.cpp	54.78%	85 Missing ⚠️

📢 Thoughts on this report? Let us know!

[NUABO]

Thanks for the contribution. Before diving into the design, two things need to be addressed first.

1. This PR does not compile. BucketStorageBackend(const FileStorageConfig&, const BucketBackendConfig&) ends up with two identical out-of-line definitions in storage_backend.cpp (the pre-existing one and the one added by this PR), which is a redefinition error. Note the new one also drops the initialization of storage_path_ and the aligned_io_buffer_ allocation — the thread-count/pool setup should be merged into the existing constructor instead.

A ready-for-review PR should at least build locally — please make sure the code compiles and passes basic tests (e.g. a BatchLoad smoke test with the new env vars enabled) before submitting, and also run the format checks (./scripts/code_format.sh) before (re)submitting.

2. The intra-bucket path is incompatible with io_uring + O_DIRECT. With use_uring enabled, bucket files are opened with O_DIRECT, but the intra-bucket chunked path issues vector_read() at plan.offset + plan.key_size, which is generally not 4096-aligned, so every read fails with EINVAL. And even setting correctness aside, I don't think this path is necessary on the uring backend: N threads each issuing blocking per-key reads only reach aggregate queue depth N at N times the syscall/thread cost, whereas a single batch_read() submission reaches queue depth 32 from one thread — threads are the wrong tool for parallelism within one file when io_uring is available.

The design would hold up much better as: within a bucket, use batch_read() on the uring path and fall back to threaded pread only for the non-uring (buffered POSIX) backend; bucket-level threading across independent files can stay as-is. This combination is worth thinking through carefully — happy to discuss before you invest more time in the current direction.

hi @LujhCoconut thanks for thorough review, both points were spot on.

Compilation and constructor: The duplicate constructor has been removed.

Intra-bucket path and io_uring: You were right — threading within a bucket made no sense for the URing backend. The revised design follows your suggestion:

• URing files within a bucket: always use batch_read() — a single io_uring submission with queue depth up to 32, regardless of thread-pool settings. No intra-bucket threads touch URing files.
• POSIX (buffered) files within a bucket: optional intra-bucket parallelism via a dedicated ThreadPool, gated on MC_BUCKET_READ_THREADS > 1. Plans are split into chunks and dispatched; pread is thread-safe
  on the same fd.
• Bucket-level parallelism: unchanged — a separate ThreadPool gated on MC_BATCH_LOAD_THREADS > 1 dispatches independent bucket files across threads.

The two pools are independent, so when both env vars are set bucket-level dispatch and intra-bucket POSIX chunking compose into true 2-level concurrency. URing files are never dispatched to either pool for
intra-bucket reads — they always go through batch_read().

When both env vars are 0 (the default), the code falls through to the original sequential path with zero change.

Happy to iterate further if anything still looks off.