closed #362 - #365
Merged
Merged
Conversation
Fix Cuckoo filter eviction-slot cycling that caused spurious :full errors The kick-eviction loop chose which slot to evict with a plain rem(fingerprint + kick_count, bucket_size), a linear function of both inputs. With only 2^fingerprint_size possible fingerprint values shared across a much larger item count, two colliding fingerprints that differed by a multiple of bucket_size could synchronize the kick sequence into a short, exactly-repeating cycle among a handful of buckets, exhausting max_kicks well below the table's designed ~95.5% load factor. Confirmed via direct source-level tracing: 500,000 sequential "session_N" keys into Cuckoo.new(capacity: 500_000) hit a 4-step cycle between 3 buckets on both the Pure and Rust backends. Fixed by routing the evicted fingerprint and kick count through the existing fingerprint-mixing hash before reducing mod bucket_size, keeping slot choice fully deterministic while eliminating the arithmetic periodicity. Ported identically to both backends; Pure and Rust now produce byte-identical serialized state for identical input. Random-key load factor confirmed at ~95% on both backends post-fix, matching the documented target. Since the eviction-slot formula changed, any Cuckoo build that exercises kick-eviction now produces a different (non-cyclic) final state than before -- a behavior change, not a binary format change; old serialized sketches still decode and work. closed #364
…d add CQF overflow detection
Quotient.member?/2 and CQF's member?/estimate_count/2/delete/2 always
decoded the entire slot table into a tuple before reading the one run
they needed, regardless of :backend (no per-item Rust NIF exists for
these). 300,000 sequential Quotient.member?/2 calls against a 524,288-
slot filter took ~40 minutes; the same calls now take ~0.13s. Fixed by
adding a lazy decode path that parses only the header and reads
individual slots on demand. Added Quotient.member_many?/2 for batch
checks.
Separately, CQF silently dropped inserts once its table filled up,
in both backends -- the shared shift-right insertion machinery was
bounded to terminate but had no way to report failure back to the
caller. Fixed by threading a success/failure result through the
insertion chain: the Pure backend's immutability makes a failed
attempt's partial mutations automatically discarded, while the Rust
NIF (which mutates in place) now checks whether each insert will fit
before applying any mutation, including preparatory metadata changes,
to avoid leaving the table structurally inconsistent on failure.
CQF.put/2 and put_many/2 now return {:ok, cqf} / {:error, :full,
partial} mirroring Cuckoo, with put!/2 added for chaining and
update/2 raising FilterFullError. Quotient's equivalent internal fix
prevents a genuine hang on a 100%-full Pure-backend table but keeps
its existing silent-no-op public behavior, since Quotient's put API
wasn't part of this change.
Also fixed the CQF tutorial's q sizing: q must budget for total
occurrence count (each duplicate costs a physical slot), not distinct
key count. The tutorial's q: 18 (262,144 slots) was sized for 50,000
distinct keys when it needed capacity for 1,000,000 occurrences,
making put_many take over an hour; q: 21 (2,097,152 slots, ~2x
headroom) finishes the same call in under a second.
Summary:
1. Livebook q sizing — bumped cqf.livemd's q: 18 → q: 21 (2,097,152 slots) for the 1,000,000-occurrence dataset; added a "Sizing: q must budget for total occurrences" section.
2. Single-item Rust bypass — Quotient.member?/2/delete/2 and CQF.member?/2/estimate_count/2/delete/2 now use a lazy binary-offset decode instead of materializing the whole slot table per call. New Quotient.member_many?/2 batch API. Confirmed: 300,000 Quotient.member? calls went from ~40 minutes to ~0.13s; CQF member? went from ~190ms/call to ~1.2μs/call.
3. CQF overflow detection — put/2/put_many/2 now return {:ok, cqf} / {:error, :full, partial} mirroring Cuckoo, with put!/2 added for chaining and update/2/update_many/2 raising FilterFullError. This required check-before-mutate restructuring in the Rust NIF (since it mutates in place, unlike Pure's immutable discard-on-failure) to avoid leaving the table structurally corrupted on a failed insert. Also fixed a real hang risk in Quotient's Pure backend (an unbounded shift loop that could spin forever on a 100%-full table). Verified Pure/Rust byte-identical parity at the overflow boundary and zero corruption of successfully-inserted items.
Introduce ExDataSketch.Config, a small helper that reads a single
`config :ex_data_sketch, defaults: [family: [...]]` key and merges it
under the caller's explicit opts (explicit opts always win). Wire it
into `new/1` (and `build/2` for XorFilter) across all 15 sketch
families, so options like backend selection or default `p`/`q`/`r`/
`capacity` can be set once in Application config instead of passed at
every call site.
Fix FilterChain performance and API:
- Add put_many/2, batching inserts through each stage's own
put_many/2 (Rust-accelerated where available) instead of looping
single-item put/2, which forced every stage onto the Pure backend
and made large inserts O(state size) per item. update_many/2 now
delegates to put_many/2 instead of reducing over update/2.
- Simplify delete/2 to return a bare t() instead of {:ok, t()} -- it
never had a real error case to report (missing items are absorbed
as no-ops per stage, unsupported stages raise upfront).
- Add :put_many to capabilities/0.
Fix livebooks/sketches/.verify_extract.exs, which extracted every
livebook's cells for automated verification but silently skipped the
Mix.install cell entirely, so any config: block a livebook set (e.g.
backend: Rust) was never applied during verification. It now parses
the Mix.install call's config: option via Code.string_to_quoted/1 and
applies it before running the cell blocks.
Add a "No NIF acceleration" note to REQ's moduledoc, matching the
existing MisraGries note -- ExDataSketch.Backend.Rust's req_* functions
are a thin pass-through to Pure, so :backend has no effect on REQ.
Add a Configuration section to guides/usage_guide.md indexing every
config :ex_data_sketch key (backend, defaults, dirty_thresholds,
storage, persistence_backends, telemetry_enabled, telemetry,
integrations), and fix the ULL Options row to point at ULL's own
Precision Range section, since ULL's p<=26 is a hard limit unlike
HLL's.
Update all sketch and integration livebooks' Mix.install cells to
nest config under ex_data_sketch: [...] (Mix.install's config: option
is keyed by OTP application, unlike the config :app, ... macro), and
set backend: Rust plus integrations: [opentelemetry: false] where
appropriate. Also:
- cqf.livemd: fix q-sizing example (q: 18 -> 21) and put/put_many call
shapes, add "Sizing" and "What full looks like" sections.
- ddsketch.livemd: fix a misleading rounded-to-zero min_value display.
- filter_chain.livemd: rewrite the Inserting section around
put_many/2, update the delete/2 example for its new bare return,
and rework the XorFilter hybrid-chain section to explain why newly
inserted items stay absent from the whole-chain query.
- hll.livemd, ull.livemd: add "Why 4..26?" sections explaining the
shared p>=4 floor and why HLL's p<=26 ceiling is a practical choice
but ULL's is a hard limit tied to a fixed-size estimator table.
- frequent_items.livemd: bump the basic-usage example from k: 20 to
k: 100.
- cms.livemd: restore the floating "~> 0.10" Mix.install pin.
Add regression tests for FilterChain.put_many/2 (including the
{:error, :full, partial} case and a performance bound) and for
ExDataSketch.Config (direct merge_defaults/2 coverage plus
integration with several families' new/1).
Remove livebooks/sketching_one_billion_rows.livemd.
…laced Caught by the new FilterChain.put_many/2 performance-regression test flaking on CI. The bitset-to-tuple "O(1) update" trick doesn't survive Enum.reduce/3's closure-based iteration -- each put_elem/3 call was a full tuple copy, making a batch cost O(n * hash_count * bit_array_size) instead of the intended O(n * hash_count). Fixed by collecting bit positions into a map and applying them in one linear pass: a 1,000-item batch at 500,000 capacity drops from 1.8-2.6s to ~80ms.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix Cuckoo filter eviction-slot cycling that caused spurious :full errors
The kick-eviction loop chose which slot to evict with a plain rem(fingerprint + kick_count, bucket_size), a linear function of both inputs. With only 2^fingerprint_size possible fingerprint values shared across a much larger item count, two colliding fingerprints that differed by a multiple of bucket_size could synchronize the kick sequence into a short, exactly-repeating cycle among a handful of buckets, exhausting max_kicks well below the table's designed ~95.5% load factor. Confirmed via direct source-level tracing: 500,000 sequential "session_N" keys into Cuckoo.new(capacity: 500_000) hit a 4-step cycle between 3 buckets on both the Pure and Rust backends.
Fixed by routing the evicted fingerprint and kick count through the existing fingerprint-mixing hash before reducing mod bucket_size, keeping slot choice fully deterministic while eliminating the arithmetic periodicity. Ported identically to both backends; Pure and Rust now produce byte-identical serialized state for identical input. Random-key load factor confirmed at ~95% on both backends post-fix, matching the documented target.
Since the eviction-slot formula changed, any Cuckoo build that exercises kick-eviction now produces a different (non-cyclic) final state than before -- a behavior change, not a binary format change; old serialized sketches still decode and work.
closed #364