docs: query-performance guide#212
Draft
coderdan wants to merge 3 commits into
Draft
Conversation
New reference page that consolidates the practical advice for getting fast queries out of EQL-encrypted columns. Centred on the two ingredients the 2.3 work surfaced: functional indexes and operator inlining. Sections: 1. Why functional indexes — small leaves, no superuser, structural planner match. 2. Operator inlining mechanics — the four conditions PG checks, the syntactic match rule, the transitive nature of the chain, EXPLAIN-based verification. 3. Natural / extractor / hybrid query forms — when each is the right default, with examples for each shape. 4. ORDER BY: the sort-key trap — three-shape contrast (natural / hybrid / fully extractor) with the 100k benchmark numbers as evidence. 5. Equality and GROUP BY / DISTINCT — including the ~425× speedup from GROUP BY on the inlined extractor vs the natural form's plpgsql hash_encrypted path. 6. LIKE / ILIKE — bloom filter recipe + the case-sensitivity caveat. 7. JSONB containment and ste_vec — per-selector vs all-selector field recipes; when to use jsonb_path_query vs hmac_256(col, selector). 8. Common pitfalls — index-creation ordering, missing ANALYZE, stale opclass indexes, pinned search_path, the natural-form ORDER BY expectation, range queries on non-Block-ORE columns. 9. Diagnosing with EXPLAIN — what to look for, what to do when the plan is wrong. Linked from docs/README.md under Reference, alongside database-indexes.md. Placement can move to a guides/ directory if we grow more action-oriented content later.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…ecipe Rewrites §5 to lead with the extractor form (`GROUP BY eql_v2.hmac_256(col)`) as the only recipe that scales, and explains the planner trap that makes the natural form (`GROUP BY col`) degrade pathologically on real-world tables. What changed in the underlying mechanics, and why the doc needs updating: * `eql_v2.hash_encrypted` was flipped from plpgsql to inlinable SQL in 2.3 (the discriminator backing the natural-form `GROUP BY`). That eliminated the "per-row plpgsql call cost" framing the previous version of this section leaned on — but the natural form is still slow at scale, and the reason is more interesting and more important to surface than the inlining detail. * The real bottleneck is the planner's HashAggregate-vs-GroupAggregate choice, which is driven by estimated hash-table size against `work_mem`. The natural form's key is the full ~1-2 KB encrypted payload; at 100k rows the estimate is 100-200 MB, way over the default 4 MB `work_mem`, so the planner refuses HashAggregate and falls back to GroupAggregate + sort. The extractor's group key is a 32-byte HMAC and fits trivially — HashAggregate every time, no `work_mem` tuning needed. Bench numbers measured on the cipherstash/benches setup post-2.3 with all three operator-inlining PRs in place (#205, #211, hash_encrypted): 100k natural, default 4 MB work_mem: ~29 s (GroupAggregate + Sort) 100k natural, 256 MB work_mem: ~780 ms (HashAggregate) 100k extractor: ~80 ms (HashAggregate, default) 1M natural, 512 MB work_mem: ~234 s (GroupAggregate) Also tightens the §8 pitfall on `hash_encrypted` — pre-2.3 it raised loudly when used against a column without `hm`, which made `GROUP BY` a natural smoke test for misconfig. Post-2.3 it falls back to data-hashing to keep the aggregate from degrading to O(N^2), so the runtime smoke signal is gone. Audit at config time via `eql_v2.has_hmac_256(col)`. Adds a new §8 pitfall calling out the natural-form `GROUP BY` trap specifically — frequent enough that it warrants its own bullet.
…not bounded Updates §4's empirical numbers to include the 1M data point from the cipherstash/benches suite. The previous framing (Top-N cost is "real but bounded — milliseconds, not seconds … if you can live with that, the natural form keeps the query readable") doesn't survive past 100k. At 1M rows on `string_encrypted_1000000` / `integer_encrypted_1000000` with a ~0.5 selectivity predicate, the natural-form Top-N takes ~8.8 s while the hybrid and fully-extractor forms stay around 1 ms. The Sort step just keeps scaling with the post-WHERE row count. Same advice surface as §5 on GROUP BY: there's a documented better recipe, the natural form's plan choice is the trap, write the extractor form. The §5 framing now reads in parallel with this one — both sections land on "the extractor form is what scales; the natural form is for toy-size data". Also flags that the bench suite drops the natural-form `range_lt_ordered_10` scenario and the redundant fully-extractor `range_lt_ore_ordered_10` scenario, keeping only the hybrid recipe. Companion bench change shipping on the `feat/json-benches-rebased` branch in cipherstash/benches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A new reference page that consolidates the practical advice for getting fast queries out of EQL-encrypted columns. The 2.3 operator-inlining work surfaced enough subtleties — natural vs extractor forms, the
ORDER BYsort-key trap, the inlining chain through helpers, when a functional index will and won't match — that they warrant a dedicated page rather than being scattered acrossdatabase-indexes.md, the upgrade notes, and tribal knowledge.Outline
ORDER BY: the sort-key trap — three-shape contrast (natural / hybrid / fully extractor) with the 100k benchmark numbers as evidence (885 ms vs 1.4 ms).GROUP BY/DISTINCT— including the ~425× speedup fromGROUP BYon the inlined extractor vs the natural form's plpgsqlhash_encryptedpath.LIKE/ILIKE— bloom-filter recipe and the case-sensitivity caveat.ANALYZE, stale opclass indexes, pinnedsearch_path, range queries on non-Block-ORE.EXPLAIN— what to look for, what to do when the plan is wrong.Placement
Landed at
docs/reference/query-performance.mdand linked fromdocs/README.mdnext todatabase-indexes.md. Happy to relocate to a newdocs/guides/directory if we want to start separating action-oriented content from reference. Not blocking on that for this PR — easier to iterate on the content first.Related
Depends on the contract established by:
</<=/>/>=). U-005 callouts in the guide reference that PR's upgrade note.=inlining (perf: flip eql_v2_encrypted infix operator implementations to inlinable SQL (RFC Phase 1) #193, perf: flip eql_v2_encrypted infix operator implementations to inlinable SQL (#193) #196) — already referenced via U-002.Test plan
reference/or move to a newguides/.Draft because the content is a first pass — happy to iterate.