Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
get_existing_candles()with one recursive-CTE query that emulates an index skip scan over the(exchange, symbol, timeframe, timestamp)compound index.exchange,symbol,start_date,end_date) and date semantics; results are now deterministically ordered by exchange and symbol.Root cause
get_existing_candles()selects the distinct exchange/symbol pairs and then issues two ordered queries per pair. TheDISTINCTalone reads every candle row, and the per-pair queries cannot use the compound index forORDER BY timestampbecausetimeframesits betweensymbolandtimestampin the key. On large databases the endpoint takes minutes, so the MCP client's fixed 10-second timeout always fires (#618).Collapsing everything into a single
GROUP BYaggregate (the approach in #616) removes the N+1 pattern but still reads all N rows in one pass. PostgreSQL has no native index skip scan (before v18), but a recursive CTE can emulate one: each recursion step jumps directly to the next(exchange, symbol, timeframe)group with a row-value probe of the compound index, then the first/last timestamps are read with two more index probes per group. Cost drops from O(N) to O(groups x log N).Benchmarks
Production database: PostgreSQL 14.23,
candletable with 663,146,368 rows (~176 GB), 536 exchange/symbol pairs.EXPLAIN ANALYZEconfirms every step is an index-only probe:Index Cond: (ROW(exchange, symbol, timeframe) > ROW(...))withLimit 1, and the min/max lookups become forward/backward index-only scans.Validation
GROUP BY exchange, symbolfull-scan aggregate on the same 663M-row database.POST /candles/existingreturns in ~60 ms against that database, and the MCPget_existing_candlestool now completes well within its 10-second timeout (previously it always failed).salehmir/jesse:latestDocker environment withPYTEST_DISABLE_PLUGIN_AUTOLOAD=1(572 passed, 3 deselected, plus the 2 new tests).Fixes #618