Skip to content

Fix get_existing_candles timeout on large databases - #621

Open
axlnur wants to merge 1 commit into
jesse-ai:masterfrom
axlnur:fix/get-existing-candles-skip-scan
Open

axlnur wants to merge 1 commit into
jesse-ai:masterfrom
axlnur:fix/get-existing-candles-skip-scan

Conversation

@axlnur

@axlnur axlnur commented Aug 20, 2026

Copy link
Copy Markdown

Summary

  • Replace the N+1 queries in get_existing_candles() with one recursive-CTE query that emulates an index skip scan over the (exchange, symbol, timeframe, timestamp) compound index.
  • Preserve the exact response shape (exchange, symbol, start_date, end_date) and date semantics; results are now deterministically ordered by exchange and symbol.
  • Add regression tests (in-memory SQLite): cross-timeframe min/max aggregation, empty database, and a single-SQL-statement assertion.

Root cause

get_existing_candles() selects the distinct exchange/symbol pairs and then issues two ordered queries per pair. The DISTINCT alone reads every candle row, and the per-pair queries cannot use the compound index for ORDER BY timestamp because timeframe sits between symbol and timestamp in the key. On large databases the endpoint takes minutes, so the MCP client's fixed 10-second timeout always fires (#618).

Collapsing everything into a single GROUP BY aggregate (the approach in #616) removes the N+1 pattern but still reads all N rows in one pass. PostgreSQL has no native index skip scan (before v18), but a recursive CTE can emulate one: each recursion step jumps directly to the next (exchange, symbol, timeframe) group with a row-value probe of the compound index, then the first/last timestamps are read with two more index probes per group. Cost drops from O(N) to O(groups x log N).

Benchmarks

Production database: PostgreSQL 14.23, candle table with 663,146,368 rows (~176 GB), 536 exchange/symbol pairs.

implementation time
current N+1 loop 15-20 min
single GROUP BY aggregate (#616 approach) 3 min 11.6 s
recursive CTE skip scan (this PR) 25-40 ms warm cache, ~1.0 s cold cache

EXPLAIN ANALYZE confirms every step is an index-only probe: Index Cond: (ROW(exchange, symbol, timeframe) > ROW(...)) with Limit 1, and the min/max lookups become forward/backward index-only scans.

Validation

  • Result set verified byte-identical (diff of 536 rows) to a GROUP BY exchange, symbol full-scan aggregate on the same 663M-row database.
  • End-to-end POST /candles/existing returns in ~60 ms against that database, and the MCP get_existing_candles tool now completes well within its 10-second timeout (previously it always failed).
  • The raw SQL is portable between PostgreSQL and SQLite and runs through the model's bound database handle, so the new tests exercise the real production code path against in-memory SQLite (row-value comparisons require SQLite >= 3.15; every Python >= 3.10 build ships a newer one).
  • Full non-slow suite passes in the salehmir/jesse:latest Docker environment with PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 (572 passed, 3 deselected, plus the 2 new tests).

Fixes #618

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

get_existing_candles times out on larger PostgreSQL databases because of N+1 queries

1 participant