What happens
I have a dashboard with a few timeseries_count widgets (grouped by a log attribute) and a log table, watching a fairly noisy worker service. At the 24h preset everything renders fine. Switch to "Last 7d" and every widget dies — all the /explore/series and /explore/logs requests come back as plain-text Internal Server Error after about 10 seconds.
Why
The ClickHouse client in apps/api/src/index.ts is created with request_timeout: 10_000, but countSeries() in apps/api/src/mcp/clickhouse.ts does a GROUP BY over every row in the window — its LIMIT 10000 bounds the output, not the scan — and the filters/group keys are map lookups, so ClickHouse reads the attribute maps for the whole range. On a 7-day window with a few million rows that takes longer than 10s, the client aborts, the throw bubbles up uncaught, and Hono returns a bare 500.
Further, a dashboard fires all its widget queries at once, so they all blow the budget together; the frontend renders the failure as "no data," indistinguishable from a quiet service; the aborted request doesn't cancel the server-side query, so ClickHouse finishes each abandoned scan anyway.
This exact failure was already fixed once, but only for the attribute-keys dropdown — countSeries never got the same treatment:
|
// Discovering which attribute keys exist only needs a representative sample of |
|
// rows, not every row in the window. High-volume projects produce millions of |
|
// spans/logs per hour, and reading the full ResourceAttributes/SpanAttributes |
|
// map columns across all of them took 15-30s — past the 10s ClickHouse |
|
// request_timeout — so the explore filter dropdown 500'd. Capping the rows each |
|
// scan reads before the arrayJoin/group keeps the query ~1s while still |
|
// surfacing effectively every key: ClickHouse reads parts in parallel, so the |
|
// cap samples across the window rather than just the head. Counts become |
|
// approximate, which is fine for ordering the dropdown. Low-volume projects read |
|
// fewer rows than the cap and stay exact. |
|
const ATTRIBUTE_KEY_SCAN_ROW_CAP = 1_000_000; |
Suggested fix
Apply the same row-cap-in-subquery trick to countSeries: a SERIES_SCAN_ROW_CAP as a LIMIT on the inner filtered scan, before the bucket/group aggregation. Wide ranges get sampled-but-fast charts instead of nothing; small projects never hit the cap and stay exact. Ideally also return sampled: true when the cap is hit so the UI can label the chart — a bare LIMIT isn't a uniform sample, so the numbers shouldn't be silently trusted. (queryLogs shouldn't be capped this way — a pre-limit would break its "latest N" semantics.)
Longer term
The real fix for wide ranges is probably pre-aggregation. Dashboard widgets are saved configs, so the heavy recurring queries are known ahead of time, which makes targeted rollups feasible — but that's a much bigger change, so probably too early for it.
Happy to throw up a PR if this looks good by adding the row-cap trick.
What happens
I have a dashboard with a few timeseries_count widgets (grouped by a log attribute) and a log table, watching a fairly noisy worker service. At the 24h preset everything renders fine. Switch to "Last 7d" and every widget dies — all the /explore/series and /explore/logs requests come back as plain-text Internal Server Error after about 10 seconds.
Why
The ClickHouse client in apps/api/src/index.ts is created with request_timeout: 10_000, but countSeries() in apps/api/src/mcp/clickhouse.ts does a GROUP BY over every row in the window — its LIMIT 10000 bounds the output, not the scan — and the filters/group keys are map lookups, so ClickHouse reads the attribute maps for the whole range. On a 7-day window with a few million rows that takes longer than 10s, the client aborts, the throw bubbles up uncaught, and Hono returns a bare 500.
Further, a dashboard fires all its widget queries at once, so they all blow the budget together; the frontend renders the failure as "no data," indistinguishable from a quiet service; the aborted request doesn't cancel the server-side query, so ClickHouse finishes each abandoned scan anyway.
This exact failure was already fixed once, but only for the attribute-keys dropdown — countSeries never got the same treatment:
superlog/apps/api/src/mcp/clickhouse.ts
Lines 573 to 583 in ef9ec92
Suggested fix
Apply the same row-cap-in-subquery trick to countSeries: a SERIES_SCAN_ROW_CAP as a LIMIT on the inner filtered scan, before the bucket/group aggregation. Wide ranges get sampled-but-fast charts instead of nothing; small projects never hit the cap and stay exact. Ideally also return sampled: true when the cap is hit so the UI can label the chart — a bare LIMIT isn't a uniform sample, so the numbers shouldn't be silently trusted. (queryLogs shouldn't be capped this way — a pre-limit would break its "latest N" semantics.)
Longer term
The real fix for wide ranges is probably pre-aggregation. Dashboard widgets are saved configs, so the heavy recurring queries are known ahead of time, which makes targeted rollups feasible — but that's a much bigger change, so probably too early for it.
Happy to throw up a PR if this looks good by adding the row-cap trick.