feat(runners): store-backed results, JSON driver, streaming fan-in

[ocervell]

Summary

Reworks how runner results are stored and read. Instead of carrying a runner's findings between Celery tasks in memory (the "chain payload"), findings live only in a store and every consumer streams them back through a run-scoped QueryEngine view. This is now the single, self-contained PR for the whole result-store change — it subsumes and supersedes the earlier stack:

• feat: add live JSON driver (per-runner report.json, concurrency-safe) #1299 (live JSON driver) — folded in, and upgraded from the O(N²) report.json rewriter to an append-only NDJSON store.
• fix(extractors): filter fan-in via QueryEngine so extractors don't OOM (RC#6) #1307 (fan-in via QueryEngine so extractors don't OOM) — folded in.
• feat(driver): add JSON driver for live report stream #1044 (previous JSON driver)
• perf(query): yield findings lazily in JSON backend #1023 (previous JSON stream attempt)

Both are closed in favour of this PR.

The model

• write-model — add_result fans each finding out to the active store driver(s) via on_item. Local runs default to the json/NDJSON driver (no backend needed); sqlite and mongodb are the DB stores.
• read-model — runner.results / findings / errors / … are lazy StreamViews over the store, scoped by the runner's {type}_id (minted once in Runner.__init__, inherited via context). Nothing is held in memory; a run's peak memory is its working set, not its result count.

Key changes

• NDJSON local driver — findings append to results.ndjson (O(1) per finding) instead of rewriting a whole report.json per finding (the old O(N²) hot path). report.json keeps info only; the query backend reads NDJSON last-wins with a legacy report.json fallback. orjson on the read/write paths.
• Streaming fan-in & exporters — the fan-in query and the csv/txt/markdown exporters stream from the store (cursor / batched iterate), so report generation stays flat instead of materializing N findings.
• Store is the sole source — removed the in-memory results fallback (self._results / context['results']); report info and read-model source errors/warnings from _type=error/warning findings; dropped errors/warnings from Runner.toDict().
• Run-scoping — extractor cascade and store reads are bound to the current run's report_dir / {type}_id, never the whole workspace (no cross-run leak, no whole-workspace scans mid-run).
• Read reduction + leaf metadata in memory — per-task store reads cut ~16 → ~4; leaf runners mirror their own findings_count/errors in memory so status never re-queries the store.
• Fix the on_interval backend throttle — last_updated_db was initialized to None and never stamped, so should_update() was always true and the runner's summary row was rewritten to the backend once per finding (1M redundant writes at 1M findings — the cause of the sqlite driver's super-linear DNF at scale). Now stamped, so it honors backend_update_frequency (5s). Helps every driver.

Results (memory + runtime)

Synthetic fan-in workflow, peak RSS + wall time, per driver:

	main (in-memory)	this PR · local	this PR · sqlite/mongodb
100k findings	745 MB	~118 MB	~110 MB
1M findings	~7.4 GB (OOM territory)	flat, bounded	flat, bounded

• main holds everything in RAM → O(N), climbs to OOM at 1M.
• feat: add live JSON driver (per-runner report.json, concurrency-safe) #1299's original local json driver was O(N²) → didn't finish 10k in 5 min.
• This PR: every driver is O(working-set) — flat memory across three orders of magnitude. The tradeoff is time (stores spend I/O to stay flat; in-memory is faster but unbounded).

Full matrix + charts: benchmark artifact (attached separately).

Testing

• Unit suite green, lint clean.
• Verified live (isolated worker): chunks run in parallel, store-only, no whole-workspace scans even at 100 chunks.

Known follow-ups (not in this PR)

• secator-api must migrate its runner.errors reads (rescan / auto-FIXED gate + health chore) to the findings query before this deploys (secator-ui already migrated).
• sqlite driver still writes the runner summary row per interval per-connection; batched finding writes (à la the never-merged mongodb bulk_write) are a further optional win, not required for O(working-set).

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added safer local JSON storage using report.json (info) plus append-only results.ndjson for findings.
  • Introduced streamed/iterable querying (lower memory use on large reports).

• Bug Fixes

  • Improved report loading/counting from NDJSON when present, with stricter run-scoped isolation.
  • Fixed task/workflow output selection and reduced duplicate emissions (DNS/ASN).
  • Improved regex-style matching for workflow target/tag conditions.

• Changes

  • JSON output is no longer in default exporters (still available via -o json / auto-enabled for local runs).

[ocervell]

@CodeRabbit give it a final review pass

[coderabbitai]

@ocervell I’ll perform a final full review of #1312, with continued focus on unnecessary complexity, duplication, DRY violations, and simplifying the store-backed result flow.

✅ Action performed

Full review finished.

[coderabbitai]

@ocervell I’ll perform a final full review of #1312, focusing on unnecessary complexity, duplication, DRY violations, and opportunities to simplify the store-backed result flow.

✅ Action performed

Full review finished.

[coderabbitai]

Actionable comments posted: 5

♻️ Duplicate comments (1)

secator/hooks/mongodb.py (1)

74-83: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

scan_chunk_id still missing from ensure_mongo_run_id's coercion set.

A previous review flagged that this tuple omits scan_chunk_id, so an uncoerced UUID reaches ObjectId(...) unchanged. The follow-up only removed the (nonexistent) workflow_chunk_id; scan_chunk_id was never added back. update_runner (Line 93-94) derives key = f'{type}_chunk_id' for a chunked scan, and update_finding (Line 156-159) reads the same key to backfill item._context[key] — both paths still hit this gap.

🐛 Proposed fix

-	for key in ('task_id', 'workflow_id', 'scan_id', 'task_chunk_id'):
+	for key in ('task_id', 'workflow_id', 'scan_id', 'task_chunk_id', 'scan_chunk_id'):

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@secator/hooks/mongodb.py` around lines 74 - 83, The ensure_mongo_run_id
function must also coerce scan_chunk_id values to valid Mongo ObjectIds. Add
scan_chunk_id to its existing key set while preserving the current idempotent
validation and coercion behavior for all other identifiers.

🧹 Nitpick comments (3)

tests/unit/test_target_filtering.py (1)

170-186: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicated _store_ctx helper (with temp-dir leak) across two test classes. Both copies write findings to a temp results.ndjson and build a matching ctx, but neither cleans up the created directory.

• tests/unit/test_target_filtering.py#L170-L186: extract to a shared module-level helper (or test mixin) and register addCleanup(shutil.rmtree, d, ignore_errors=True).
• tests/unit/test_target_filtering.py#L253-L269: remove this duplicate copy and call the shared helper instead.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/test_target_filtering.py` around lines 170 - 186, The duplicated
_store_ctx helper in tests/unit/test_target_filtering.py at lines 170-186 and
253-269 should be consolidated into one shared module-level helper or test
mixin. Update the shared implementation to register cleanup with
shutil.rmtree(d, ignore_errors=True), and remove the duplicate at lines 253-269
so both test classes call the shared helper.

secator/output_types/_base.py (1)

209-217: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Unguarded klass.load(doc) can abort the whole stream on one bad record.

load() raises TypeError when a stored dict's _type matches but none of the mapped fields resolve to a value (e.g. legacy/corrupted record). Since this is a generator feeding StreamView/report/export pipelines, one bad row propagates the exception up and aborts the entire query for that run instead of just skipping the offending record.

♻️ Proposed fix

 		if isinstance(doc, dict):
 			klass = by_name.get(doc.get('_type'))
 			if not klass:
 				continue
-			item = klass.load(doc)
+			try:
+				item = klass.load(doc)
+			except TypeError:
+				continue
 			if not item._uuid:
 				item._uuid = str(doc.get('_uuid') or doc.get('_id') or '')
 			yield item

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@secator/output_types/_base.py` around lines 209 - 217, Guard the
klass.load(doc) call in the document-loading loop so a TypeError from an invalid
or legacy record skips that record and allows iteration to continue. Keep the
existing type lookup, UUID assignment, and successful-item yielding behavior
unchanged, and only suppress the expected load failure.

secator/loader.py (1)

303-329: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Docstring contradicts the implementation (and mongodb_enabled is unused).

The body never inspects mongodb_enabled, yet the docstring claims the json default is applied only when "the mongodb addon disabled" and that "In prod (mongodb addon enabled) ... json is not forced". Per the intended behavior, json is appended whenever no mongodb/sqlite/api driver is present in drivers — regardless of the addon. Align the docstring with the code so future maintainers don't misread the driver-selection contract, and either consume or drop the unused parameter.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@secator/loader.py` around lines 303 - 329, Update apply_default_drivers so
its documentation states that json is appended whenever no store driver is
present, regardless of the mongodb addon state; remove the obsolete
mongodb_enabled parameter and update all callers accordingly, or otherwise
consume it consistently with the documented contract. Keep the existing
STORE_DRIVERS check and json-appending behavior unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@secator/query/ast.py`:
- Line 140: Update the AST rendering path around the ast.unparse call to
preserve the declared Python 3.8 support by replacing it with a compatible
alternative; only raise the project’s minimum Python version instead if that is
the intended compatibility change, and keep the existing output behavior.

In `@secator/query/utils.py`:
- Around line 202-205: Update the _TRUTHY representation and its SQLite query
handling so truthiness checks do not generate NOT IN predicates containing NULL.
Preserve absent-field exclusion and ensure values other than '', False, and 0
match consistently across backends, adding or updating the corresponding real
SQLite-backed test.
- Around line 329-330: Update the shared regex normalization helper used by both
JSON and MongoDB query handling to remove leading '*' characters from the
stripped pattern before prepending the '(?i)' prefix. Preserve the existing
quote trimming and ensure inputs such as '*foo' produce a valid normalized regex
consistently.

In `@tests/unit/test_atomic_json.py`:
- Around line 205-206: Rename the ambiguous comprehension variable l to line in
tests/unit/test_atomic_json.py lines 205-206 and tests/unit/test_json_driver.py
lines 208-209. In tests/unit/test_atomic_json.py, also read open(path) within a
with block so the file handle is closed; preserve the existing JSON parsing
behavior.

In `@tests/unit/test_runners_helpers.py`:
- Around line 60-76: Update _store_ctx to register cleanup of the directory
created by tempfile.mkdtemp() using self.addCleanup() (and the appropriate
directory-removal utility), ensuring each temporary results directory is removed
after its test completes while preserving the existing file-writing and context
behavior.

---

Duplicate comments:
In `@secator/hooks/mongodb.py`:
- Around line 74-83: The ensure_mongo_run_id function must also coerce
scan_chunk_id values to valid Mongo ObjectIds. Add scan_chunk_id to its existing
key set while preserving the current idempotent validation and coercion behavior
for all other identifiers.

---

Nitpick comments:
In `@secator/loader.py`:
- Around line 303-329: Update apply_default_drivers so its documentation states
that json is appended whenever no store driver is present, regardless of the
mongodb addon state; remove the obsolete mongodb_enabled parameter and update
all callers accordingly, or otherwise consume it consistently with the
documented contract. Keep the existing STORE_DRIVERS check and json-appending
behavior unchanged.

In `@secator/output_types/_base.py`:
- Around line 209-217: Guard the klass.load(doc) call in the document-loading
loop so a TypeError from an invalid or legacy record skips that record and
allows iteration to continue. Keep the existing type lookup, UUID assignment,
and successful-item yielding behavior unchanged, and only suppress the expected
load failure.

In `@tests/unit/test_target_filtering.py`:
- Around line 170-186: The duplicated _store_ctx helper in
tests/unit/test_target_filtering.py at lines 170-186 and 253-269 should be
consolidated into one shared module-level helper or test mixin. Update the
shared implementation to register cleanup with shutil.rmtree(d,
ignore_errors=True), and remove the duplicate at lines 253-269 so both test
classes call the shared helper.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 988d0801-8949-458f-8282-19aab19afe34

📥 Commits

Reviewing files that changed from the base of the PR and between 158427d and c0afcc6.

📒 Files selected for processing (57)

• docs/superpowers/plans/2026-07-16-json-driver-ndjson.md
• docs/superpowers/specs/2026-07-16-json-driver-ndjson-store-design.md
• pyproject.toml
• secator/celery.py
• secator/cli.py
• secator/cli_helper.py
• secator/config.py
• secator/configs/workflows/host_recon.yaml
• secator/configs/workflows/subdomain_recon.yaml
• secator/configs/workflows/url_params_fuzz.yaml
• secator/configs/workflows/url_vuln.yaml
• secator/definitions.py
• secator/exporters/csv.py
• secator/hooks/json.py
• secator/hooks/mongodb.py
• secator/hooks/sqlite.py
• secator/loader.py
• secator/output_types/__init__.py
• secator/output_types/_base.py
• secator/query/__init__.py
• secator/query/_base.py
• secator/query/_stream.py
• secator/query/ast.py
• secator/query/json.py
• secator/query/mongodb.py
• secator/query/sqlite.py
• secator/query/utils.py
• secator/report.py
• secator/runners/_base.py
• secator/runners/_helpers.py
• secator/runners/scan.py
• secator/runners/task.py
• secator/runners/workflow.py
• secator/tasks/ai.py
• secator/tasks/dnsx.py
• secator/tasks/getasn.py
• secator/utils.py
• tests/integration/test_celery.py
• tests/unit/conftest.py
• tests/unit/test_ai_actions.py
• tests/unit/test_atomic_json.py
• tests/unit/test_celery.py
• tests/unit/test_cli.py
• tests/unit/test_driver_ordering.py
• tests/unit/test_error_scoping.py
• tests/unit/test_extractor_query.py
• tests/unit/test_json_driver.py
• tests/unit/test_on_build.py
• tests/unit/test_query.py
• tests/unit/test_query_utils.py
• tests/unit/test_query_utils_extractors.py
• tests/unit/test_report.py
• tests/unit/test_runners.py
• tests/unit/test_runners_helpers.py
• tests/unit/test_sqlite_driver.py
• tests/unit/test_target_filtering.py
• tests/unit/test_template.py

[coderabbitai]

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

tests/unit/test_celery.py (1)

132-244: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Guard is still always-true — 4 tests remain dead code.

if httpx not in TEST_TASKS: (lines 136, 172, 205, 228) repeats the exact stale-comparison bug already flagged: TEST_TASKS holds TemplateLoaders, so <class> in TEST_TASKS is always False, and every one of these four tests returns before exercising break_task's (now simplified) chunking/rate-limit/queue-routing logic. The file already defines and correctly uses TEST_TASK_NAMES elsewhere for this exact purpose.

🐛 Proposed fix (apply at all 4 occurrences)

-		if httpx not in TEST_TASKS:
+		if 'httpx' not in TEST_TASK_NAMES:
 			return

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/test_celery.py` around lines 132 - 244, Replace the four stale `if
httpx not in TEST_TASKS:` guards in the affected test methods with the existing
`TEST_TASK_NAMES`-based membership check, matching its established usage
elsewhere in the file, so all chunking, rate-limit, queue-routing, and
disabled-chunking tests execute instead of returning early.

♻️ Duplicate comments (4)

secator/query/json.py (1)

41-50: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Sanitize runner path segments before building report_file (still unaddressed). runner_type/runner_id are joined directly into the report path, so a token containing /, \, or .. can escape the workspace report tree. Validate the trimmed/singularized segments (reject separators and traversal) before opening.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@secator/query/json.py` around lines 41 - 50, Validate the normalized
runner_type and runner_id segments in the token-processing function before
constructing report_file or opening it. Reject segments containing forward or
backslash separators or traversal values such as “..”, and preserve the existing
fallback of appending part when validation fails.

Source: Linters/SAST tools

secator/query/ast.py (1)

140-140: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

ast.unparse still breaks the declared Python 3.8 floor. ast.unparse was added in 3.9; if pyproject.toml still declares requires-python = '>=3.8', this raises AttributeError on the supported floor. Either raise the minimum to 3.9+ or avoid ast.unparse.

#!/bin/bash
fd -HI pyproject.toml -x rg -n 'requires-python|python_requires' {}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@secator/query/ast.py` at line 140, Resolve the Python version mismatch in the
AST unparsing path around ast.unparse: either raise the project’s declared
requires-python floor to Python 3.9 or newer, or replace ast.unparse with a
Python 3.8-compatible implementation. Keep the behavior of the affected AST
conversion function unchanged for supported versions.

Source: Linters/SAST tools

secator/runners/_base.py (1)

875-894: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Items added with hooks=False (while enable_hooks=True) are still never persisted. The on_item fan-out at Line 875 is gated on the hooks parameter, but the store fallback at Line 884 only fires on not self.enable_hooks. So add_result(item, hooks=False) with hooks enabled — e.g. run_hooks's except-block self.add_result(error, hooks=False) on a hook failure — neither runs on_item nor calls _persist_to_store, so the item never reaches the store. Aggregators read errors from the store, so a hook-execution failure can leave a workflow/scan reporting SUCCESS.

🐛 Proposed fix

 		self.uuids.add(item._uuid)
-		if not self.enable_hooks:
+		if not self.enable_hooks or not hooks:
 			self._persist_to_store(item)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@secator/runners/_base.py` around lines 875 - 894, Update the result
persistence logic in the add_result flow around the hooks-gated on_item fan-out
so items added with hooks=False are persisted whenever on_item was not executed,
including when enable_hooks is true. Preserve the existing no-duplicate behavior
for items that already went through on_item, and ensure hook failures passed
from run_hooks via add_result(error, hooks=False) reach the store.

tests/unit/test_celery.py (1)

433-466: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Unused httpx import (F401) still present.

from secator.tasks import httpx at line 438 is never used — abandon_task is invoked with the string 'httpx'.

🧹 Proposed fix

 		import tempfile
 		from pathlib import Path
-		from secator.tasks import httpx
 		from secator.celery import abandon_task

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/test_celery.py` around lines 433 - 466, Remove the unused `from
secator.tasks import httpx` import from
`test_abandon_task_persists_failure_error_to_store`; keep the string-based
`'httpx'` task reference and all existing test behavior unchanged.

Source: Linters/SAST tools

🧹 Nitpick comments (5)

secator/hooks/mongodb.py (1)

88-88: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Rename type to avoid builtin shadowing (and match sqlite.py's convention).

type = self.config.type shadows the builtin, flagged by Ruff (A001). secator/hooks/sqlite.py's update_runner already uses _type = self.config.type for the identical purpose — align this file with that convention.

♻️ Proposed rename

 def update_runner(self):
 	client = get_mongodb_client()
 	db = client.main
-	type = self.config.type
-	collection = f'{type}s'
-	chunk = self.context.get(f'{type}_chunk_id') is not None
+	_type = self.config.type
+	collection = f'{_type}s'
+	chunk = self.context.get(f'{_type}_chunk_id') is not None
 	ensure_mongo_run_id(self.context)
 	update = self.toDict()
-	key = f'{type}_chunk_id' if chunk else f'{type}_id'
+	key = f'{_type}_chunk_id' if chunk else f'{_type}_id'

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@secator/hooks/mongodb.py` at line 88, In the MongoDB hook’s update_runner
flow, rename the local variable `type` to `_type` when assigning
`self.config.type`, and update all references to that local variable
accordingly, matching the convention used by sqlite.py.

Source: Linters/SAST tools

tests/unit/test_target_filtering.py (1)

170-186: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Same _store_ctx tempdir-leak pattern copy-pasted across three test helpers. Each helper calls tempfile.mkdtemp() and never removes the directory, so every test run leaks one dir per invocation; the shared root cause is a missing self.addCleanup(shutil.rmtree, d).

• tests/unit/test_target_filtering.py#L170-L186: add self.addCleanup(shutil.rmtree, d, ignore_errors=True) right after d = Path(tempfile.mkdtemp()) in TestProcessExtractorScopeFiltering._store_ctx.
• tests/unit/test_target_filtering.py#L253-L269: apply the same cleanup registration in TestRunExtractorsScopeFallback._store_ctx.
• tests/unit/test_runners_helpers.py#L60-L76: apply the same fix in TestExtractorFunctions._store_ctx (already flagged in a prior review pass with an equivalent diff).

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/test_target_filtering.py` around lines 170 - 186, The three
_store_ctx helpers leak their temporary directories; register shutil.rmtree
cleanup immediately after each tempfile.mkdtemp call. Apply this in
tests/unit/test_target_filtering.py:170-186 for
TestProcessExtractorScopeFiltering._store_ctx,
tests/unit/test_target_filtering.py:253-269 for
TestRunExtractorsScopeFallback._store_ctx, and
tests/unit/test_runners_helpers.py:60-76 for TestExtractorFunctions._store_ctx,
using ignore_errors=True.

secator/loader.py (1)

303-329: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Docstring contradicts the actual (and correct) behavior; mongodb_enabled is dead.

The opening docstring line says json is forced only when "the mongodb addon disabled" and that in prod "mongodb still wins and json is not forced" — but mongodb_enabled is never read, and the inline comment right below (lines 323-326) plus the retrieved fix history confirm the intended behavior is the opposite: json is still forced when the addon is enabled but 'mongodb' isn't literally in drivers. This stale docstring could mislead a future edit into "using" mongodb_enabled and reintroducing the earlier data-integrity gap (empty drivers list → no persistence hooks).

♻️ Proposed docstring fix

-	Applied ONLY when no store backend is otherwise active: no
-	mongodb/sqlite/api driver in ``drivers`` and the mongodb addon disabled. In prod
-	(mongodb addon enabled) mongodb still wins and json is not forced.
+	Applied ONLY when no store backend (``mongodb``/``sqlite``/``api``) is present in
+	``drivers``. Deliberately ignores ``mongodb_enabled``: even with the mongodb addon
+	on, if ``mongodb`` isn't in ``drivers`` there is still no persistence hook registered,
+	so json is forced regardless (see the inline comment below).
 
 	Args:
 		drivers (list[str]): Resolved driver names (config defaults + CLI --driver).
-		mongodb_enabled (bool): Whether the mongodb addon is enabled.
+		mongodb_enabled (bool): Unused; kept for call-site compatibility.

Separately, Ruff's RUF005 hint on line 328 is worth a quick fix too: drivers = [*drivers, 'json'] instead of concatenation.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@secator/loader.py` around lines 303 - 329, Update apply_default_drivers so
its docstring accurately states that json is appended whenever no store driver
is present, including when the mongodb addon is enabled but mongodb is absent
from drivers; remove the unused mongodb_enabled parameter if callers permit, or
otherwise leave it unused without implying it controls behavior. Also replace
the drivers concatenation with the Ruff-compliant unpacking form while
preserving existing ordering and duplicate checks.

Source: Linters/SAST tools

tests/unit/test_template.py (1)

47-47: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Prefer next(...) over materializing the whole (streaming) result set for a single element.

Both list(findings)[0] (line 47) and [r for r in findings if r._type == 'vulnerability'][0] (line 71) fully materialize findings — now a store-backed, potentially lazy StreamView per this PR's redesign — just to grab one element. Flagged by Ruff (RUF015).

⚡ Proposed fix

-		self.assertTrue(self.expected_vuln == Vulnerability.load(list(findings)[0].toDict()))
+		self.assertTrue(self.expected_vuln == Vulnerability.load(next(iter(findings)).toDict()))

-			vuln = [r for r in findings if r._type == 'vulnerability'][0]
+			vuln = next(r for r in findings if r._type == 'vulnerability')

Also applies to: 71-72

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/test_template.py` at line 47, Update the assertions using list
materialization in the test flow, including the comparison around
Vulnerability.load and the filtered vulnerability lookup, to retrieve the first
matching result with next(...) instead of converting the lazy findings stream to
a list. Preserve the existing filtering and assertion behavior.

Source: Linters/SAST tools

tests/integration/test_celery.py (1)

34-40: 📐 Maintainability & Code Quality | 🔵 Trivial

Chain/chord-against-seeded-store test still not restored.

In the prior review thread you agreed ("Yes do that") to restore a hand-built chain/chord test that seeds the store with a Url, dispatches a hand-assembled chord of run_command sigs joined by join_results, and asserts seeded + new findings via the store. That test doesn't appear in this file yet — only test_httpx_async/test_httpx_chunk/test_nmap_async/test_ffuf_chunked/test_url_vuln_workflow are present.

Want me to draft that test_httpx_chord (or equivalent) test now, or is it tracked in a separate follow-up commit?

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/test_celery.py` around lines 34 - 40, The TestCelery
integration coverage is missing the agreed hand-built chord test against a
seeded store. Add a test method such as test_httpx_chord that seeds the store
with a Url and initial finding, builds run_command signatures joined by
join_results, executes the chord through the Celery workflow, and verifies both
seeded and newly produced findings via the store-backed results.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@secator/query/_stream.py`:
- Around line 37-45: Update __contains__ to preserve the stream’s self._query
scope when building the keyed existence query, merging the item comparison
fields into that existing query rather than replacing it. Keep the non-dataclass
iteration fallback unchanged, and ensure membership only matches records within
the current run or other configured view scope.

In `@secator/report.py`:
- Around line 67-68: Update the report data construction to materialize the lazy
warnings stream before assigning it to info['warnings']. In the relevant
report-building method, store a list from self.runner.warnings, matching the
existing list-valued errors behavior so JsonExporter can serialize report.data
directly.

In `@tests/unit/test_error_scoping.py`:
- Around line 53-64: Update the Scan constructor in
test_scan_aggregates_all_descendant_errors to pass run_opts={'dry_run': True},
matching the other tests in this file and ensuring add_result uses the safe
direct-persist path without real hooks.

In `@tests/unit/test_extractor_query.py`:
- Line 445: Update the JsonBackend instantiations in the affected tests to stop
passing results=, since JsonBackend accepts only workspace_id, config, and
context. Seed the backend through the existing workspace or report-directory
fixture so the tests obtain the findings through the supported initialization
path, including the similar instantiation near the other reported location.

In `@tests/unit/test_query.py`:
- Around line 289-294: Update python_expr_to_mongo and
test_regex_value_with_glob_start so a leading glob '*' is removed before
prepending the '(?i)' regex prefix; assert the resulting pattern is
'(?i)CVE-2026-28780' rather than retaining the invalid leading wildcard.

---

Outside diff comments:
In `@tests/unit/test_celery.py`:
- Around line 132-244: Replace the four stale `if httpx not in TEST_TASKS:`
guards in the affected test methods with the existing `TEST_TASK_NAMES`-based
membership check, matching its established usage elsewhere in the file, so all
chunking, rate-limit, queue-routing, and disabled-chunking tests execute instead
of returning early.

---

Duplicate comments:
In `@secator/query/ast.py`:
- Line 140: Resolve the Python version mismatch in the AST unparsing path around
ast.unparse: either raise the project’s declared requires-python floor to Python
3.9 or newer, or replace ast.unparse with a Python 3.8-compatible
implementation. Keep the behavior of the affected AST conversion function
unchanged for supported versions.

In `@secator/query/json.py`:
- Around line 41-50: Validate the normalized runner_type and runner_id segments
in the token-processing function before constructing report_file or opening it.
Reject segments containing forward or backslash separators or traversal values
such as “..”, and preserve the existing fallback of appending part when
validation fails.

In `@secator/runners/_base.py`:
- Around line 875-894: Update the result persistence logic in the add_result
flow around the hooks-gated on_item fan-out so items added with hooks=False are
persisted whenever on_item was not executed, including when enable_hooks is
true. Preserve the existing no-duplicate behavior for items that already went
through on_item, and ensure hook failures passed from run_hooks via
add_result(error, hooks=False) reach the store.

In `@tests/unit/test_celery.py`:
- Around line 433-466: Remove the unused `from secator.tasks import httpx`
import from `test_abandon_task_persists_failure_error_to_store`; keep the
string-based `'httpx'` task reference and all existing test behavior unchanged.

---

Nitpick comments:
In `@secator/hooks/mongodb.py`:
- Line 88: In the MongoDB hook’s update_runner flow, rename the local variable
`type` to `_type` when assigning `self.config.type`, and update all references
to that local variable accordingly, matching the convention used by sqlite.py.

In `@secator/loader.py`:
- Around line 303-329: Update apply_default_drivers so its docstring accurately
states that json is appended whenever no store driver is present, including when
the mongodb addon is enabled but mongodb is absent from drivers; remove the
unused mongodb_enabled parameter if callers permit, or otherwise leave it unused
without implying it controls behavior. Also replace the drivers concatenation
with the Ruff-compliant unpacking form while preserving existing ordering and
duplicate checks.

In `@tests/integration/test_celery.py`:
- Around line 34-40: The TestCelery integration coverage is missing the agreed
hand-built chord test against a seeded store. Add a test method such as
test_httpx_chord that seeds the store with a Url and initial finding, builds
run_command signatures joined by join_results, executes the chord through the
Celery workflow, and verifies both seeded and newly produced findings via the
store-backed results.

In `@tests/unit/test_target_filtering.py`:
- Around line 170-186: The three _store_ctx helpers leak their temporary
directories; register shutil.rmtree cleanup immediately after each
tempfile.mkdtemp call. Apply this in tests/unit/test_target_filtering.py:170-186
for TestProcessExtractorScopeFiltering._store_ctx,
tests/unit/test_target_filtering.py:253-269 for
TestRunExtractorsScopeFallback._store_ctx, and
tests/unit/test_runners_helpers.py:60-76 for TestExtractorFunctions._store_ctx,
using ignore_errors=True.

In `@tests/unit/test_template.py`:
- Line 47: Update the assertions using list materialization in the test flow,
including the comparison around Vulnerability.load and the filtered
vulnerability lookup, to retrieve the first matching result with next(...)
instead of converting the lazy findings stream to a list. Preserve the existing
filtering and assertion behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 988d0801-8949-458f-8282-19aab19afe34

📥 Commits

Reviewing files that changed from the base of the PR and between 158427d and c0afcc6.

📒 Files selected for processing (57)

• docs/superpowers/plans/2026-07-16-json-driver-ndjson.md
• docs/superpowers/specs/2026-07-16-json-driver-ndjson-store-design.md
• pyproject.toml
• secator/celery.py
• secator/cli.py
• secator/cli_helper.py
• secator/config.py
• secator/configs/workflows/host_recon.yaml
• secator/configs/workflows/subdomain_recon.yaml
• secator/configs/workflows/url_params_fuzz.yaml
• secator/configs/workflows/url_vuln.yaml
• secator/definitions.py
• secator/exporters/csv.py
• secator/hooks/json.py
• secator/hooks/mongodb.py
• secator/hooks/sqlite.py
• secator/loader.py
• secator/output_types/__init__.py
• secator/output_types/_base.py
• secator/query/__init__.py
• secator/query/_base.py
• secator/query/_stream.py
• secator/query/ast.py
• secator/query/json.py
• secator/query/mongodb.py
• secator/query/sqlite.py
• secator/query/utils.py
• secator/report.py
• secator/runners/_base.py
• secator/runners/_helpers.py
• secator/runners/scan.py
• secator/runners/task.py
• secator/runners/workflow.py
• secator/tasks/ai.py
• secator/tasks/dnsx.py
• secator/tasks/getasn.py
• secator/utils.py
• tests/integration/test_celery.py
• tests/unit/conftest.py
• tests/unit/test_ai_actions.py
• tests/unit/test_atomic_json.py
• tests/unit/test_celery.py
• tests/unit/test_cli.py
• tests/unit/test_driver_ordering.py
• tests/unit/test_error_scoping.py
• tests/unit/test_extractor_query.py
• tests/unit/test_json_driver.py
• tests/unit/test_on_build.py
• tests/unit/test_query.py
• tests/unit/test_query_utils.py
• tests/unit/test_query_utils_extractors.py
• tests/unit/test_report.py
• tests/unit/test_runners.py
• tests/unit/test_runners_helpers.py
• tests/unit/test_sqlite_driver.py
• tests/unit/test_target_filtering.py
• tests/unit/test_template.py