Skip to content

feat(sec-core): migrate PII checker to Rust - #3286

Draft
1570005763 wants to merge 5 commits into
agentic-os-org:mainfrom
1570005763:codex/sec-core-pii-v2
Draft

1570005763 wants to merge 5 commits into
agentic-os-org:mainfrom
1570005763:codex/sec-core-pii-v2

Conversation

@1570005763

@1570005763 1570005763 commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Why

PII scanning still depended on the V1 Python runtime. This migrates phase 1 to the Rust V2 daemon while preserving the supported scan and Hook contracts and preparing typed evidence for later policy integration.

What changed

  • Port 11 built-in detectors, validators, confidence, Unicode spans and redaction with 142 frozen V1 cases and a separate labeled corpus for intentional scanner 2.0.0 changes.
  • Compile one administrator-managed custom ruleset at startup and report execution status, coverage, input digests and rule identity.
  • Route action.pii_scan and compatible scan-pii CLI through shared ActionRuntime, sanitized projection and Finalizer, including authorized parameter failures.
  • Bound response details after complete detection and redaction, retaining the full verdict and counts so dense findings do not become a transport error or bypass Hook blocking.
  • Add real UDS/CLI, shared V1/V2 PII, six Hook and installed RPM acceptance, plus bilingual migration/usage documentation.

Five logical commits: core, rules, runtime/audit, RPC/CLI, acceptance/docs. The shared infrastructure from #3246 is already on the main base and is not included in this PR.

Related issue

no-issue: implements the approved two-stage PII migration design and five-commit phase-1 delivery plan.

User / Agent impact

V2 supports scan-pii without Python detection or a local fallback. Custom rules use /etc/agent-sec/pii-checker/rules.yaml (or daemon --pii-rules) and restart to update. Completed pass/warn/deny results exit 0; coverage exposes incomplete detection. Existing host behavior is preserved through direct Hook contract tests; this PR does not switch running Agent hosts or perform PDP enforcement.

Risk and compatibility

  • Public CLI, API, configuration, or documented behavior changed
  • Privileged or security-sensitive behavior changed
  • Cross-component contract changed
  • Migration or rollback guidance is needed

The daemon authorizes LocalUser from UDS peer identity and never accepts input/rule paths from scan RPC. Audit uses a field allowlist and excludes original text, raw evidence, full redacted output and rule/exception content. Invalid custom rules leave built-ins running with partial coverage. Custom rules follow native fancy-regex semantics, documented with positive/negative span controls. invalid_regex reports actual syntax, compilation or engine-limit failures; semantic differences from Python alone are not errors. Match-budget differences are documented; no default truncation is added, and the existing 4 MiB frame bound remains explicit. Opaque V1 trace metadata is not treated as OpenTelemetry IDs.

Detection behavior is versioned as summary.scanner_version: "2.0.0", independently of package versions and CI commit IDs, and participates in ruleset identity. Compact JWT claims objects are now detected; Unicode decimal ID dates/check digits are normalized before validation; all-zero card placeholders are rejected. Rust's structural JWT parser intentionally accepts valid large-number/deeply nested objects without recreating Python integer/recursion limits. Signature authenticity is outside detector scope. Preserved V1 fixtures exclude only nondeterministic/additive fields and explicitly versioned engine provenance; new behavior has dedicated positive and negative tests. These tests do not establish a production-wide false-positive or false-negative rate.

Large reports use a 512 KiB formatted-JSON budget. The executor keeps one real finding per type/severity, removes raw evidence, and marks reduced details with summary.findings_truncated. Complete verdict/counts/digests/scan coverage remain intact. If full redacted output still cannot fit, it is replaced entirely with [REDACTED: output size limit] and summary.redacted_text_omitted=true. Six Hook adapters use full summary counts in notices. Normal reports and existing failure policies retain their behavior; no pagination, cache, or new policy infrastructure is introduced.

Validation

Local environment: macOS, Python 3.11.6 and Node.js.

  • Targeted Python PII Hook tests for Codex, Cosh, Hermes, Qwen Code and Qoder:
    252 passed.
  • OpenClaw tests/unit/pii-scan-test.ts: 26 passed.
  • make python-code-pretty and cargo fmt --all -- --check: passed.
  • Changed-file Ruff comparison: no new diagnostics; 10 existing Hermes diagnostics remain.
  • bash scripts/docs-lint.sh, python3 scripts/docs-link-check.py, and
    git diff --check: passed.
  • PII CLI and Hook E2E collection: 238 collected. This was collection-only,
    not local execution of those E2E cases.

Linux-only Rust build, Clippy, workspace tests, rustdoc, UDS runtime and installed-RPM
acceptance were not run on this local macOS machine.

Documentation and rollback

Updated English/Chinese PII guides, READMEs, two-stage design, relevant versioned behavior contracts and the bundled Skill. Restore the previous central YAML and restart for a rules rollback. For runtime rollback, stop the V2 validation daemon and restore V1 packages/entrypoints and retained user rules. V1 detection implementation is preserved. Full PIP/PDP/PEP and real host cutover remain separate work.

@github-actions github-actions Bot added component:sec-core src/agent-sec-core/ scope:ci ./.github/ scope:documentation ./docs/|./*.md|./NOTICE labels Sep 15, 2026
@1570005763
1570005763 force-pushed the codex/sec-core-pii-v2 branch 4 times, most recently from 67010e9 to 408e63e Compare September 16, 2026 07:41
Add a transport-independent detector with Unicode character spans,
validated builtin findings and explicit input coverage and digests.
Linear candidates retain Python boundary semantics for large tokens.

Retain frozen V1 cases and label deliberate detection changes as scanner 2.0.0.
Cover compact JWTs, Unicode IDs and zero-card placeholders explicitly.
Central custom rules and runtime integration follow in separate commits.

Assisted-by: Codex:0.153.4
Signed-off-by: 1570005763 <daniel.duan@linux.alibaba.com>
Compile administrator YAML into an immutable shared rule collection.
Keep builtin detection active when the custom collection is invalid,
and report configuration and matching limits through explicit coverage.

Bound parsing, backtracking and findings without promising interruption
of an individual regex call. Rule updates require a newly loaded set.

Use versioned native regex semantics with positive and negative span controls.
Reject actual syntax and resource failures without a Python-compatibility lexer.

Assisted-by: Codex:0.153.4
Signed-off-by: 1570005763 <daniel.duan@linux.alibaba.com>
Compose PII execution and explicit audit projections with the shared
Finalizer. Add a common rejection entry so authorized parameter failures
receive a terminal event without decoding or persisting unsafe input.

Keep scan outcomes independent of JSONL and SQLite write health. Persist
only approved summaries and findings; raw evidence remains client-only.

Assisted-by: Codex:0.153.4
Signed-off-by: 1570005763 <daniel.duan@linux.alibaba.com>
@1570005763
1570005763 force-pushed the codex/sec-core-pii-v2 branch from 408e63e to f60ef34 Compare September 16, 2026 07:58
Route authorized PII requests through typed parameters and shared
finalization. Load centralized rules once at daemon startup and keep
file access at the client boundary, preserving kernel peer identity.

Preserve scan-pii output, trace aliases and explicit UTF-8 truncation.
Scanning has no Python fallback and does not perform policy enforcement.
Global-option duplicate checks must distinguish scan text from options.

Fixes: 355d55e ("feat(sec-core): route scan-code through v2 cli")
Bound returned details after complete detection and redaction so large reports
retain their verdict across RPC. Keep full totals in audit and Hook notices.

Assisted-by: Codex:0.153.4
Signed-off-by: 1570005763 <daniel.duan@linux.alibaba.com>
Exercise the same PII assertions against both runtimes and run all six
Hook adapters with real Rust subprocesses. Isolate only unmigrated
observability storage so installed RPM acceptance can hide V1 detection.

Document centralized rules, coverage, rollback and future policy roles.
Keep real host cutover and full policy enforcement outside this phase.

Cover dense PII and output omission through real UDS and all six Hook adapters.
Document output bounds separately from scan coverage.

Assisted-by: Codex:0.153.4
Signed-off-by: 1570005763 <daniel.duan@linux.alibaba.com>
@1570005763
1570005763 force-pushed the codex/sec-core-pii-v2 branch from f60ef34 to eddcc48 Compare September 16, 2026 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component:sec-core src/agent-sec-core/ scope:ci ./.github/ scope:documentation ./docs/|./*.md|./NOTICE

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant