feat(sec-core): migrate PII checker to Rust - #3286
Draft
1570005763 wants to merge 5 commits into
Draft
1570005763 wants to merge 5 commits into
1570005763 wants to merge 5 commits into
Conversation
1570005763
force-pushed
the
codex/sec-core-pii-v2
branch
4 times, most recently
from
September 16, 2026 07:41
67010e9 to
408e63e
Compare
Add a transport-independent detector with Unicode character spans, validated builtin findings and explicit input coverage and digests. Linear candidates retain Python boundary semantics for large tokens. Retain frozen V1 cases and label deliberate detection changes as scanner 2.0.0. Cover compact JWTs, Unicode IDs and zero-card placeholders explicitly. Central custom rules and runtime integration follow in separate commits. Assisted-by: Codex:0.153.4 Signed-off-by: 1570005763 <daniel.duan@linux.alibaba.com>
Compile administrator YAML into an immutable shared rule collection. Keep builtin detection active when the custom collection is invalid, and report configuration and matching limits through explicit coverage. Bound parsing, backtracking and findings without promising interruption of an individual regex call. Rule updates require a newly loaded set. Use versioned native regex semantics with positive and negative span controls. Reject actual syntax and resource failures without a Python-compatibility lexer. Assisted-by: Codex:0.153.4 Signed-off-by: 1570005763 <daniel.duan@linux.alibaba.com>
Compose PII execution and explicit audit projections with the shared Finalizer. Add a common rejection entry so authorized parameter failures receive a terminal event without decoding or persisting unsafe input. Keep scan outcomes independent of JSONL and SQLite write health. Persist only approved summaries and findings; raw evidence remains client-only. Assisted-by: Codex:0.153.4 Signed-off-by: 1570005763 <daniel.duan@linux.alibaba.com>
1570005763
force-pushed
the
codex/sec-core-pii-v2
branch
from
September 16, 2026 07:58
408e63e to
f60ef34
Compare
Route authorized PII requests through typed parameters and shared finalization. Load centralized rules once at daemon startup and keep file access at the client boundary, preserving kernel peer identity. Preserve scan-pii output, trace aliases and explicit UTF-8 truncation. Scanning has no Python fallback and does not perform policy enforcement. Global-option duplicate checks must distinguish scan text from options. Fixes: 355d55e ("feat(sec-core): route scan-code through v2 cli") Bound returned details after complete detection and redaction so large reports retain their verdict across RPC. Keep full totals in audit and Hook notices. Assisted-by: Codex:0.153.4 Signed-off-by: 1570005763 <daniel.duan@linux.alibaba.com>
Exercise the same PII assertions against both runtimes and run all six Hook adapters with real Rust subprocesses. Isolate only unmigrated observability storage so installed RPM acceptance can hide V1 detection. Document centralized rules, coverage, rollback and future policy roles. Keep real host cutover and full policy enforcement outside this phase. Cover dense PII and output omission through real UDS and all six Hook adapters. Document output bounds separately from scan coverage. Assisted-by: Codex:0.153.4 Signed-off-by: 1570005763 <daniel.duan@linux.alibaba.com>
1570005763
force-pushed
the
codex/sec-core-pii-v2
branch
from
September 16, 2026 08:21
f60ef34 to
eddcc48
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
PII scanning still depended on the V1 Python runtime. This migrates phase 1 to the Rust V2 daemon while preserving the supported scan and Hook contracts and preparing typed evidence for later policy integration.
What changed
action.pii_scanand compatiblescan-piiCLI through shared ActionRuntime, sanitized projection and Finalizer, including authorized parameter failures.Five logical commits: core, rules, runtime/audit, RPC/CLI, acceptance/docs. The shared infrastructure from #3246 is already on the main base and is not included in this PR.
Related issue
no-issue: implements the approved two-stage PII migration design and five-commit phase-1 delivery plan.
User / Agent impact
V2 supports scan-pii without Python detection or a local fallback. Custom rules use
/etc/agent-sec/pii-checker/rules.yaml(or daemon--pii-rules) and restart to update. Completed pass/warn/deny results exit 0; coverage exposes incomplete detection. Existing host behavior is preserved through direct Hook contract tests; this PR does not switch running Agent hosts or perform PDP enforcement.Risk and compatibility
The daemon authorizes LocalUser from UDS peer identity and never accepts input/rule paths from scan RPC. Audit uses a field allowlist and excludes original text, raw evidence, full redacted output and rule/exception content. Invalid custom rules leave built-ins running with partial coverage. Custom rules follow native
fancy-regexsemantics, documented with positive/negative span controls.invalid_regexreports actual syntax, compilation or engine-limit failures; semantic differences from Python alone are not errors. Match-budget differences are documented; no default truncation is added, and the existing 4 MiB frame bound remains explicit. Opaque V1 trace metadata is not treated as OpenTelemetry IDs.Detection behavior is versioned as
summary.scanner_version: "2.0.0", independently of package versions and CI commit IDs, and participates in ruleset identity. Compact JWT claims objects are now detected; Unicode decimal ID dates/check digits are normalized before validation; all-zero card placeholders are rejected. Rust's structural JWT parser intentionally accepts valid large-number/deeply nested objects without recreating Python integer/recursion limits. Signature authenticity is outside detector scope. Preserved V1 fixtures exclude only nondeterministic/additive fields and explicitly versioned engine provenance; new behavior has dedicated positive and negative tests. These tests do not establish a production-wide false-positive or false-negative rate.Large reports use a 512 KiB formatted-JSON budget. The executor keeps one real finding per type/severity, removes raw evidence, and marks reduced details with
summary.findings_truncated. Complete verdict/counts/digests/scan coverage remain intact. If full redacted output still cannot fit, it is replaced entirely with[REDACTED: output size limit]andsummary.redacted_text_omitted=true. Six Hook adapters use full summary counts in notices. Normal reports and existing failure policies retain their behavior; no pagination, cache, or new policy infrastructure is introduced.Validation
Local environment: macOS, Python 3.11.6 and Node.js.
252 passed.
tests/unit/pii-scan-test.ts: 26 passed.make python-code-prettyandcargo fmt --all -- --check: passed.bash scripts/docs-lint.sh,python3 scripts/docs-link-check.py, andgit diff --check: passed.not local execution of those E2E cases.
Linux-only Rust build, Clippy, workspace tests, rustdoc, UDS runtime and installed-RPM
acceptance were not run on this local macOS machine.
Documentation and rollback
Updated English/Chinese PII guides, READMEs, two-stage design, relevant versioned behavior contracts and the bundled Skill. Restore the previous central YAML and restart for a rules rollback. For runtime rollback, stop the V2 validation daemon and restore V1 packages/entrypoints and retained user rules. V1 detection implementation is preserved. Full PIP/PDP/PEP and real host cutover remain separate work.