[Bug]: Legacy uint16-truncated records are permanently undeletable — delete/upsert still json.loads corrupt old_fields, no repair path for pre-#2171 stores

[lRoccoon]

Bug Description

Follow-up to #2117 / #1892: stores that contain legacy uint16-truncated records (written by versions before #2171) still have no cleanup path on current main.

Great progress has already landed for this family of corruption:

• fix(vectordb): reject oversized bytes row strings #2171 rejects oversized BytesRow strings at write time (no new corruption);
• PersistCollection._replay_recovery_records skips corrupt delta records during startup recovery (batch → individual retry → skip);
• fix(vectordb): skip candidates with corrupted JSON fields in search / 检索时跳过 fields 损坏的候选 #2302 skips candidates with corrupt fields during search.

However, all of these are read-side mitigations. The mutation paths still hard-fail on a corrupt record:

• LocalIndex.delete_data() and upsert_data() both run _convert_delta_list_for_index() (openviking/storage/vectordb/index/local_index.py:339), which calls FieldTypeConverter.convert_fields_for_index() → a bare json.loads(fields_json) on fields / old_fields with no error handling.
• StoreManager.delete_data() populates DeltaRecord.old_fields directly from the stored (corrupt) candidate value.

Net effect: a legacy-corrupt record can never be deleted or overwritten through normal APIs — the JSONDecodeError makes the delete/upsert fail every time. In our production incident this had a nasty second-order consequence: a memory-cleanup job deleted the AGFS .md files, but the corresponding vectordb deletes silently failed, leaving 41 "ghost" records (source file gone, corrupt vector record permanently stuck, Candidate data is None ... skipping warnings on every query).

This is exactly the "Bug 1 needs its own thread" part that was deferred in #1892 and (as far as I can search) never got that thread.

Steps to Reproduce

1. Run a pre-fix(vectordb): reject oversized bytes row strings #2171 image (e.g. the 2026-05-15 ghcr.io/volcengine/openviking:latest) with vectordb.backend: local, and commit a memory whose serialized fields JSON exceeds 65535 bytes (easy with CJK content: \uXXXX escaping inflates UTF-8 roughly 2×, so a ~40 KB Chinese .md is enough). The uint16 length prefix wraps mod 65536 and a truncated record is persisted.
2. Upgrade to current main. Startup recovery and search now tolerate the record (thanks!).
3. Try to delete (or overwrite) that record via any path that reaches LocalCollection.delete_data() — e.g. forget, or a memory-cleanup flow.

Expected Behavior

• Deleting a record whose stored fields/old_fields no longer parse should still succeed — best-effort: treat unparseable old_fields as empty for index-delete purposes (the label is all the engine really needs to drop it), log a warning, and remove the record from CandsTable.
• Ideally, an offline fsck/repair subcommand that scans CandsTable/DeltaTable for records whose fields fail json.loads, reports them, and quarantines/purges them on request — so operators of pre-fix(vectordb): reject oversized bytes row strings #2171 stores have an official migration path instead of raw RocksDB surgery.

Actual Behavior

_convert_delta_list_for_index raises json.decoder.JSONDecodeError, the delete/upsert fails, and the corrupt record is permanently stuck. Combined with higher layers that delete the source file first, this produces undeletable ghost records.

Minimal Reproducible Example

# maintenance-style scan we used to quantify the damage (run inside the image)
import json
from openviking.storage.vectordb.store.data import CandidateData, DeltaRecord
from openviking.storage.vectordb.store.store_manager import StoreManager, create_store_manager

sm = create_store_manager("local", "/app/data/vectordb/context/store")
for k, v in sm.storage.read_all(StoreManager.CandsTable):
    c = CandidateData.from_bytes(data=v)
    try:
        json.loads(c.fields)
    except Exception as e:
        print("corrupt candidate", k, e)
        # LocalCollection.delete_data([c.label]) -> raises the same JSONDecodeError,
        # so there is no API-level way to get rid of this record.

Error Logs

Real-world incident (store written by the 2026-05-15 image; 425 candidates, 42 corrupt; 3628 delta records, 268 corrupt; container crash-looped 397 times on that image — current main survives startup but cannot clean the store):

File ".../openviking/storage/vectordb/index/local_index.py", line 355, in _convert_delta_list_for_index
    self.field_type_converter.convert_fields_for_index(data.old_fields)
File ".../openviking/storage/vectordb/utils/data_processor.py", line 337, in convert_fields_for_index
    data = json.loads(fields_json)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 296 (char 295)

Forensics note for anyone hitting this: the truncated values are not recoverable — the writer sized the buffer with the true length but only memcpy'd len & 0xFFFF bytes, so the blob tail after the wrapped length is uninitialized memory. A verified workaround is to back up data/vectordb, then delete the corrupt KV pairs directly via StoreManager.storage.delete(keys, table) for both DeltaTable and CandsTable; vectors for the healthy records live in CandsTable, so no re-embedding is needed.

Environment

• OpenViking: ghcr.io/volcengine/openviking:latest (image created 2026-05-15) for the corruption; code inspection of current main for the still-broken delete/upsert path
• Storage: local (PersistCollection + RocksDB store), engine variant x86_avx512
• OS: Linux (Debian, kernel 6.10), Python 3.13 (in-image)

References

• BUG: Unterminated string JSON parse error in VikingVectorIndexBackend query #2117 (root cause confirmed), fix(vectordb): reject oversized bytes row strings #2171 (write-side fail-fast, merged)
• PersistCollection._recover aborts on a single malformed delta record; deferred init turns this into silent 500s #1892 — "Bug 1 needs more discussion — data integrity tradeoffs deserve their own thread"; this issue is that thread
• fix(vectordb): skip candidates with corrupted JSON fields in search / 检索时跳过 fields 损坏的候选 #2302 (search-side skip, merged)