Add opt-in semantic search for every authenticated Mistral document format

[salmonumbrella]

Problem

Msgvault stores attachment bytes, but their contents are not searchable. Relevant evidence can be trapped inside PDFs, Office files, spreadsheets, EPUBs, structured text, source code, or email containers while message and filename search return nothing.

Proposed direction

Add an opt-in document indexing lane with separate consent for each hosted provider:

• classify attachment occurrences with authoritative role and source-part provenance;
• send eligible standalone documents to the stateless Mistral OCR endpoint using private inline data URLs;
• retain normalized source units, bounded chunks, citations, and lexical indexes locally while discarding raw provider JSON and full provider Markdown;
• generate document vectors with the configured Voyage text embedding profile;
• expose dedicated lexical and semantic document search with exact attachment, message, source-unit, extraction, and vector-generation provenance;
• hide stale results immediately on replacement, deletion, role change, or message lifecycle change.

Format scope

The target is every non-visual attachment format accepted by the pinned stateless Mistral OCR endpoint, not PDF alone. The authenticated capability probe must cover:

• PDF, DOCX/DOC, ODT, RTF;
• PPTX/PPT;
• XLSX/XLS, ODS, CSV, Numbers;
• EPUB;
• TXT, Markdown, RST, LaTeX;
• JSON/JSONL, XML, YAML;
• representative supported source-code formats;
• EML and MSG.

Mistral explicitly documents OCR document inputs including PDF, DOCX, and PPTX, while its broader upload products list additional families. Documentation is discovery evidence, not upload authority: each media type enters Msgvault only after a synthetic authenticated probe proves inline stateless input, stable bounded source units, safe failure behavior, and usable citations. Passing formats ship with family-specific caps and quality fixtures. Rejected formats remain visible as provider-unsupported; Msgvault must not silently create a Mistral File or Library as a fallback.

Standalone images, GIFs, and video stay in the separate Voyage multimodal issue. Audio remains a transcription concern. Archives, executables, encrypted inputs, and unknown containers fail closed.

Delivery boundary

This lands as a reviewable stack: consume the shared attachment-role foundation, add the common Mistral client/schema and lexical indexing, enable every passing format family, then add required Voyage vectors and semantic/RRF retrieval. Lexical-only operation is a useful fallback but does not complete this issue. Files-workspace integration can follow independently.

The common extraction and lexical phases can begin before #589 merges. Only the final vector integration depends on the final #589 interfaces. Keep this issue open until the complete format capability matrix and semantic retrieval are merged and verified.

Both hosted lanes default off and require independent consent. All committed fixtures and public artifacts must be synthetic and free of personal or private downstream data.

Official provider references:

• https://docs.mistral.ai/studio/document-processing/basic_ocr
• https://docs.mistral.ai/vibe/work/libraries

[salmonumbrella]

Implementation progress: the provider-independent extraction and lexical-search stack is active on the implementation branch, and the format scope remains the full 26-row non-visual candidate matrix described above—not PDF-only. A private synthetic corpus now covers every row, including real legacy DOC/PPT/XLS, Numbers, and Outlook MSG containers. All 26 fixtures pass production format detection and the documents probe-mistral --validate-only command without a provider request. The complete tagged Linux suite passes on the retained CrabBox VM.

Two gates remain before this issue can close: authenticated provider acceptance needs a purpose-scoped 1Password Environment ID for the repository-approved secret wrapper, and the required Voyage document-vector/RRF phase waits for #589 to merge. No provider key or private fixture has been committed or copied to the VM.

[salmonumbrella]

Authenticated provider gate is now complete against the pinned EU mistral-ocr-4-0 endpoint using the full 26-format synthetic matrix. The Office candidates were regenerated with real Pandoc/LibreOffice-authored containers before the final run.

Passing (18): PDF, DOC, ODT, RTF, PPT, XLSX, CSV, EPUB, TXT, Markdown, RST, JSON, JSONL, YAML, Go, Python, JavaScript, and MSG.

Provider-rejected (8): DOCX, PPTX, XLS, ODS, Numbers, LaTeX, XML, and EML. These rows remain in the complete manifest but cannot become runtime upload authority.

A separate isolated end-to-end run also passed: consented profile creation, standalone PDF reconciliation, one verified upload, one-unit extraction, atomic publication, lexical retrieval of the exact synthetic sentinel, and zero files left in the private spool. No raw provider response or full Markdown was retained. The extraction/lexical work is no longer blocked; only the separate Voyage semantic-vector phase remains gated on #589 merging.