feat: department-filtered search for local document indexes (+2 PDF pipeline fixes)

[locphamnguyen]

Summary

Adds metadata-filtered search for indexes built from local document trees, and fixes two bugs in the local-PDF pipeline discovered while implementing it. Motivation: indexing an internal SOP corpus organized as docs/<department>/<file>.pdf and serving per-department filtered retrieval (RBAC-style) from one index.

1. Department metadata + filtered search

• pixelrag index build now records a department field per article in articles.json, derived from the first sub-directory under the local source root. Web/kiwix sources and files directly under the root get "" — fully backward compatible (additive key).
• POST /search accepts an optional department: vector positions for that department are pre-filtered inside FAISS via IDSelectorBatch (SearchParametersIVF on IVF indexes, plain SearchParameters otherwise), so n_docs results are guaranteed without over-fetch + post-filter. Position lists are cached per department.
• Unknown department → 404 listing valid names; index without department metadata → clear 400. New GET /departments endpoint returns departments with document counts.

2. Fix: unstable article ids for local PDF sources

render_pdf names tile dirs after the PDF filename stem, while embed/serve map vectors to articles.json by the sequential position index. Non-numeric stems fell back to hash(stem) % 2**31 (salted per process), silently breaking URL/path resolution for every local-PDF index. The pipeline now renames PDF tile dirs to the position index, matching the URL/text branches.

3. Fix: tile serving for PDF page images

_resolve_path only looked for materialized chunk_XXXX_YY.png files (Wikipedia layout). PDF pages are stored as whole tile_XXXX.jpg files, so include_images returned null and /tile/... 404'd for PDF indexes. Added a fallback to the page image, and /tile now returns the correct MIME type for JPEG.

Tests

• tests/test_department_filter.py: build-side directory→department mapping (nested dirs, root files, file:// URLs, out-of-root paths) and serve-side selector filtering on both IndexFlatIP and IndexIVFFlat, plus the 404/400 error paths. Full suite: 37 passed.
• Deployed end-to-end on a 45-document / 9-department corpus (Linux + CUDA): filtered results stay within the requested department; include_images and /tile serve PDF pages correctly.

[vercel]

@locphamnguyen is attempting to deploy a commit to the andylizf's projects Team on Vercel.

A member of the Team first needs to authorize it.