feat(search): hybrid retrieval + FTS/vector indexes — removes the search scaling cliff#934
Closed
seonghobae wants to merge 1 commit into
Closed
feat(search): hybrid retrieval + FTS/vector indexes — removes the search scaling cliff#934seonghobae wants to merge 1 commit into
seonghobae wants to merge 1 commit into
Conversation
…ts the scalability plan) /api/search previously computed ts_rank_cd minus cosine_distance over every owner-scoped row on every query — a full sequential scan that no index could serve (verified by EXPLAIN: the planner ignored GIN/HNSW even when present) and an unnormalized fusion of incomparable scales. Reshape (docs/plans/2026-07-05-search-scalability-hybrid-retrieval.md): - Candidate generation is now up to four index-eligible arms: lexical email/ attachment arms gated by @@ (GIN-served, ts_rank_cd ordered, LIMIT) and vector email/attachment arms as pure cosine-distance ORDER BY + LIMIT (HNSW-served). The embedding-failure fallback keeps working (lexical arms only). - Fusion moves to the app as reciprocal-rank fusion (RRF, k=60), replacing the score subtraction. Metadata + reply counts are fetched in a second bounded query for the fused candidate ids only, so joins never disqualify the ANN index. - Owner (user/organization) scoping preserved on every arm and the metadata query. Migration 0010 (postgresql-guarded, CONCURRENTLY in an autocommit block): GIN expression indexes on email/attachment text and hnsw (ivfflat below pgvector 0.5) vector_cosine_ops indexes, with the DDL shared between the migration and tests via db/search_indexes.py. Verification: - tests/test_search_pg_indexes.py (postgres marker) ran against a real pgvector 16 instance: EXPLAIN shows the lexical arm using ix_email_records_body_fts and the vector arm using ix_email_records_embedding_vec. - tests/test_search.py rewritten for the new shape (16 passed): endpoint behavior, owner scoping, fallback, session split, RRF fusion, arm shapes. - Regressions green: alembic/bootstrap (37), emails_api/db_session (66), ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RkKdtHRLG4wSLh6PVsp8J
This was referenced Jul 6, 2026
Contributor
Author
|
Folded into #938, which is now retargeted to |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current headc30e005c5077b97b4f1a799493ab07e8ec7c3f6f. -
Head SHA:
c30e005c5077b97b4f1a799493ab07e8ec7c3f6f -
Workflow run: 28767585117
-
Workflow attempt: 1
Coverage evidence
Coverage Evidence
- Head SHA:
c30e005c5077b97b4f1a799493ab07e8ec7c3f6f - Required test evidence: supported repository test suites must pass.
- Required docstring evidence: repository-owned docstring gates must pass when configured; otherwise docstring coverage is advisory.
Python project dependencies (backend/requirements.txt)
Defaulting to user installation because normal site-packages is not writeable
Collecting fastapi==0.138.2 (from -r requirements.txt (line 1))
Downloading fastapi-0.138.2-py3-none-any.whl.metadata (26 kB)
Collecting starlette==1.3.1 (from -r requirements.txt (line 2))
Downloading starlette-1.3.1-py3-none-any.whl.metadata (6.4 kB)
Collecting uvicorn==0.49.0 (from -r requirements.txt (line 3))
Downloading uvicorn-0.49.0-py3-none-any.whl.metadata (6.7 kB)
Requirement already satisfied: pytest==9.1.1 in /home/runner/.local/lib/python3.12/site-packages (from -r requirements.txt (line 4)) (9.1.1)
Collecting httpx==0.28.1 (from -r requirements.txt (line 5))
Downloading httpx-0.28.1-py3-none-any.whl.metadata (7.1 kB)
Collecting pydantic-settings==2.14.2 (from -r requirements.txt (line 6))
Downloading pydantic_settings-2.14.2-py3-none-any.whl.metadata (3.4 kB)
Collecting aiosmtplib==5.1.2 (from -r requirements.txt (line 7))
Downloading aiosmtplib-5.1.2-py3-none-any.whl.metadata (3.6 kB)
Collecting aioimaplib==2.0.1 (from -r requirements.txt (line 8))
Downloading aioimaplib-2.0.1-py3-none-any.whl.metadata (9.5 kB)
Collecting sqlalchemy==2.0.51 (from -r requirements.txt (line 9))
Downloading sqlalchemy-2.0.51-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (9.5 kB)
Collecting alembic==1.18.5 (from -r requirements.txt (line 10))
Downloading alembic-1.18.5-py3-none-any.whl.metadata (7.2 kB)
Collecting greenlet==3.5.3 (from -r requirements.txt (line 11))
Downloading greenlet-3.5.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (3.8 kB)
Collecting asyncpg==0.31.0 (from -r requirements.txt (line 12))
Downloading asyncpg-0.31.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (4.4 kB)
Collecting pgvector==0.4.2 (from -r requirements.txt (line 13))
Downloading pgvector-0.4.2-py3-none-any.whl.metadata (19 kB)
Collecting pytest-asyncio==1.4.0 (from -r requirements.txt (line 14))
Downloading pytest_asyncio-1.4.0-py3-none-any.whl.metadata (4.1 kB)
Collecting openai==2.44.0 (from -r requirements.txt (line 15))
Downloading openai-2.44.0-py3-none-any.whl.metadata (34 kB)
Collecting langchain-text-splitters==1.1.2 (from -r requirements.txt (line 16))
Downloading langchain_text_splitters-1.1.2-py3-none-any.whl.metadata (3.3 kB)
Collecting tiktoken==0.13.0 (from -r requirements.txt (line 17))
Downloading tiktoken-0.13.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (6.7 kB)
Collecting google-api-python-client==2.198.0 (from -r requirements.txt (line 18))
Downloading google_api_python_client-2.198.0-py3-none-any.whl.metadata (7.0 kB)
Collecting google-auth-httplib2==0.4.0 (from -r requirements.txt (line 19))
Downloading google_auth_httplib2-0.4.0-py3-none-any.whl.metadata (3.0 kB)
Collecting google-auth-oauthlib==1.4.0 (from -r requirements.txt (line 20))
Downloading google_auth_oauthlib-1.4.0-py3-none-any.whl.metadata (2.6 kB)
Collecting email-validator==2.3.0 (from -r requirements.txt (line 21))
Downloading email_validator-2.3.0-py3-none-any.whl.metadata (26 kB)
Collecting cryptography==49.0.0 (from -r requirements.txt (line 22))
Downloading cryptography-49.0.0-cp311-abi3-manylinux_2_34_x86_64.whl.metadata (4.3 kB)
Collecting python-multipart==0.0.32 (from -r requirements.txt (line 23))
Downloading python_multipart-0.0.32-py3-none-any.whl.metadata (2.1 kB)
Collecting prometheus-fastapi-instrumentator==8.0.2 (from -r requirements.txt (line 24))
Downloading prometheus_fastapi_instrumentator-8.0.2-py3-none-any.whl.metadata (13 kB)
Collecting opentelemetry-api==1.43.0 (from -r requirements.txt (line 25))
Downloading opentelemetry_api-1.43.0-py3-none-any.whl.metadata (1.4 kB)
Collecting opentelemetry-sdk==1.43.0 (from -r requirements.txt (line 26))
Downloading opentelemetry_sdk-1.43.0-py3-none-any.whl.metadata (1.7 kB)
Collecting opentelemetry-instrumentation-fastapi==0.64b0 (from -r requirements.txt (line 27))
Downloading opentelemetry_instrumentation_fastapi-0.64b0-py3-none-any.whl.metadata (2.2 kB)
Collecting opentelemetry-exporter-otlp==1.43.0 (from -r requirements.txt (line 28))
Downloading opentelemetry_exporter_otlp-1.43.0-py3-none-any.whl.metadata (2.4 kB)
Collecting protobuf==7.35.1 (from -r requirements.txt (line 29))
Downloading protobuf-7.35.1-cp310-abi3-manylinux2014_x86_64.whl.metadata (595 bytes)
Collecting setuptools==82.0.1 (from -r requirements.txt (line 30))
Downloading setuptools-82.0.1-py3-none-any.whl.metadata (6.5 kB)
Collecting wheel==0.47.0 (from -r requirements.txt (line 31))
Downloading wheel-0.47.0-py3-none-any.whl.metadata (2.3 kB)
Collecting websockets==16.0 (from -r requirements.txt (line 32))
Downloading websockets-16.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (6.8 kB)
Collecting PyJWT==2.13.0 (from -r requirements.txt (line 33))
Downloading pyjwt-2.13.0-py3-none-any.whl.metadata (3.4 kB)
Collecting icalendar==7.2.0 (from -r requirements.txt (line 34))
Downloading icalendar-7.2.0-py3-none-any.whl.metadata (6.9 kB)
Collecting ruff==0.15.20 (from -r requirements.txt (line 35))
Downloading ruff-0.15.20-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (26 kB)
Collecting defusedxml==0.7.1 (from -r requirements.txt (line 36))
Downloading defusedxml-0.7.1-py2.py3-none-any.whl.metadata (32 kB)
Collecting pydantic>=2.9.0 (from fastapi==0.138.2->-r requirements.txt (line 1))
Downloading pydantic-2.13.4-py3-none-any.whl.metadata (109 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.4/109.4 kB 21.8 MB/s eta 0:00:00
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/lib/python3/dist-packages (from fastapi==0.138.2->-r requirements.txt (line 1)) (4.10.0)
Collecting typing-inspection>=0.4.2 (from fastapi==0.138.2->-r requirements.txt (line 1))
Downloading typing_inspection-0.4.2-py3-none-any.whl.metadata (2.6 kB)
Collecting annotated-doc>=0.0.2 (from fastapi==0.138.2->-r requirements.txt (line 1))
Downloading annotated_doc-0.0.4-py3-none-any.whl.metadata (6.6 kB)
Collecting anyio<5,>=3.6.2 (from starlette==1.3.1->-r requirements.txt (line 2))
Downloading anyio-4.14.1-py3-none-any.whl.metadata (4.6 kB)
Requirement already satisfied: click>=7.0 in /usr/lib/python3/dist-packages (from uvicorn==0.49.0->-r requirements.txt (line 3)) (8.1.6)
Collecting h11>=0.8 (from uvicorn==0.49.0->-r requirements.txt (line 3))
Downloading h11-0.16.0-py3-none-any.whl.metadata (8.3 kB)
Requirement already satisfied: iniconfig>=1.0.1 in /home/runner/.local/lib/python3.12/site-packages (from pytest==9.1.1->-r requirements.txt (line 4)) (2.3.0)
Requirement already satisfied: packaging>=22 in /usr/lib/python3/dist-packages (from pytest==9.1.1->-r requirements.txt (line 4)) (24.0)
Requirement already satisfied: pluggy<2,>=1.5 in /home/runner/.local/lib/python3.12/site-packages (from pytest==9.1.1->-r requirements.txt (line 4)) (1.6.0)
Requirement already satisfied: pygments>=2.7.2 in /usr/lib/python3/dist-packages (from pytest==9.1.1->-r requirements.txt (line 4)) (2.17.2)
Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from httpx==0.28.1->-r requirements.txt (line 5)) (2023.11.17)
Collecting httpcore==1.* (from httpx==0.28.1->-r requirements.txt (line 5))
Downloading httpcore-1.0.9-py3-none-any.whl.metadata (21 kB)
Requirement already satisfied: idna in /usr/lib/python3/dist-packages (from httpx==0.28.1->-r requirements.txt (line 5)) (3.6)
Collecting python-dotenv>=0.21.0 (from pydantic-settings==2.14.2->-r requirements.txt (line 6))
Downloading python_dotenv-1.2.2-py3-none-any.whl.metadata (27 kB)
Collecting Mako (from alembic==1.18.5->-r requirements.txt (line 10))
Downloading mako-1.3.12-py3-none-any.whl.metadata (2.9 kB)
Collecting typing-extensions>=4.8.0 (from fastapi==0.138.2->-r requirements.txt (line 1))
Downloading typing_extensions-4.16.0-py3-none-any.whl.metadata (3.3 kB)
Collecting numpy (from pgvector==0.4.2->-r requirements.txt (line 13))
Downloading numpy-2.5.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (6.6 kB)
Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai==2.44.0->-r requirements.txt (line 15)) (1.9.0)
Collecting jiter<1,>=0.10.0 (from openai==2.44.0->-r requirements.txt (line 15))
Downloading jiter-0.16.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.2 kB)
Collecting sniffio (from openai==2.44.0->-r requirements.txt (line 15))
Downloading sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)
Collecting tqdm>4 (from openai==2.44.0->-r requirements.txt (line 15))
Downloading tqdm-4.68.3-py3-none-any.whl.metadata (57 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.4/57.4 kB 21.0 MB/s eta 0:00:00
Collecting langchain-core<2.0.0,>=1.2.31 (from langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading langchain_core-1.4.8-py3-none-any.whl.metadata (4.7 kB)
Collecting regex (from tiktoken==0.13.0->-r requirements.txt (line 17))
Downloading regex-2026.6.28-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (40 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.5/40.5 kB 9.0 MB/s eta 0:00:00
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from tiktoken==0.13.0->-r requirements.txt (line 17)) (2.31.0)
Requirement already satisfied: httplib2<1.0.0,>=0.19.0 in /usr/lib/python3/dist-packages (from google-api-python-client==2.198.0->-r requirements.txt (line 18)) (0.20.4)
Collecting google-auth!=2.24.0,!=2.25.0,<3.0.0,>=1.32.0 (from google-api-python-client==2.198.0->-r requirements.txt (line 18))
Downloading google_auth-2.55.1-py3-none-any.whl.metadata (5.1 kB)
Collecting google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0,>=1.31.5 (from google-api-python-client==2.198.0->-r requirements.txt (line 18))
Downloading google_api_core-2.31.0-py3-none-any.whl.metadata (3.2 kB)
Collecting uritemplate<5,>=3.0.1 (from google-api-python-client==2.198.0->-r requirements.txt (line 18))
Downloading uritemplate-4.2.0-py3-none-any.whl.metadata (2.6 kB)
Collecting requests-oauthlib>=0.7.0 (from google-auth-oauthlib==1.4.0->-r requirements.txt (line 20))
Downloading requests_oauthlib-2.0.0-py2.py3-none-any.whl.metadata (11 kB)
Collecting dnspython>=2.0.0 (from email-validator==2.3.0->-r requirements.txt (line 21))
Downloading dnspython-2.8.0-py3-none-any.whl.metadata (5.7 kB)
Collecting cffi>=2.0.0 (from cryptography==49.0.0->-r requirements.txt (line 22))
Downloading cffi-2.0.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.6 kB)
Collecting prometheus-client<1.0.0,>=0.8.0 (from prometheus-fastapi-instrumentator==8.0.2->-r requirements.txt (line 24))
Downloading prometheus_client-0.25.0-py3-none-any.whl.metadata (2.1 kB)
Collecting opentelemetry-semantic-conventions==0.64b0 (from opentelemetry-sdk==1.43.0->-r requirements.txt (line 26))
Downloading opentelemetry_semantic_conventions-0.64b0-py3-none-any.whl.metadata (2.4 kB)
Collecting opentelemetry-instrumentation-asgi==0.64b0 (from opentelemetry-instrumentation-fastapi==0.64b0->-r requirements.txt (line 27))
Downloading opentelemetry_instrumentation_asgi-0.64b0-py3-none-any.whl.metadata (2.0 kB)
Collecting opentelemetry-instrumentation==0.64b0 (from opentelemetry-instrumentation-fastapi==0.64b0->-r requirements.txt (line 27))
Downloading opentelemetry_instrumentation-0.64b0-py3-none-any.whl.metadata (7.2 kB)
Collecting opentelemetry-util-http==0.64b0 (from opentelemetry-instrumentation-fastapi==0.64b0->-r requirements.txt (line 27))
Downloading opentelemetry_util_http-0.64b0-py3-none-any.whl.metadata (2.6 kB)
Collecting opentelemetry-exporter-otlp-proto-grpc==1.43.0 (from opentelemetry-exporter-otlp==1.43.0->-r requirements.txt (line 28))
Downloading opentelemetry_exporter_otlp_proto_grpc-1.43.0-py3-none-any.whl.metadata (2.6 kB)
Collecting opentelemetry-exporter-otlp-proto-http==1.43.0 (from opentelemetry-exporter-otlp==1.43.0->-r requirements.txt (line 28))
Downloading opentelemetry_exporter_otlp_proto_http-1.43.0-py3-none-any.whl.metadata (2.4 kB)
Requirement already satisfied: python-dateutil in /usr/lib/python3/dist-packages (from icalendar==7.2.0->-r requirements.txt (line 34)) (2.8.2)
Collecting tzdata>=2025.3 (from icalendar==7.2.0->-r requirements.txt (line 34))
Downloading tzdata-2026.2-py2.py3-none-any.whl.metadata (1.4 kB)
Collecting googleapis-common-protos~=1.57 (from opentelemetry-exporter-otlp-proto-grpc==1.43.0->opentelemetry-exporter-otlp==1.43.0->-r requirements.txt (line 28))
Downloading googleapis_common_protos-1.75.0-py3-none-any.whl.metadata (8.6 kB)
Collecting grpcio<2.0.0,>=1.63.2 (from opentelemetry-exporter-otlp-proto-grpc==1.43.0->opentelemetry-exporter-otlp==1.43.0->-r requirements.txt (line 28))
Downloading grpcio-1.82.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (3.7 kB)
Collecting opentelemetry-exporter-otlp-proto-common==1.43.0 (from opentelemetry-exporter-otlp-proto-grpc==1.43.0->opentelemetry-exporter-otlp==1.43.0->-r requirements.txt (line 28))
Downloading opentelemetry_exporter_otlp_proto_common-1.43.0-py3-none-any.whl.metadata (1.8 kB)
Collecting opentelemetry-proto==1.43.0 (from opentelemetry-exporter-otlp-proto-grpc==1.43.0->opentelemetry-exporter-otlp==1.43.0->-r requirements.txt (line 28))
Downloading opentelemetry_proto-1.43.0-py3-none-any.whl.metadata (2.3 kB)
Collecting wrapt<3.0.0,>=1.0.0 (from opentelemetry-instrumentation==0.64b0->opentelemetry-instrumentation-fastapi==0.64b0->-r requirements.txt (line 27))
Downloading wrapt-2.2.2-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (7.4 kB)
Collecting asgiref~=3.0 (from opentelemetry-instrumentation-asgi==0.64b0->opentelemetry-instrumentation-fastapi==0.64b0->-r requirements.txt (line 27))
Downloading asgiref-3.11.1-py3-none-any.whl.metadata (9.3 kB)
Collecting pycparser (from cffi>=2.0.0->cryptography==49.0.0->-r requirements.txt (line 22))
Downloading pycparser-3.0-py3-none-any.whl.metadata (8.2 kB)
Collecting proto-plus<2.0.0,>=1.24.0 (from google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0,>=1.31.5->google-api-python-client==2.198.0->-r requirements.txt (line 18))
Downloading proto_plus-1.28.0-py3-none-any.whl.metadata (2.2 kB)
Collecting requests (from tiktoken==0.13.0->-r requirements.txt (line 17))
Downloading requests-2.34.2-py3-none-any.whl.metadata (4.8 kB)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/lib/python3/dist-packages (from google-auth!=2.24.0,!=2.25.0,<3.0.0,>=1.32.0->google-api-python-client==2.198.0->-r requirements.txt (line 18)) (0.2.8)
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /usr/lib/python3/dist-packages (from httplib2<1.0.0,>=0.19.0->google-api-python-client==2.198.0->-r requirements.txt (line 18)) (3.1.1)
Collecting jsonpatch<2.0.0,>=1.33.0 (from langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading jsonpatch-1.33-py2.py3-none-any.whl.metadata (3.0 kB)
Collecting langchain-protocol>=0.0.17 (from langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading langchain_protocol-0.0.18-py3-none-any.whl.metadata (2.4 kB)
Collecting langsmith<1.0.0,>=0.3.45 (from langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading langsmith-0.9.7-py3-none-any.whl.metadata (22 kB)
Requirement already satisfied: pyyaml<7.0.0,>=5.3.0 in /usr/lib/python3/dist-packages (from langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16)) (6.0.1)
Collecting tenacity!=8.4.0,<10.0.0,>=8.1.0 (from langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading tenacity-9.1.4-py3-none-any.whl.metadata (1.2 kB)
Collecting uuid-utils<1.0,>=0.12.0 (from langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading uuid_utils-0.16.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.4 kB)
Collecting annotated-types>=0.6.0 (from pydantic>=2.9.0->fastapi==0.138.2->-r requirements.txt (line 1))
Downloading annotated_types-0.7.0-py3-none-any.whl.metadata (15 kB)
Collecting pydantic-core==2.46.4 (from pydantic>=2.9.0->fastapi==0.138.2->-r requirements.txt (line 1))
Downloading pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting charset_normalizer<4,>=2 (from requests->tiktoken==0.13.0->-r requirements.txt (line 17))
Downloading charset_normalizer-3.4.7-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (40 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.9/40.9 kB 131.0 kB/s eta 0:00:00
Requirement already satisfied: urllib3<3,>=1.26 in /usr/lib/python3/dist-packages (from requests->tiktoken==0.13.0->-r requirements.txt (line 17)) (2.0.7)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/lib/python3/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib==1.4.0->-r requirements.txt (line 20)) (3.2.2)
Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/lib/python3/dist-packages (from Mako->alembic==1.18.5->-r requirements.txt (line 10)) (2.1.5)
Requirement already satisfied: jsonpointer>=1.9 in /usr/lib/python3/dist-packages (from jsonpatch<2.0.0,>=1.33.0->langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16)) (2.0)
Collecting orjson>=3.9.14 (from langsmith<1.0.0,>=0.3.45->langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading orjson-3.11.9-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (41 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.0/42.0 kB 11.9 MB/s eta 0:00:00
Collecting requests-toolbelt>=1.0.0 (from langsmith<1.0.0,>=0.3.45->langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading requests_toolbelt-1.0.0-py2.py3-none-any.whl.metadata (14 kB)
Collecting xxhash>=3.0.0 (from langsmith<1.0.0,>=0.3.45->langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading xxhash-3.8.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (14 kB)
Collecting zstandard>=0.23.0 (from langsmith<1.0.0,>=0.3.45->langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading zstandard-0.25.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (3.3 kB)
Downloading fastapi-0.138.2-py3-none-any.whl (129 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.3/129.3 kB 35.7 MB/s eta 0:00:00
Downloading starlette-1.3.1-py3-none-any.whl (73 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 73.6/73.6 kB 12.9 MB/s eta 0:00:00
Downloading uvicorn-0.49.0-py3-none-any.whl (71 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.4/71.4 kB 21.4 MB/s eta 0:00:00
Downloading httpx-0.28.1-py3-none-any.whl (73 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 73.5/73.5 kB 13.0 MB/s eta 0:00:00
Downloading pydantic_settings-2.14.2-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.7/61.7 kB 5.7 MB/s eta 0:00:00
Downloading aiosmtplib-5.1.2-py3-none-any.whl (28 kB)
Downloading aioimaplib-2.0.1-py3-none-any.whl (34 kB)
Downloading sqlalchemy-2.0.51-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (3.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 159.6 MB/s eta 0:00:00
Downloading alembic-1.18.5-py3-none-any.whl (264 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 264.7/264.7 kB 77.6 MB/s eta 0:00:00
Downloading greenlet-3.5.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (614 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 614.2/614.2 kB 120.6 MB/s eta 0:00:00
Downloading asyncpg-0.31.0-cp312-cp312-manylinux_2_28_x86_64.whl (3.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 168.3 MB/s eta 0:00:00
Downloading pgvector-0.4.2-py3-none-any.whl (27 kB)
Downloading pytest_asyncio-1.4.0-py3-none-any.whl (16 kB)
Downloading openai-2.44.0-py3-none-any.whl (1.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 151.3 MB/s eta 0:00:00
- Result: PASS
Python coverage with missing-line report (backend)
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.1.1, pluggy-1.6.0
rootdir: /home/runner/work/naruon/naruon/pr-head/backend
configfile: pytest.ini
## Changed-File Evidence Map
```mermaid
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Backend (5 files)"]
S1 --> I1["API and service runtime"]
I1 --> R1["Review risk: Backend (5 files)"]
R1 --> V1["backend tests"]
Contributor
OpenCode Review Overview
Pull request overviewOpenCode cannot approve yet because required coverage evidence did not pass. Review outcome1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
Coverage evidenceCoverage Evidence
Python project dependencies (backend/requirements.txt)
Python coverage with missing-line report (backend) |
seonghobae
added a commit
that referenced
this pull request
Jul 11, 2026
…-answer feat(search): hybrid retrieval + FTS/vector indexes + grounded cited answer (folds #934)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What (implements #922's plan)
/api/searchseq-scanned every owner-scoped row per query — proven index-blind (EXPLAIN ignored GIN+HNSW even when present) because of query shape: no@@predicate and FTS+vector fused in one ORDER BY of incomparable scales.Reshape
@@-gated → GIN,ts_rank_cdordered, LIMIT) + vector email/attachment (pure<=>ORDER BY + LIMIT → HNSW). Embedding-failure fallback preserved (lexical-only).ts_rank_cd − cosine_distancesubtraction.Migration 0010
GIN expression indexes (email body, attachment content) +
hnsw vector_cosine_ops(ivfflat < pgvector 0.5),CONCURRENTLYin an autocommit block, postgres-guarded. DDL is a single source (db/search_indexes.py) shared by migration and tests.Verification
tests/test_search_pg_indexes.pyran against pgvector:pg16 — EXPLAIN shows the lexical arm onix_email_records_body_ftsand the vector arm onix_email_records_embedding_vec. ✅tests/test_search.pyrewritten for the new shape — 16 passed (endpoint, owner scoping, fallback, session split, RRF, arm shapes).Behavior notes for review
Result ranking and membership change by design (that's the fix): lexical candidates now require an actual FTS match; semantic-only hits still surface via the vector arms. Scores are RRF values (rank-based), not raw score arithmetic.
🤖 Generated with Claude Code