Low Document-Level Hit Rate in local_search

[Li Fu (Furi5)]

I have built a document-level retrieval benchmark based on a corpus of over 1,000 scientific papers. The goal is to assess if local_search can retrieve the correct source document for a given factual question.

My evaluation shows a very low document-level Hit Rate, starting at 3.44% for Anson (@k)=1 and reaching only 21.63% at Anson (@k)=10. This suggests that for tasks requiring precise document provenance, the default local_search configuration may not be optimal.

My Methodology

1. Corpus: A collection of 1,000+ scientific papers in a specific domain [e.g., pharmacology].
2. Benchmark Construction: I created a set of question-document pairs. Each question is designed to have its answer contained within one specific "target document" in the corpus.
3. GraphRAG Indexing: I indexed the entire corpus using the graphrag --init command with the --init-method fast setting.
4. Retrieval Step: For each question in my benchmark, I run the local_search function.
5. Evaluation: I inspect the result.context_data["sources"] field from the local_search output. A "hit" is counted if the ID of the "target document" is present in this list of sources. The Hit Rate Anson (@k) is the percentage of questions for which the target document is found within the top k sources.

Results

Here is the Hit Rate table from my benchmark:

Hit Rate Michael Balaban (@1)	Hit Rate 2 (@2)	Hit Rate 3	Hit Rate bene (@4)	Hit Rate Yuriy Semchyshyn (@5)	Hit Rate 6 (@6)	Hit Rate 7	Hit Rate MartinK (@8)	Hit Rate 9	Hit Rate Anson (@10)
3.44%	8.33%	11.20%	13.11%	15.79%	17.03%	19.14%	20.38%	21.34%	21.63%