Add KnnRecallRunner to Rally core

[davidkyle]

There are 5 implementations of KnnRecallRunner in Rally tracks all of which are very similar. The functionality is obviously useful and moving it into Rally would enable reuse of the code.

An alternative is to create a shared class the rally tracks repo, see elastic/rally-tracks#1186 (comment)

1. https://github.com/elastic/rally-tracks/blob/8848994edbc299ad7f43835a362ef6eafd393d29/openai_vector/track.py#L123
2. https://github.com/elastic/rally-tracks/blob/8848994edbc299ad7f43835a362ef6eafd393d29/dense_vector/track.py#L205
3. https://github.com/elastic/rally-tracks/blob/8848994edbc299ad7f43835a362ef6eafd393d29/msmarco-v2-vector/track.py#L187
4. https://github.com/elastic/rally-tracks/blob/8848994edbc299ad7f43835a362ef6eafd393d29/wiki_en_cohere_vector_int8/track.py#L140
5. https://github.com/elastic/rally-tracks/blob/8848994edbc299ad7f43835a362ef6eafd393d29/so_vector/track.py#L235

[pri12ya871]

Hi @davidkyle — I'd like to pick this up. Before writing anything I wanted to check direction, because the choice between the two options seems to depend on how much of the divergence between the five implementations we want to absorb.

I diffed them at 8848994. They share the same skeleton (iterate query vectors → knn search → intersect hits with ground truth → avg/min recall), but a few differences aren't cosmetic:

Ground truth source. openai_vector and wiki_en_cohere_vector_int8 read precomputed neighbors from a loaded store — get_neighbors_for_query(query_id, k), synchronous. dense_vector instead resolves ground truth by querying the cluster: get_neighbors_for_query(index, query_id, target_k, request_cache, es) — async, and it also carries target_k and an invalidate_vector_store flag.

Document identity. dense_vector intersects on hit["_id"]; the others use a docid field — retrieved via docvalue_fields in openai_vector and wiki_en_cohere_vector_int8, but via fields in msmarco-v2-vector.

Query shape. The vector field is emb in three tracks and vector in dense_vector. msmarco-v2-vector supports visit_percentage as an alternative to num_candidates, names its oversample parameter oversample_rescore rather than oversample, and places rescore_vector directly on the knn clause where openai_vector nests it one level deeper.

Returned metrics. dense_vector runs with profile: true and additionally reports avg_nodes_visited and 99th_percentile_nodes_visited, but no max_recall — which openai_vector and wiki_en_cohere_vector_int8 both report.

So, three questions:

1. Rally core, or the shared class in rally-tracks? Since ground truth retrieval reaches into a param source — and dense_vector's variant needs the es client — a core runner would need that dependency defined at the Rally level.
2. Does the companion KnnRecallParamSource / KnnVectorStore move as well, or should the runner stay agnostic and receive neighbors through params? That mostly determines the size of the change.
3. For the behaviors that differ, would you rather they all become options on a single runner, or standardize on one shape and update the tracks to match?

Happy to go whichever way you prefer.

[davidkyle]

Sorry @pri12ya871 I missed the notification on your comment.

The goal here is to remove the duplicated code and to make it easy to add a knn recall test without copy-pasting code. With that in mind:

1. Rally core makes sense as the best place to write shared code.
2. KnnRecallParamSource / KnnVectorStore can go in core too. Individual tracks still have the option of creating a new *ParamSource or *VectorRecall class for their specific needs if the core class does not satisfy those needs.
3. Options on a single runner is the preferred option. If it is not possible to standardise to a single runner then individual tracks can be left untouched and continue using their existing code. The tracks should not be updated to fit a standard runner.