Benchmark questions

[jianshu93]

Dear PanSpace team,

I attached a comment in the bioRxiv page here (https://www.biorxiv.org/content/10.1101/2025.03.19.644115v2.abstract), related to benchmarks against other software. Let me know your thoughts/comments.

Best,
Jianshu

[jorgeavilacartes]

Dear Jianshu,
Sorry for the late reply.
Thanks for your detailed comment. We are addressing your suggestions and are currently working on including new benchmarks to the experiments.

Regarding the construction of the GSearch index using SetSketch, you mentioned that it is "very space efficient and also fast". The construction using ProbMinHash took around 25 hours, while the one with SetSketch is running for more than 4 days now. Do you think that there might be some issue?

Here are the commands we executed,

gsearch --pio 1000 --nbthreads 24 tohnsw -d $DIR_SEQUENCES -s 12000 -k 16 --ef 1600 -n 128 --algo prob

gsearch --pio 1000 --nbthreads 24 tohnsw -d $DIR_SEQUENCES -s 12000 -k 16 --ef 1600 -n 128 --algo hll

Thanks again.

Best,
Jorge.

[jianshu93]

Hi @jorgeavilacartes, for setsketch, sketch size that big is not helpful. If you check the pre-built database, setsketch size should never be larger than 4096. I would recommend the below one (newest bioconda version v0.2.9, we add a new HNSW option, FlatNAV/HubNSW): gsearch --pio 2000 --nbthreads 64 tohnsw -d ../files -k 16 -s 4096 -n 128 --ef 1600 --algo hll --scale_modify_f 0.25

And for optimal densification MinHash:
gsearch --pio 2000 --nbthreads 64 tohnsw -d ../files -k 16 -s 12000 -n 128 --ef 1600 --algo optdens --scale_modify_f 0.25

Jianshu

[jorgeavilacartes]

Hi @jianshu93 that's great. Thanks for your advice, we'll create new indexes with those commands and see how it goes.

Best,
Jorge.