Questions about FlowER and out-of-sample datasets

[HaCTang]

Hi,

Thank you for your outstanding work on FlowER, recently published in Nature. It has been highly inspiring for our ongoing research.
I have a few questions and requests regarding your paper:

In Fig. 2a, models such as G2S, G2S+H, and MT seem to perform poorly, especially for atom, proton, and electro tasks, whereas they show much better results in top-K step accuracy and top-K pathway accuracy in Fig. 2b and 2c. Could you please clarify the reason behind this discrepancy? I would also greatly appreciate it if you could share the code used to generate the result and these figures.

You mentioned 22,000 out-of-sample reactions and a 32-reaction fine-tuned dataset in the paper. Would it be possible to access the corresponding dataset and code so that we can evaluate our model under the same settings?

Thank you very much for your time and for sharing such an inspiring piece of research.

Best regards,
Haocheng

[FongMunHong]

Hi,

Thank you for your interest in our work.

In Fig. 2a, models such as G2S, G2S+H, and MT seem to perform poorly, especially for atom, proton, and electro tasks, whereas they show much better results in top-K step accuracy and top-K pathway accuracy in Fig. 2b and 2c. Could you please clarify the reason behind this discrepancy?

When we compute top k step/pathway accuracy, we assume the best case of G2S and MT, where a simple filter check is applied to check for atom and electron mismatch, thus there are able to perform almost as well as FlowER, however when inspecting inputs most of them do suffer of hallucination.

I would also greatly appreciate it if you could share the code used to generate the result and these figures

sequence_evaluation.py is used to generate the top k results I believe. As for figures I have to check if I still have the code for it.

You mentioned 22,000 out-of-sample reactions and a 32-reaction fine-tuned dataset in the paper. Would it be possible to access the corresponding dataset and code so that we can evaluate our model under the same settings?

We are planning to upload it soon. They are mostly data from USPTO and mapped using RxnMapper.

Best,
Mun Hong