Issues training custom model

[laura-raes]

Hi!

I’m experiencing some issues training a custom SquiggleNet model. Would it be possible to check whether I’m doing something wrong?

Classification using your pretrained models, on human and E. coli R9 reads

First I tried running SquiggleNet inference using your pretrained models, on human and E. coli nanopore R9 data. This gave very nice results: an accuracy of 83-86% (depending on the model).

I did assume that human = 1 and bacterial = 0 in these models. Is this correct?

Classification using a custom model, on human and SARS-CoV2 R9 reads

Then I tried training a model using human and SARS-CoV2 data. Classifying the test data using this custom model, resulted in an accuracy of only 3%.

I am not sure whether I executed your scripts correctly. It would be very much appreciated if you wanted to check whether I ran your scripts as intended.

1. Splitting up the data into training, validation and test datasets
   - I used an equal amount of target (= SARS-CoV2) and non-target (=human) reads (269507 reads)
   - 80% of the reads were randomly selected and allocated to the training dataset, another 10% were allocated to the validation dataset, the remaining reads were allocated to the test dataset
   - the remaining reads (not included in the 269507 reads I started with) were also allocated to the test dataset

2. Preprocessing the training data
   - python ./SquiggleNet/preprocess.py -gp sarscov2_train_readids.txt -gn human_train_readids.txt -i fast5_human_and_sarscov2 -o outfolder_train
   - this resulted in 21 neg_*.pt and 15 pos_*.pt data batches

3. Preprocessing the validation data
   - python ./SquiggleNet/preprocess.py -gp sarscov2_val_readids.txt -gn human_val_readids.txt -i fast5_human_and_sarscov2 -o outfolder_val
   - this resulted in 2 neg_*.pt and 1 pos_*.pt data batches

4. Training a custom model
   - I was confused as to how the trainer.py script had to be executed. The preprocessing-script resulted in multiple pytorch tensors, but only one file can be specified at once to execute the trainer-script I think?
   - I used 15 batches (each iteration one target, one non-target) to train the model. I used the --intermediate option to finetune the previous model after the first iteration. Is this how the trainer script is intended to be used?
   - python ./SquiggleNet/trainer.py -tt outfolder_train/pos_10000.pt -nt outfolder_train/neg_10000.pt -tv outfolder_val/pos_10000.pt -nv outfolder_val/neg_10000.pt -o trainedModel_b1.ckpt -e 3
   - python ./SquiggleNet/trainer.py -tt outfolder_train/pos_20000.pt -nt outfolder_train/neg_20000.pt -tv outfolder_val/pos_10000.pt -nv outfolder_val/neg_10000.pt -i ./trainedModel_b1.ckpt -e 3 -o trainedModel_b2.ckpt
   - […]
   - python ./SquiggleNet/trainer.py -tt outfolder_train/pos_150000.pt -nt outfolder_train/neg_150000.pt -tv outfolder_val/pos_10000.pt -nv outfolder_val/neg_10000.pt -i ./trainedModel_b14.ckpt -e 3 -o trainedModel_b15.ckpt

5. Classifying the test data using the custom model
   - python ./SquiggleNet/inference.py -m trainedModel_b16.ckpt -i fast5_human_and_sarscov2_testdata/ -o classification_results_trainedModel_b16/
   - only 3% of the reads were allocated to the correct class

Any help would be greatly appreciated!

[Anbyew]

Hello, Thank you for your interests in SquiggleNet.
Your scripts look reasonable.

One thing that I noticed is the consistency of labeling, i.e. what is target (1) vs non-target (0).
For the human VS E. coli nanopore R9 data, it was mentioned that you assumed "human = 1 (target) and bacterial = 0 (non-target)", whereas for the human and SARS-CoV2 data, it was assumed "target (= SARS-CoV2) and non-target (=human)".

It could go either way, depending on which on set of sequences you would like to call 'target', as long as the prediction labels are consistent with the ground truth labeling.

(A different angle to interpret the 3% accuracy: it is a binary classifier. The worst performance a model could get would be random guessing -- 50% (on balanced data). If the accuracy is only 3% then you could flip the labels and get 97% in accuracy.)

Hope it helps but please feel free to follow up if you have more questions!

[laura-raes]

Hi,

Thanks a lot for taking the time to write a reply! I really appreciate it.

Concerning the consistency of the labeling: it is confusing indeed, I’m sorry! This happened because I’m talking about two different models: your models, for which I guessed the labeling, and my own models, for which I was free to choose the labeling. I thought defining SARS-CoV2 as target made more sense from a practical point of view.

Thank you for looking at my script!

I read your comment about flipping the labels and getting a very high accuracy. The testing dataset isn’t balanced, so the result of random guessing wouldn’t be 50% in this case. But the model still would perform a lot better if the labels were flipped. This is weird, since I’m quite sure about the labels...

I think something is going wrong, since the model I trained performs very poorly compared to your models.

I have some additional questions:

1. Is it correct that in your models (model_B4_2_3000_totmad_br_2.ckpt and model_B4t2_3000_tot32.ckpt) human = 1 and bacterial = 0?

2. I do not understand why (approximately) the same amount of read ids in the -gp sarscov2_train_readids.txt and the -gn human_train_readids.txtinput for the preprocessing script, can result in 21 neg_*.pt and 15 pos_*.pt files. Does this seem ok to you, or might something be going wrong there? I would expect the number of output pt-files to be equal?

3. Is my approach of first training an initial model with the first pt-files, and then finetuning this model iteratively with the other pt-files correct?

4. If this approach is correct: there are less validation pt-files than training pt-files, how do you choose which training-pt files and validation-pt files you will use together?

5. I also do not understand why all pt-files outputted by the same preprocess.py-run have an exactly equal size. I would expect them to be different in size, at least between the negative and positive samples? For example, this is the output of preprocessing the validation data:

neg_10000.pt - 120000753 bytes
neg_20000.pt - 120000753 bytes
pos_10000.pt - 120000753 bytes

6. In another Github Issue, I read it was possible to receive the fast5 datasets you used in your paper. Would this still be possible?

Thank you!