Training models have exact same AUPRC for each epoch and assigns each particle the same precision recall score

[gaandree]

Hello!

I am using Topaz as a standalone program and there seems to be something going on when training the models. I have a dataset of 2480 micrographs, and have picked a total of 1009 particles (using EMAN2), then split out a test set with 220 particles (21.8% total picked particles). I then trained 6 models with 10 epochs, each differing only by the number or particles/image (I tried n=10, 50, 100, 200, 300, 500). For every epoch in each model, the AUPRC is exactly the same and also pretty low (1.57444e-05, also shown in screenshot below).

This was pretty surprising, so I double checked my input files to confirm they were what I thought they were before extracting a random epoch from a model to see what it was picking (I choose modelc, epoch 10).

It extracted 10864881 particles, however when I tried to apply a score threshold (I used -2), there was nothing above this threshold. Next, I looked at the precision recall curves and it appears like every particle has the exact same score (-4.535172, a screenshot of the PRCcurve Jupyter notebook below)

Has anyone else seen this happen before? Or know what might be going on? I happy to provide more information on what I have done/clarify anything I wrote above. Thanks in advance for any help!

[tbepler]

That's strange. It looks like the model isn't training for some reason. Can you share your python environment to see the package versions you have installed and the exact command that you ran?

[gaandree]

Thank you so much for a quick response!

Here is the info about the python version and Topaz version that we have (we are using Topaz that is supported by SBGrid). Please let me know if this is not enough info/not the right info.
python --version
Python 2.7.2

python.topaz --version
Python 3.8.1

sbwhich topaz
/programs/x86_64-linux/topaz/0.2.5/bin/topaz

Here are all of the exact commands I ran involving Topaz, in the order that I ran them:
Preprocessing:
topaz preprocess --scale=4 --destdir=topazpreprocess1 MotionCorr/job002/movies/*_Fractions.mrc --verbose

Convert box files to topaz coordinates:
topaz boxes_to_coordinates boxfiles/* --imagedir=MotionCorr/job002/movies --image-ext=mrc --output=topaztrain/MotionCorr_topaz_coordinates1.txt

Scale coordinates to match preprocess scale:
topaz scale_coordinates topaztrain/MotionCorr_topaz_coordinates1.txt --scale=0.25 > topaztrain/topaz_scaled_coordinates1.txt
 
Generate image list:
python ~/../ed/topaz/scripts/generate_image_list.py topazpreprocess1/*.mrc > topaztrain/full_image_list.txt

Split into the test & train sets:
python ~/../ed/topaz/scripts/train_test_split.py --images=topaztrain/full_image_list.txt --targets=topaztrain/topaz_scaled_coordinates1.txt --number=650

Modelc as an example of a model command (all other models were the same, except for the n number and device number):
topaz train --train-images topaztrain/full_image_list_train.txt --train-targets topaztrain/topaz_scaled_coordinates1_train.txt --test-images topaztrain/full_image_list_test.txt --test-targets topaztrain/topaz_scaled_coordinates1_test.txt --radius 4 -n 200 --output topaztrain/modelc --save-prefix topaztrain/training_models/modelc --device 3 > topaztrain/modelcout.txt 2> topaztrain/modelcerror.txt &

Thank you again - please let me know what other information you need from me!

[tbepler]

Do you have another dataset you can test this on to see if it's something specific to your dataset vs. your topaz install? Can you run your topaz install on the tutorial dataset (e.g., see https://github.com/tbepler/topaz/blob/master/tutorial/01_quick_start_guide.ipynb)?

I don't think this is the source of the problem, but I also recommend using topaz convert instead of "boxes_to_coordinates" and "scale_coordinates". Topaz convert is an all-in-one particle coordinate processing command.

[gaandree]

Hello! I just tried using the same commands on another dataset and it worked fine, so I think the problem lies within my data. I checked the inputs again and they seemed fine so I think something might have gone wrong with the initial steps of processing my data. I'm also trying to use topaz within relion 4 to see if that works better. Thanks again for your help!

[tbepler]

One other thing that pops out to me reading your commands is the .../movies/* part of your command. Are you putting movie frames into Topaz? It needs your micrographs to have already been aligned and summed.

[gaandree]

Nope, the /movies directory is in the output MotionCorr directory from Relion and contains the corrected .mrcs from MotionCorr, so they have been aligned and summed. My original directory was named "movies" and so I've just kept that naming throughout the steps, but it is a little misleading - my apologies!

[jasonkey]

Hi all,

Jason from SBGrid here -

This topaz version was installed with conda via

conda install topaz cudatoolkit=9.0 -c tbepler -c pytorch

Here's the version :

CapsuleShell-topaz $ topaz --version 
TOPAZ 0.2.5a

the environment is isolated, similar to a virtual environment. Let me know if I can provide more specifics.

We also have a newer version available installed from a more recent commit - we can try that if it would be helpful.

[jasonkey]

I did a fresh install of topaz from the latest commit - conda installed the dependencies, pip installed topaz from repo.
That process is shown below. @gaandree can comment more but has reported the same results with this new installation as reported above.

$ pip install .                                                                                                                                               

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com                                                                                       

Processing /programs/x86_64-linux/topaz/0.2.5-2/topaz                                                                                                          

  Preparing metadata (setup.py) ... done                                                                                                                       

Requirement already satisfied: torch>=1.0.0 in /programs/x86_64-linux/topaz/0.2.5-2/topaz_extlib/miniconda3-4.8.2-b5qb/envs/topaz/lib/python3.6/site-packages 

(from topaz-em==0.2.5a0) (1.4.0)                                                                                                                               

Requirement already satisfied: torchvision in /programs/x86_64-linux/topaz/0.2.5-2/topaz_extlib/miniconda3-4.8.2-b5qb/envs/topaz/lib/python3.6/site-packages (

from topaz-em==0.2.5a0) (0.5.0)                                                                                                                                

Requirement already satisfied: numpy>=1.11 in /programs/x86_64-linux/topaz/0.2.5-2/topaz_extlib/miniconda3-4.8.2-b5qb/envs/topaz/lib/python3.6/site-packages (

from topaz-em==0.2.5a0) (1.19.5)                                                                                                                               

Requirement already satisfied: pandas in /programs/x86_64-linux/topaz/0.2.5-2/topaz_extlib/miniconda3-4.8.2-b5qb/envs/topaz/lib/python3.6/site-packages (from 

topaz-em==0.2.5a0) (1.1.5)                                                                                                                                     

Requirement already satisfied: scikit-learn>=0.19.0 in /programs/x86_64-linux/topaz/0.2.5-2/topaz_extlib/miniconda3-4.8.2-b5qb/envs/topaz/lib/python3.6/site-p

ackages (from topaz-em==0.2.5a0) (0.24.2)                                                                                                                      

Requirement already satisfied: scipy>=0.17.0 in /programs/x86_64-linux/topaz/0.2.5-2/topaz_extlib/miniconda3-4.8.2-b5qb/envs/topaz/lib/python3.6/site-packages

 (from topaz-em==0.2.5a0) (1.5.3)                                                                                                                              

Requirement already satisfied: pillow>=6.2.0 in /programs/x86_64-linux/topaz/0.2.5-2/topaz_extlib/miniconda3-4.8.2-b5qb/envs/topaz/lib/python3.6/site-packages

 (from topaz-em==0.2.5a0) (8.3.2)                                                                                                                              

Collecting future                                                                                                                                              

  Downloading future-0.18.2.tar.gz (829 kB)                                                                                                                    

     |████████████████████████████████| 829 kB 10.3 MB/s                                                                                                       

  Preparing metadata (setup.py) ... done                                                                                                                       

Requirement already satisfied: joblib>=0.11 in /programs/x86_64-linux/topaz/0.2.5-2/topaz_extlib/miniconda3-4.8.2-b5qb/envs/topaz/lib/python3.6/site-packages 

(from scikit-learn>=0.19.0->topaz-em==0.2.5a0) (1.1.0)                                                                                                         

Requirement already satisfied: threadpoolctl>=2.0.0 in /programs/x86_64-linux/topaz/0.2.5-2/topaz_extlib/miniconda3-4.8.2-b5qb/envs/topaz/lib/python3.6/site-p

ackages (from scikit-learn>=0.19.0->topaz-em==0.2.5a0) (3.0.0)                                                                                                 

Requirement already satisfied: python-dateutil>=2.7.3 in /programs/x86_64-linux/topaz/0.2.5-2/topaz_extlib/miniconda3-4.8.2-b5qb/envs/topaz/lib/python3.6/site

-packages (from pandas->topaz-em==0.2.5a0) (2.8.2)                                                                                                             

Requirement already satisfied: pytz>=2017.2 in /programs/x86_64-linux/topaz/0.2.5-2/topaz_extlib/miniconda3-4.8.2-b5qb/envs/topaz/lib/python3.6/site-packages 

(from pandas->topaz-em==0.2.5a0) (2021.3)                                                                                                                      

Requirement already satisfied: six in /programs/x86_64-linux/topaz/0.2.5-2/topaz_extlib/miniconda3-4.8.2-b5qb/envs/topaz/lib/python3.6/site-packages (from tor

chvision->topaz-em==0.2.5a0) (1.16.0)                                                                                                                          

Building wheels for collected packages: topaz-em, future                                                                                                       

  Building wheel for topaz-em (setup.py) ... done                                                                                                              

  Created wheel for topaz-em: filename=topaz_em-0.2.5a0-py3-none-any.whl size=48185457 sha256=d01c6db6e3e3ca1f879afdc38c5e2f248114895354174a082cb6d8e7b118a5a9 

  Stored in directory: /tmp/pip-ephem-wheel-cache-jph7sv7z/wheels/a3/17/eb/bb0cd5d6c3be899fcdf44439345e09baf2b289a457645d8691                                  

  Building wheel for future (setup.py) ... done                                                                                                                

  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491058 sha256=24de6f740f18c0052d25b5f87d65fe1dbf1efac39200a341a6c68797bd39cb32        

  Stored in directory: /tmp/pip-ephem-wheel-cache-jph7sv7z/wheels/6e/9c/ed/4499c9865ac1002697793e0ae05ba6be33553d098f3347fb94                                  

Successfully built topaz-em future                                                                                                                             

Installing collected packages: future, topaz-em                                                                                                                

Successfully installed future-0.18.2 topaz-em-0.2.5a0          

happy to provide more info -
-Jason

[tbepler]

Interesting, nothing pops out here as unusual.

[gaandree]

Hi everyone - I think I found the problem! We had a computer go down so I had to copy my data across a few different places and something must have gone wrong with the transferring or when the computer went down. In my directories, all of the file names were copied but a few files had smaller size than before. This didn't happen to every file, so I am not sure which ones were affected but when I started everything from the beginning on one computer, with no transferring, the topaz models worked as expected. Thank you for all your help with this!

[tbepler]

Great! Glad to hear everything is working now!