Question about estimate_roc

[frozenfas]

Hello, I am having a problem running pytom_estimate_roc.py, and I would like to know if you have tips on how I can solve the problem.

I run the template match like this:
/opt/miniconda3/envs/pytom_tm/bin/pytom_match_template.py --template External/job702/templates/emd_16139_resize.mrc --mask External/job702/templates/emd_16139_resize_mask.mrc --tomogram Tomograms/job014/tomograms/rec_ts-L1T3.mrc --destination External/job702/results --particle-diameter 280.0 --tilt-angles External/job702/datasets/ts-L1T3.rawtlt --low-pass 30.0 --per-tilt-weighting --defocus External/job702/datasets/ts-L1T3.defocus --amplitude-contrast 0.1 --spherical-aberration 2.7 --voltage 300.0 --tomogram-mask External/job018/predictions/ts-L1T3/rec_ts-L1T3_MemBrain_seg_v10_alpha.ckpt_segmented.mrc --gpu-ids 0 1 2 3 --tomogram-ctf-model phase-flip --volume-split 2 2 2 --random-phase-correction

and when viewing the results in napri, I think it is working well:

I then try to run pytom_estimate_roc and get the following error:
/opt/miniconda3/envs/pytom_tm/bin/pytom_estimate_roc.py --job-file External/job702/results/rec_ts-L1T3_job.json -n 10000 -r 12 --bins 16 --show-plot --crop-plot 100%|███████████████████████████████████████████████████████████████████████████████████████| 10000/10000 [26:16<00:00, 6.34it/s] Traceback (most recent call last): File "/opt/miniconda3/envs/pytom_tm/lib/python3.12/site-packages/pytom_tm/plotting.py", line 294, in plist_quality_gaussian_fit params, cov = curve_fit( ^^^^^^^^^^ File "/opt/miniconda3/envs/pytom_tm/lib/python3.12/site-packages/scipy/optimize/_minpack_py.py", line 1018, in curve_fit res = least_squares(func, p0, jac=jac, bounds=bounds, method=method, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/pytom_tm/lib/python3.12/site-packages/scipy/optimize/_lsq/least_squares.py", line 820, in least_squar es raise ValueError("x0is infeasible.") ValueError:x0 is infeasible.

But pops up the following plot, which is incomplete:

If I try to extract candidates and specify a cut of 0.2, it runs, but if I omit the cutoff (i.e. to set automatically), it fails like this:
/opt/miniconda3/envs/pytom_tm/bin/pytom_ extract_candidates.py --job-file External/job702/results/rec_ts-L1T3_job.json -n 9000 -r 12 --log info --relion5-compat INFO:root:cut off for particle extraction: 0.518585467789753 0%| | 0/9000 [00:00<?, ?it/s] /opt/miniconda3/envs/pytom_tm/lib/python3.12/site-packages/pytom_tm/extract.py:345: UserWarning: Attempting to set identical low a nd high ylims makes transformation singular; automatically expanding. ax.set_ylim(bottom=0, top=2 * max(y))

It also fails if I try to use the tophat filters:
/opt/miniconda3/envs/pytom_tm/bin/pytom_ extract_candidates.py --job-file External/job702/results/rec_ts-L1T3_job.json -n 9000 -r 12 --log info --relion5-compat --tophat -filter -c 0.2 0%| | 0/9000 [00:00<?, ?it/s] /opt/miniconda3/envs/pytom_tm/lib/python3.12/site-packages/pytom_tm/extract.py:345: UserWarning: Attempting to set identical low a nd high ylims makes transformation singular; automatically expanding. ax.set_ylim(bottom=0, top=2 * max(y))

I guess I want to ask quickly if you understand why it is failing and if there are any settings I should tweak to get it running. Is it, for example, a problem that I have so many particles? In particular, I am interested in the tophat filter. As you can see in the naperi screenshots, I have long-connected contamination that is being picked, and I would like to eliminate that.

Thanks in advance for your help.

sean

[sroet]

I don't quickly see why it's failing, but could you try running it without the --tomogram-mask?
It probably does nothing to speed up the TM, as it only skips a complete subvolume at a time, to make sure adding a mask doesn't alter the result, and we don't apply the mask in the last step as it felt wasteful to remove data we computed.

However, it might prevent extraction (as it will automatically not extract particles wherever the mask given to the TM-job <0)

(both estimate_roc and extract_candidates use the internal extract_candidates function, which automatically grabs the TMjob tomogram-mask)

If this was the issue, I will probably implement an --ignore-tomogram-mask option for those functions to help testing this without rerunning the template matching in the future

[frozenfas]

Thanks for the help. I just started a new TM run without the mask. I will let you know tomorrow how it goes :)

[sroet]

@frozenfas Just checking in, did rerunning without a mask solve your issue?

[frozenfas]

yes, if I left out the mask pytom_estimate_roc.py finished correctly and yielded plots. The suggested cutoff was to high and I used my own cutoff to extract particles.... but it worked well. Thanks

[sroet]

Thanks for reporting back. happy to hear it worked in the end.
I will make an issue to include a way to run these scripts without a mask without rerunning the complete template matching again