error is in rule Checkm2ContigAnalysis

[bak1121]

Hello @dportik ,
The HiFi-MAG pipeline was running but shut down with the following error:
And the error is in rule Checkm2ContigAnalysis:

Config file config.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
Provided cores: 8
Rules claiming more threads will be scaled down.
Job stats:
job                   count    min threads    max threads
------------------  -------  -------------  -------------
AssessCheckm2Bins         1              1              1
Checkm2BinAnalysis        1              8              8
CloseCheckm2Fork          1              1              1
CloseLongbinFork          1              1              1
ConvertJGIBamDepth        1              1              1
CopyDAStoolBins           1              1              1
DASinputMetabat2          1              1              1
DASinputSemiBin2          1              1              1
DAStoolAnalysis           1              8              8
FilterSuperBins           1              1              1
IndexBam                  1              8              8
JGIBamDepth               1              1              1
LongContigsToBins         1              1              1
Metabat2Analysis          1              8              8
MinimapIndex              1              8              8
MinimapToBam              1              8              8
SemiBin2Analysis          1              8              8
all                       1              1              1
total                    18              1              8

Select jobs to execute...

[Thu Aug 29 18:38:55 2024]
rule MinimapIndex:
    input: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/inputs/MDBG661.fasta, /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/inputs/MDBG661.contigs.fasta
    output: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.contigs.mmi
    log: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/logs/MDBG661.MinimapIndex.log
    jobid: 7
    benchmark: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/benchmarks/MDBG661.MinimapIndex.tsv
    reason: Missing output files: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.contigs.mmi
    wildcards: sample=MDBG661
    threads: 8
    resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/e82d62586d95e68edaa56983fc33f32e_
[Thu Aug 29 18:39:09 2024]
Finished job 7.
1 of 18 steps (6%) done
Select jobs to execute...

[Thu Aug 29 18:39:09 2024]
rule MinimapToBam:
    input: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/inputs/MDBG661.fasta, /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.contigs.mmi
    output: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.bam
    log: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/logs/MDBG661.MinimapToBam.log
    jobid: 6
    benchmark: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/benchmarks/MDBG661.MinimapToBam.tsv
    reason: Missing output files: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.bam; Input files updated by another job: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.contigs.mmi
    wildcards: sample=MDBG661
    threads: 8
    resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/e82d62586d95e68edaa56983fc33f32e_
[bam_sort_core] merging from 0 files and 8 in-memory blocks...
[Thu Aug 29 18:42:29 2024]
Finished job 6.
2 of 18 steps (11%) done
Select jobs to execute...

[Thu Aug 29 18:42:29 2024]
rule IndexBam:
    input: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.bam
    output: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.bam.bai, /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.index.completed.txt
    log: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/logs/MDBG661.IndexBam.log
    jobid: 5
    benchmark: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/benchmarks/MDBG661.IndexBam.tsv
    reason: Missing output files: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.bam.bai, /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.index.completed.txt; Input files updated by another job: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.bam
    wildcards: sample=MDBG661
    threads: 8
    resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/e82d62586d95e68edaa56983fc33f32e_
[Thu Aug 29 18:42:34 2024]
Finished job 5.
3 of 18 steps (17%) done
Select jobs to execute...

[Thu Aug 29 18:42:34 2024]
localcheckpoint LongContigsToBins:
    input: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/inputs/MDBG661.contigs.fasta
    output: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/1-long-contigs/MDBG661/MDBG661.bin_key.txt
    log: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/logs/MDBG661.LongContigsToBins.log
    jobid: 2
    reason: Missing output files: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/1-long-contigs/MDBG661/MDBG661.bin_key.txt
    wildcards: sample=MDBG661
    resources: tmpdir=/tmp
Downstream jobs will be updated after completion.


Activating conda environment: .snakemake/conda/203e2287027cc01bf9a99303f3de6fdd_
[Thu Aug 29 18:42:34 2024]
rule JGIBamDepth:
    input: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.index.completed.txt, /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.bam, /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.bam.bai
    output: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.JGI.depth.txt
    log: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/logs/MDBG661.JGIBamDepth.log
    jobid: 4
    benchmark: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/benchmarks/MDBG661.JGIBamDepth.tsv
    reason: Missing output files: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.JGI.depth.txt; Input files updated by another job: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.bam, /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.bam.bai, /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/2-bam/MDBG661.index.completed.txt
    wildcards: sample=MDBG661
    resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/12a185818392f06cadd3fb18aed00ff7_
[Thu Aug 29 18:42:53 2024]
Finished job 4.
4 of 18 steps (22%) done
[Thu Aug 29 18:42:54 2024]
Finished job 2.
5 of 18 steps (28%) done
Select jobs to execute...

[Thu Aug 29 18:42:54 2024]
rule Checkm2ContigAnalysis:
    input: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/CheckM2_database/uniref100.KO.1.dmnd, /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/1-long-contigs/MDBG661/MDBG661.bin_key.txt
    output: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/1-long-contigs/MDBG661/checkm2/quality_report.tsv
    log: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/logs/MDBG661.Checkm2ContigAnalysis.log
    jobid: 22
    benchmark: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/benchmarks/MDBG661.Checkm2ContigAnalysis.tsv
    reason: Missing output files: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/1-long-contigs/MDBG661/checkm2/quality_report.tsv
    wildcards: sample=MDBG661
    threads: 8
    resources: tmpdir=/tmp

Activating conda environment: .snakemake/conda/58ac322be76242f60095b5eb8b462df2_
[Thu Aug 29 18:43:41 2024]
Error in rule Checkm2ContigAnalysis:
    jobid: 22
    input: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/CheckM2_database/uniref100.KO.1.dmnd, /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/1-long-contigs/MDBG661/MDBG661.bin_key.txt
    output: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/1-long-contigs/MDBG661/checkm2/quality_report.tsv
    log: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/logs/MDBG661.Checkm2ContigAnalysis.log (check log file(s) for error message)
    conda-env: /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/.snakemake/conda/58ac322be76242f60095b5eb8b462df2_
    shell:
        checkm2 predict -i /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/1-long-contigs/MDBG661/bins/ -o /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/1-long-contigs/MDBG661/checkm2/ -x fa -t 8 --force --database_path /mnt/mountdoom/mnt/work/bioinfo/home/bak21/metagenomics/start_august2024/HiFi-MAG/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/CheckM2_database/uniref100.KO.1.dmnd --remove_intermediates --tmpdir /mnt/mountdoom/mnt/work/bioinfo/home/scratch/hifi-mag_baban &> /home/sak/pb-metagenomics-tools-3.2.0/HiFi-MAG-Pipeline/logs/MDBG661.Checkm2ContigAnalysis.log
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-08-29T183751.598938.snakemake.log

The config file and Checkm2ContigAnalysis.log.txt are attached here:
MDBG661.Checkm2ContigAnalysis.log.txt
config.yaml.txt

Could you please help with this?
Kind regards,

[dportik]

Hi @bak1121 ,
This appears to be a known issue with CheckM2. Please see the threads here for possible solutions:
chklovski/CheckM2#82
chklovski/CheckM2#100

I noticed from the log file that it appears you only have 8 threads available for analyses. This may not be enough resources to run GTDB-Tk, particularly the memory requirements.

[bak1121]

Hi @dportik ,
Thank you for pointing out similar issues. An error was solved when I changed a Python 3.6 to 3.8.