Training process ends with message "process 1 terminated with signal SIGKILL"

[lukasvanderstricht]

Hello everyone!

For the past few weeks I have been training a model using MONAILabel and 3D Slicer. I have been doing this on a Google Cloud Virtual Machine. For several reasons, I now want to migrate my work to a different VM. I managed to transfer all my data and my radiology app that contains all the models I have trained.

Now I would want to continue training in this new location. When I start the server like I did before, everything works fine. But then, once I press the Train button, a problem occurs. The logs of the training process come to a halt before the first epoch even begins. After quite a long period of time, I get an error message stating process 1 terminated with signal SIGKILL.
I think there must be something wrong with my installation or configuration as everything works fine on the original VM and I'm using the same monailabel version with which the model was trained (0.4.1). The full logs can be found below.

monailabel start_server --app radiology_psoas_azd --studies psoas-azd-images/train-images/ --conf models deepedit
Using PYTHONPATH=/home/dellxpsazdelta:

[2023-08-23 08:08:57,589] [34938] [MainThread] [INFO] (main:269) - USING:: app = /home/dellxpsazdelta/radiology_psoas_azd
[2023-08-23 08:08:57,589] [34938] [MainThread] [INFO] (main:269) - USING:: studies = /home/dellxpsazdelta/psoas-azd-images/train-images
[2023-08-23 08:08:57,589] [34938] [MainThread] [INFO] (main:269) - USING:: verbose = INFO
[2023-08-23 08:08:57,590] [34938] [MainThread] [INFO] (main:269) - USING:: conf = [['models', 'deepedit']]
[2023-08-23 08:08:57,590] [34938] [MainThread] [INFO] (main:269) - USING:: host = 0.0.0.0
[2023-08-23 08:08:57,590] [34938] [MainThread] [INFO] (main:269) - USING:: port = 8000
[2023-08-23 08:08:57,590] [34938] [MainThread] [INFO] (main:269) - USING:: uvicorn_app = monailabel.app:app
[2023-08-23 08:08:57,590] [34938] [MainThread] [INFO] (main:269) - USING:: ssl_keyfile = None
[2023-08-23 08:08:57,590] [34938] [MainThread] [INFO] (main:269) - USING:: ssl_certfile = None
[2023-08-23 08:08:57,590] [34938] [MainThread] [INFO] (main:269) - USING:: ssl_keyfile_password = None
[2023-08-23 08:08:57,590] [34938] [MainThread] [INFO] (main:269) - USING:: ssl_ca_certs = None
[2023-08-23 08:08:57,590] [34938] [MainThread] [INFO] (main:269) - USING:: workers = None
[2023-08-23 08:08:57,590] [34938] [MainThread] [INFO] (main:269) - USING:: limit_concurrency = None
[2023-08-23 08:08:57,590] [34938] [MainThread] [INFO] (main:269) - USING:: access_log = False
[2023-08-23 08:08:57,590] [34938] [MainThread] [INFO] (main:269) - USING:: log_config = None
[2023-08-23 08:08:57,590] [34938] [MainThread] [INFO] (main:269) - USING:: dryrun = False
[2023-08-23 08:08:57,590] [34938] [MainThread] [INFO] (main:269) - USING:: action = start_server
[2023-08-23 08:08:57,591] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_API_STR =
[2023-08-23 08:08:57,591] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_PROJECT_NAME = MONAILabel
[2023-08-23 08:08:57,591] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_APP_DIR =
[2023-08-23 08:08:57,591] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_STUDIES =
[2023-08-23 08:08:57,591] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_APP_CONF = '{}'
[2023-08-23 08:08:57,591] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_TASKS_TRAIN = True
[2023-08-23 08:08:57,591] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_TASKS_STRATEGY = True
[2023-08-23 08:08:57,591] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_TASKS_SCORING = True
[2023-08-23 08:08:57,591] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_TASKS_BATCH_INFER = True
[2023-08-23 08:08:57,591] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_USERNAME =
[2023-08-23 08:08:57,591] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PASSWORD =
[2023-08-23 08:08:57,591] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_PATH =
[2023-08-23 08:08:57,592] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_QIDO_PREFIX =
[2023-08-23 08:08:57,592] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_WADO_PREFIX =
[2023-08-23 08:08:57,592] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_STOW_PREFIX =
[2023-08-23 08:08:57,592] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_FETCH_BY_FRAME = False
[2023-08-23 08:08:57,592] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DATASTORE_AUTO_RELOAD = True
[2023-08-23 08:08:57,592] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_DATASTORE_FILE_EXT = '[".nii.gz", ".nii", ".nrrd", ".jpg", ".png", ".tif", ".svs", ".xml"]'
[2023-08-23 08:08:57,592] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_SERVER_PORT = 8000
[2023-08-23 08:08:57,592] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_CORS_ORIGINS = '[]'
[2023-08-23 08:08:57,592] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_SESSIONS = True
[2023-08-23 08:08:57,592] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_SESSION_PATH =
[2023-08-23 08:08:57,592] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_SESSION_EXPIRY = 3600
[2023-08-23 08:08:57,592] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_INFER_CONCURRENCY = -1
[2023-08-23 08:08:57,592] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_INFER_TIMEOUT = 600
[2023-08-23 08:08:57,592] [34938] [MainThread] [INFO] (main:273) - ENV SETTINGS:: MONAI_LABEL_AUTO_UPDATE_SCORING = True
[2023-08-23 08:08:57,593] [34938] [MainThread] [INFO] (main:274) -
[2023-08-23 08:08:58,439] [34938] [MainThread] [INFO] (uvicorn.error:75) - Started server process [34938]
[2023-08-23 08:08:58,439] [34938] [MainThread] [INFO] (uvicorn.error:45) - Waiting for application startup.
[2023-08-23 08:08:58,439] [34938] [MainThread] [INFO] (monailabel.interfaces.utils.app:38) - Initializing App from: /home/dellxpsazdelta/radiology_psoas_azd; studies: /home/dellxpsazdelta/psoas-azd-images/train-images; conf: {'models': 'deepedit'}
[2023-08-23 08:09:02,420] [34938] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2023-08-23 08:09:02,439] [34938] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2023-08-23 08:09:02,445] [34938] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2023-08-23 08:09:02,446] [34938] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.deepeditL3.DeepEdit'>
[2023-08-23 08:09:02,447] [34938] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.model.Segmentation'>
[2023-08-23 08:09:02,448] [34938] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2023-08-23 08:09:02,449] [34938] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2023-08-23 08:09:02,449] [34938] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2023-08-23 08:09:02,449] [34938] [MainThread] [INFO] (main:83) - +++ Adding Model: deepedit => lib.configs.deepedit.DeepEdit
[2023-08-23 08:09:03,374] [34938] [MainThread] [INFO] (lib.configs.deepedit:145) - EPISTEMIC Enabled: 0; Samples: 5
[2023-08-23 08:09:03,374] [34938] [MainThread] [INFO] (lib.configs.deepedit:149) - TTA Enabled: 0; Samples: 5
[2023-08-23 08:09:03,374] [34938] [MainThread] [INFO] (main:87) - +++ Using Models: ['deepedit']
[2023-08-23 08:09:03,375] [34938] [MainThread] [INFO] (monailabel.interfaces.app:126) - Init Datastore for: /home/dellxpsazdelta/psoas-azd-images/train-images
[2023-08-23 08:09:03,375] [34938] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: True; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '.xml']
[2023-08-23 08:09:03,387] [34938] [MainThread] [INFO] (monailabel.datastore.local:540) - Invalidate count: 0
[2023-08-23 08:09:03,387] [34938] [MainThread] [INFO] (monailabel.datastore.local:145) - Start observing external modifications on datastore (AUTO RELOAD)
[2023-08-23 08:09:03,389] [34938] [MainThread] [INFO] (main:112) - +++ Adding Inferer:: deepedit => <lib.infers.deepedit.DeepEdit object at 0x7fb775157460>
[2023-08-23 08:09:03,389] [34938] [MainThread] [INFO] (main:112) - +++ Adding Inferer:: deepedit_seg => <lib.infers.deepedit.DeepEdit object at 0x7fb76c25fb50>
[2023-08-23 08:09:03,389] [34938] [MainThread] [INFO] (main:161) - +++ Adding Trainer:: deepedit => <lib.trainers.deepedit.DeepEdit object at 0x7fb76c1efca0>
[2023-08-23 08:09:03,390] [34938] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/dellxpsazdelta/.cache/monailabel/sessions
[2023-08-23 08:09:03,390] [34938] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2023-08-23 08:09:03,390] [34938] [MainThread] [INFO] (monailabel.interfaces.app:429) - App Init - completed
[2023-08-23 08:09:03,390] [timeloop] [INFO] Starting Timeloop..
[2023-08-23 08:09:03,390] [34938] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2023-08-23 08:09:03,390] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x7fb76e3c68b0>
[2023-08-23 08:09:03,390] [34938] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x7fb76e3c68b0>
[2023-08-23 08:09:03,390] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2023-08-23 08:09:03,390] [34938] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2023-08-23 08:09:03,391] [34938] [MainThread] [INFO] (uvicorn.error:59) - Application startup complete.
[2023-08-23 08:09:03,391] [34938] [MainThread] [INFO] (uvicorn.error:206) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[2023-08-23 08:11:41,162] [34938] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Train request: {'model': 'deepedit', 'name': 'train_01', 'pretrained': 1, 'device': 'cuda', 'max_epochs': 5, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'Dataset', 'dataloader': 'ThreadDataLoader', 'client_id': 'user-xyz'}
[2023-08-23 08:11:41,163] [34938] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:58) - COMMAND:: /home/dellxpsazdelta/anaconda3/envs/monailabel-env/bin/python -m monailabel.interfaces.utils.app -m train -r {"model":"deepedit","name":"train_01","pretrained":1,"device":"cuda","max_epochs":5,"early_stop_patience":-1,"val_split":0.2,"train_batch_size":1,"val_batch_size":1,"multi_gpu":true,"gpus":"all","dataset":"Dataset","dataloader":"ThreadDataLoader","client_id":"user-xyz"}
[2023-08-23 08:11:41,259] [34969] [MainThread] [INFO] (main:38) - Initializing App from: /home/dellxpsazdelta/radiology_psoas_azd; studies: /home/dellxpsazdelta/psoas-azd-images/train-images; conf: {'models': 'deepedit'}
[2023-08-23 08:11:44,270] [34969] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2023-08-23 08:11:44,277] [34969] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2023-08-23 08:11:44,280] [34969] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2023-08-23 08:11:44,281] [34969] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.deepeditL3.DeepEdit'>
[2023-08-23 08:11:44,281] [34969] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.model.Segmentation'>
[2023-08-23 08:11:44,282] [34969] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2023-08-23 08:11:44,282] [34969] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2023-08-23 08:11:44,283] [34969] [MainThread] [INFO] (monailabel.utils.others.class_utils:36) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2023-08-23 08:11:44,283] [34969] [MainThread] [INFO] (main:83) - +++ Adding Model: deepedit => lib.configs.deepedit.DeepEdit
[2023-08-23 08:11:45,150] [34969] [MainThread] [INFO] (lib.configs.deepedit:145) - EPISTEMIC Enabled: 0; Samples: 5
[2023-08-23 08:11:45,150] [34969] [MainThread] [INFO] (lib.configs.deepedit:149) - TTA Enabled: 0; Samples: 5
[2023-08-23 08:11:45,151] [34969] [MainThread] [INFO] (main:87) - +++ Using Models: ['deepedit']
[2023-08-23 08:11:45,151] [34969] [MainThread] [INFO] (monailabel.interfaces.app:126) - Init Datastore for: /home/dellxpsazdelta/psoas-azd-images/train-images
[2023-08-23 08:11:45,151] [34969] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: False; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '.xml']
[2023-08-23 08:11:45,164] [34969] [MainThread] [INFO] (monailabel.datastore.local:540) - Invalidate count: 0
[2023-08-23 08:11:45,164] [34969] [MainThread] [INFO] (main:112) - +++ Adding Inferer:: deepedit => <lib.infers.deepedit.DeepEdit object at 0x7fc2676f91f0>
[2023-08-23 08:11:45,164] [34969] [MainThread] [INFO] (main:112) - +++ Adding Inferer:: deepedit_seg => <lib.infers.deepedit.DeepEdit object at 0x7fc26055d8b0>
[2023-08-23 08:11:45,165] [34969] [MainThread] [INFO] (main:161) - +++ Adding Trainer:: deepedit => <lib.trainers.deepedit.DeepEdit object at 0x7fc260174940>
[2023-08-23 08:11:45,165] [34969] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/dellxpsazdelta/.cache/monailabel/sessions
[2023-08-23 08:11:45,165] [34969] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2023-08-23 08:11:45,165] [34969] [MainThread] [INFO] (monailabel.tasks.train.basic_train:353) - Train Request (input): {'model': 'deepedit', 'name': 'train_01', 'pretrained': 1, 'device': 'cuda', 'max_epochs': 5, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'Dataset', 'dataloader': 'ThreadDataLoader', 'client_id': 'user-xyz', 'local_rank': 0}
[2023-08-23 08:11:45,202] [34969] [MainThread] [INFO] (monailabel.tasks.train.basic_train:363) - CUDA_VISIBLE_DEVICES: None
[2023-08-23 08:11:45,204] [34969] [MainThread] [INFO] (monailabel.tasks.train.basic_train:373) - Distributed/Multi GPU Training = TRUE
[2023-08-23 08:11:51.444] INFO - Main Worker: 0
[2023-08-23 08:11:51.445] INFO - Main Worker: 1
[2023-08-23 08:11:51.446] INFO - 1 - Train Request (final): {'name': 'train_01', 'pretrained': 1, 'device': 'cuda', 'max_epochs': 5, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'Dataset', 'dataloader': 'ThreadDataLoader', 'model': 'deepedit', 'client_id': 'user-xyz', 'local_rank': 0, 'run_id': '20230823_0811'}
[2023-08-23 08:11:51.446] INFO - 0 - Train Request (final): {'name': 'train_01', 'pretrained': 1, 'device': 'cuda', 'max_epochs': 5, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'Dataset', 'dataloader': 'ThreadDataLoader', 'model': 'deepedit', 'client_id': 'user-xyz', 'local_rank': 0, 'run_id': '20230823_0811'}
[2023-08-23 08:11:52.562] INFO - Added key: store_based_barrier_key:1 to store for rank: 1
[2023-08-23 08:11:52.565] INFO - Added key: store_based_barrier_key:1 to store for rank: 0
[2023-08-23 08:11:52.566] INFO - Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
[2023-08-23 08:11:52.566] INFO - ++++ Rank:0 => Using GPU-0
[2023-08-23 08:11:52.573] INFO - Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
[2023-08-23 08:11:52.573] INFO - ++++ Rank:1 => Using GPU-1
[2023-08-23 08:11:52.628] INFO - 0 - Using Device: cuda:0; IDX: 0
[2023-08-23 08:11:52.630] INFO - 1 - Using Device: cuda:1; IDX: 1
[2023-08-23 08:11:52.631] INFO - Total Records for Training: 16
[2023-08-23 08:11:52.631] INFO - Total Records for Validation: 4
Traceback (most recent call last):
File "/home/dellxpsazdelta/anaconda3/envs/monailabel-env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/dellxpsazdelta/anaconda3/envs/monailabel-env/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/dellxpsazdelta/.local/lib/python3.9/site-packages/monailabel/interfaces/utils/app.py", line 132, in
run_main()
File "/home/dellxpsazdelta/.local/lib/python3.9/site-packages/monailabel/interfaces/utils/app.py", line 117, in run_main
result = a.train(request)
File "/home/dellxpsazdelta/.local/lib/python3.9/site-packages/monailabel/interfaces/app.py", line 380, in train
result = task(request, self.datastore())
File "/home/dellxpsazdelta/.local/lib/python3.9/site-packages/monailabel/tasks/train/basic_train.py", line 378, in call
torch.multiprocessing.spawn(main_worker, nprocs=world_size, args=(world_size, req, datalist, self))
File "/home/dellxpsazdelta/.local/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/dellxpsazdelta/.local/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
while not context.join():
File "/home/dellxpsazdelta/.local/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 140, in join
raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 1 terminated with signal SIGKILL
[2023-08-23 08:25:45,107] [34938] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:76) - Return code: 1

Does anyone have an idea where it went wrong?

Kind regards

Lukas

[SachidanandAlle]

When u run train.. try not use multi GPU.. and see if you can train on single gpu

[lukasvanderstricht]

Hi @SachidanandAlle

Thanks for your reply.
The problem is that my GPU consistently ran out of memory during the training process. Therefore, I added a GPU to the virtual machine. So disabling the multi-GPU functionality kind of isn't an option for me. Is there any way to solve this error without losing the use of both my GPUs?

Kind regards

Lukas

[SachidanandAlle]

I am asking you to isolate the problem... For debugging only

[lukasvanderstricht]

Thank you @SachidanandAlle

I indeed get another error when I uncheck the box "multi-gpu" in the Developer mode of the MONAILabel Slicer module before I start the training. The output is the following:

[2023-08-25 07:39:11,734] [1071] [MainThread] [INFO] (monailabel.utils.async_tasks.task:41) - Train request: {'model': 'deepedit', 'name': 'train_01', 'pretrained': False, 'device': 'Tesla T4:0', 'max_epochs': 5, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': False, 'gpus': 'all', 'dataset': 'Dataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'client_id': 'user-xyz'}
[2023-08-25 07:39:11,735] [1071] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:49) - Before:: /home/lukas/miniconda3/envs::/home/lukas/radiology
[2023-08-25 07:39:11,735] [1071] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:53) - After:: /home/lukas/miniconda3/envs::/home/lukas/radiology
[2023-08-25 07:39:11,735] [1071] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:65) - COMMAND:: /home/lukas/miniconda3/envs/monailabel-env/bin/python -m monailabel.interfaces.utils.app -m train -r {"model":"deepedit","name":"train_01","pretrained":false,"device":"Tesla T4:0","max_epochs":5,"early_stop_patience":-1,"val_split":0.2,"train_batch_size":1,"val_batch_size":1,"multi_gpu":false,"gpus":"all","dataset":"Dataset","dataloader":"ThreadDataLoader","tracking":"mlflow","tracking_uri":"","tracking_experiment_name":"","client_id":"user-xyz"}
[2023-08-25 07:39:12,111] [1183] [MainThread] [INFO] (__main__:37) - Initializing App from: /home/lukas/radiology; studies: /home/lukas/psoas-azd-images/train-images; conf: {'models': 'deepedit'}
[2023-08-25 07:39:15,489] [1183] [MainThread] [INFO] (torch.distributed.nn.jit.instantiator:21) - Created a temporary directory at /tmp/tmp4v5h0h4n
[2023-08-25 07:39:15,490] [1183] [MainThread] [INFO] (torch.distributed.nn.jit.instantiator:76) - Writing /tmp/tmp4v5h0h4n/_remote_module_non_scriptable.py
[2023-08-25 07:39:20,515] [1183] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2023-08-25 07:39:20,530] [1183] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2023-08-25 07:39:20,534] [1183] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2023-08-25 07:39:20,535] [1183] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'>
[2023-08-25 07:39:20,536] [1183] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'>
[2023-08-25 07:39:20,537] [1183] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'>
[2023-08-25 07:39:20,538] [1183] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2023-08-25 07:39:20,539] [1183] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2023-08-25 07:39:20,540] [1183] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2023-08-25 07:39:20,540] [1183] [MainThread] [INFO] (main:93) - +++ Adding Model: deepedit => lib.configs.deepedit.DeepEdit
[2023-08-25 07:39:21,537] [1183] [MainThread] [INFO] (lib.configs.deepedit:136) - EPISTEMIC Enabled: False; Samples: 5
[2023-08-25 07:39:21,537] [1183] [MainThread] [INFO] (main:96) - +++ Using Models: ['deepedit']
[2023-08-25 07:39:21,538] [1183] [MainThread] [INFO] (monailabel.interfaces.app:135) - Init Datastore for: /home/lukas/psoas-azd-images/train-images
[2023-08-25 07:39:21,538] [1183] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: False; Extensions: ['*.nii.gz', '*.nii', '*.nrrd', '*.jpg', '*.png', '*.tif', '*.svs', '*.xml']
[2023-08-25 07:39:21,564] [1183] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2023-08-25 07:39:21,613] [1183] [MainThread] [INFO] (main:126) - +++ Adding Inferer:: deepedit => <lib.infers.deepedit.DeepEdit object at 0x7fac4286f9d0>
[2023-08-25 07:39:21,613] [1183] [MainThread] [INFO] (main:126) - +++ Adding Inferer:: deepedit_seg => <lib.infers.deepedit.DeepEdit object at 0x7fac10134d60>
[2023-08-25 07:39:21,613] [1183] [MainThread] [INFO] (main:191) - {'deepedit': <lib.infers.deepedit.DeepEdit object at 0x7fac4286f9d0>, 'deepedit_seg': <lib.infers.deepedit.DeepEdit object at 0x7fac10134d60>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x7fac10134d90>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x7fac100a7130>}
[2023-08-25 07:39:21,614] [1183] [MainThread] [INFO] (main:206) - +++ Adding Trainer:: deepedit => <lib.trainers.deepedit.DeepEdit object at 0x7fac100a7190>
[2023-08-25 07:39:21,614] [1183] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/lukas/.cache/monailabel/sessions
[2023-08-25 07:39:21,614] [1183] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2023-08-25 07:39:21,615] [1183] [MainThread] [INFO] (monailabel.tasks.train.basic_train:432) - Train Request (input): {'model': 'deepedit', 'name': 'train_01', 'pretrained': False, 'device': 'Tesla T4:0', 'max_epochs': 5, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': False, 'gpus': 'all', 'dataset': 'Dataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'client_id': 'user-xyz', 'local_rank': 0}
[2023-08-25 07:39:21,615] [1183] [MainThread] [INFO] (monailabel.tasks.train.basic_train:445) - CUDA_VISIBLE_DEVICES: None
[2023-08-25 07:39:21,617] [1183] [MainThread] [INFO] (monailabel.tasks.train.basic_train:465) - Distributed Training = FALSE
[2023-08-25 07:39:21,617] [1183] [MainThread] [INFO] (monailabel.tasks.train.basic_train:492) - 0 - Train Request (final): {'name': 'train_01', 'pretrained': False, 'device': 'cuda:0', 'max_epochs': 5, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': False, 'gpus': 'all', 'dataset': 'Dataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'model': 'deepedit', 'client_id': 'user-xyz', 'local_rank': 0, 'run_id': '20230825_073921'}
[2023-08-25 07:39:21,617] [1183] [MainThread] [INFO] (monailabel.tasks.train.basic_train:625) - 0 - Using Device: cuda:0; IDX: 0
[2023-08-25 07:39:21,617] [1183] [MainThread] [INFO] (monailabel.tasks.train.basic_train:518) - Run/Output Path: /home/lukas/radiology/model/deepedit_dynunet/train_01
[2023-08-25 07:39:21,618] [1183] [MainThread] [INFO] (monailabel.tasks.train.basic_train:534) - Tracking: mlflow
[2023-08-25 07:39:21,618] [1183] [MainThread] [INFO] (monailabel.tasks.train.basic_train:535) - Tracking URI: file:///home/lukas/radiology/model/deepedit_dynunet/train_01/mlruns;
[2023-08-25 07:39:21,618] [1183] [MainThread] [INFO] (monailabel.tasks.train.basic_train:536) - Tracking Experiment Name: deepedit; Run Name: run_20230825_073921
[2023-08-25 07:39:21,618] [1183] [MainThread] [INFO] (monailabel.tasks.train.basic_train:410) - Total Records for Training: 16
[2023-08-25 07:39:21,618] [1183] [MainThread] [INFO] (monailabel.tasks.train.basic_train:411) - Total Records for Validation: 4
monai.handlers.stats_handler StatsHandler.__init__:name: Current default value of argument `name=None` has been deprecated since version 1.1. It will be changed to `name=StatsHandler` in version 1.3.
monai.transforms.io.dictionary LoadImaged.__init__:image_only: Current default value of argument `image_only=False` has been deprecated since version 1.1. It will be changed to `image_only=True` in version 1.3.
[2023-08-25 07:39:24,074] [1183] [MainThread] [INFO] (monailabel.tasks.train.basic_train:328) - 0 - Records for Validation: 4
[2023-08-25 07:39:24,082] [1183] [MainThread] [INFO] (monailabel.tasks.train.basic_train:318) - 0 - Adding Validation to run every '1' interval
[2023-08-25 07:39:24,084] [1183] [MainThread] [INFO] (monailabel.tasks.train.basic_train:264) - 0 - Records for Training: 16
[2023-08-25 07:39:24,089] [1183] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:876) - Engine run resuming from iteration 0, epoch 0 until 5 epochs
[2023-08-25 07:39:41,929] [1071] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: -9

I hope this helps in the debugging.

Kind regards

Lukas

[diazandr3s]

Hi @lukasvanderstricht,

The error:

2023-08-25 07:39:41,929] [1071] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: -9

Seems to be a memory error. You can confirm this by checking the memory consumed by the model. You can check with htop and nvidia-smi while training the model.