Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Colab disconnects and all the files in the colab VM are lost #4

Open
rishabh700 opened this issue Jul 17, 2020 · 1 comment
Open

Colab disconnects and all the files in the colab VM are lost #4

rishabh700 opened this issue Jul 17, 2020 · 1 comment

Comments

@rishabh700
Copy link

In case when the colab gets disconnected in the middle of the training, all the files in the VM are lost (such as the darknet folder). Does this mean that I will have to clone the darknet repo and upload all the other pre-requisite files (images, labels, etc) again to start the training?
If yes, would it be helpful to also backup the darknet folder with the gdrive (in addition to the weights file)?

@adhilcodes
Copy link

1 ) Yes you have to clone the darknet repo and upload all the other pre-requisite files (images, labels, etc) again to start the training once it is terminated from the training, the storage in ColabVM is temporary .

  1. After every 100 iterations your weights are saved in the backup folder in your Gdrive , Also after every 1000 iterations also your weights get saved in the backup folder. So if you terminate from the training you can restart in from where did you stopped it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants