Sharing training data/fine-tuned model

[camappel]

I've been trying to fine-tune the live-stock detection model using a large labelled dataset obtained from Harvard Dataverse, and I'd like to discuss best practices regarding open-source training data and model training.

I have made the data available here:

There are 2497 training annotations
There are 896 validation annotations

An abridged version of the training process is available here. I used the following hyperparameters:

model.config['gpus'] = '-1'  # Use GPU (set to '0' for the first GPU or '-1' for all GPUs)
model.config["train"]["csv_file"] = train_file  # Path to training CSV
model.config["train"]["root_dir"] = os.path.dirname(train_file)  # Root directory for training images
model.config["score_thresh"] = 0.4  # Set score threshold
model.config["train"]['epochs'] = 5  # Number of epochs
model.config["validation"]["csv_file"] = valid_file  # Path to validation CSV
model.config["validation"]["root_dir"] = os.path.dirname(valid_file)

callback = ModelCheckpoint(
    dirpath='checkpoints/',  # Directory to save checkpoints
    monitor='box_recall',  # Metric to monitor
    mode="max",  # Save when the metric is maximized
    save_top_k=3,  # Save the top 3 checkpoints
    filename="box_recall-{epoch:02d}-{box_recall:.2f}"  # File name format for checkpoints
)

My goal is to evaluate the model's performance on a validation dataset before and after fine-tuning. Assuming the new dataset yields some increased accuracy, is there somewhere I can upload the updated weights?

[ethanwhite]

@camappel - did you close this on purpose? Just trying to figure out if you still want feedback or decided to wait

[camappel]

I thought it might be better to wait until I have a .safetensor file to share, but would greatly appreciate feedback in the meantime!

[ethanwhite]

Great! Your goal will guide a lot of this so can you tell us what you're hoping to accomplish? Is it a better general livestock detector, a general cow detector, a model that works for well for this particular dataset, etc?

[camappel]

The aim is a general cow detector. Is it OK that the training dataset will be unbalanced (predominately images from one field in the Netherlands)?

[camappel]

After training the prebuilt livestock detector model on a dataset of 3,390 labelled cows (2269 training annotations, 785 validation annotations, and 339 for testing from the same dataset) from fields in Juchowo, Poland and Wageningen, the Netherlands, the evaluation metrics improved from:

Baseline average per-image recall: 0.41216216216216217
Baseline average per-image precision: 0.4238887521496217

Fine-tuned average per-image recall: 0.9428512928512929
Fine-tuned average per-image precision: 0.7844495947437123

The updated model is saved as a .safetensor file here; can I open a PR on the huggingface? The aim of the project is to demonstrate the open source process of fine-tuning existing models (in order to obtain more accurate detections of livestock in this case)

[ethanwhite]

Wow, that's a really great improvement @camappel!

The fact that it's primarily a single data source will put some limits on how well it generalizes, but that doesn't mean it's not a useful step that other folks will find valuable.

A few general thoughts to get us started:

1. Since the image data is provided by another source it's generally good practice to avoid redistributing it separately there is a decent way to let users download it from the original source. This lets the original source get credit for the downloads and avoids you being responsible for maintaining access to imagery.
2. In terms of sharing the resulting model I think there are a couple of general ways to go. First, you can also place the model on some sort of model sharing site. We use https://huggingface.co/, which is the most popular approach to this right now. We're about to merge some changes that will make it possible for others to easily download, load, run, and fine tune the model all directly from DeepForest even if it's in your own HuggingFace account. Second, we could talk about putting it in a weecology deepforest repo and adding it to our list of models. I think the open question for me is whether it generalizes well enough to be broadly useful and I'd like to get @bw4sz's take on whether he things this makes sense to add this officially to the deepforest suite of models.
3. Something else we could talk about if you're interested is retraining the livestock detector but adding these new data into the training/test set. This would almost certainly help improve it's performance.
4. I definitely like the idea of demonstrating the process of fine-tuning existing models. This is commonly known in a lot of computer vision contexts, but much less well appreciate in the ecology/conservation/management application space.

[camappel]

Thanks a lot for your feedback!

I'll try to avoid redistributing the image data. This may be difficult in the context of the notebook, however, as each user would have to download all the composite zips in the entire dataset within the Binder environment, then recombine with pyunpack, just to get a test set.

Re. sharing the model, I'd be happy with whatever you decide, and I'd be keen to contribute. For now, I've uploaded the model to HuggingFace and will open a pull request once I've fixed the metadata

[camappel]

I'm working on finishing this right now and I noticed the new load_model function; should I wait until this is added to pip?

FYI:

!pip install git+https://bitbucket.org/weecology/deepforest

Collecting git+https://bitbucket.org/weecology/deepforest
  Cloning https://bitbucket.org/weecology/deepforest to /tmp/pip-req-build-092sfjrs
  Running command git clone --filter=blob:none --quiet https://bitbucket.org/weecology/deepforest /tmp/pip-req-build-092sfjrs
  remote: You may not have access to this repository or it no longer exists in this workspace. If you think this repository exists and you have access, make sure you are authenticated.
  fatal: repository 'https://bitbucket.org/weecology/deepforest/' not found
  error: subprocess-exited-with-error
  
  × git clone --filter=blob:none --quiet https://bitbucket.org/weecology/deepforest /tmp/pip-req-build-092sfjrs did not run successfully.
  │ exit code: 128
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git clone --filter=blob:none --quiet https://bitbucket.org/weecology/deepforest /tmp/pip-req-build-092sfjrs did not run successfully.
│ exit code: 128
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[bw4sz]

looks like you tried to download from bitbucket, DeepForest is on github.

pip install git+https://github.com/Weecology/DeepForest.git

[camappel]

Thanks you, load_model now works as expected.

I did encounter one bug, however (possibly due to running it in a Colab notebook environment):

!pip install git+https://github.com/Weecology/DeepForest.git
deepforest import main

ImportError: cannot import name 'RevisionNotFoundError' from 'huggingface_hub.errors' (/usr/local/lib/python3.10/dist-packages/huggingface_hub/errors.py)

Fix:

!pip install git+https://github.com/Weecology/DeepForest.git
!pip install --upgrade huggingface_hub

WARNING: The following packages were previously imported in this runtime:
  [huggingface_hub]
You must restart the runtime in order to use newly installed versions.

[camappel]

PR with updated checkpoint opened here