Unexpected Spike in train/kobj_loss at Epoch 90 While Training YOLOv8m Pose Estimation Model

[UranLajciSolaborate]

I am observing a peculiar behavior in the train/kobj_loss (Keypoint Objectness Loss) during training. Specifically, around epoch 90, there's a sudden spike in the kobj_loss, which increases from 0.149 to 0.196, before stabilizing and decreasing again after epoch 90 (see attached image for reference).

My questions are:

What could be causing this sudden spike in kobj_loss?
Are there any typical adjustments or investigations I should perform to address this issue?
Is this behavior expected in any case, such as learning rate changes or data-related factors?

I’d appreciate any insights or suggestions on what might be causing this issue and how to address it.

[UltralyticsAssistant]

👋 Hello @UranLajciSolaborate, thank you for reaching out and for using Ultralytics 🚀! Your curiosity about the spike in train/kobj_loss is understandable, and we'll certainly work through it.

To assist further, if this is a 🐛 Bug Report, please provide a minimum reproducible example to help us analyze it.

🔍 If you're seeking advice on training dynamics, such as sudden loss spikes, consider these steps:

1. Ensure your training setup aligns with our Tips for Best Training Results.
2. Check if any scheduled learning rate changes are impacting your results.
3. Verify your dataset consistency to rule out potential data anomalies.

For real-time troubleshooting, join us on Discord 🎧 or explore detailed threads on our Discourse.

Upgrade

Upgrade to the latest ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8 to verify your issue is not already resolved in the latest version:

pip install -U ultralytics

Environments

YOLOv8 can be run in any of the following up-to-date verified environments:

• Notebooks:
• Google Cloud: See GCP Quickstart Guide
• Amazon: See AWS Quickstart Guide
• Docker Image: See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing.

This is an automated response, but rest assured, an Ultralytics engineer will assist soon. Thank you for your patience! 😊

[glenn-jocher]

@UranLajciSolaborate close mosaic kicks in 10 epochs before the end of training to help the model adapt to more real-world data it might encounter rather than the earlier training mosaics it is provided. This is causing the changes in loss you see.

[UranLajciSolaborate]

What are your reasons?

[glenn-jocher]

The loss spike is clearly correlated with close mosaic since these both happen at epoch 90.

[UranLajciSolaborate]

Okey. Currently I have set close mosaic set to 10 and I am getting this spike. What do you propose to do in order to escape it. Should I make this argument 20 or some larger number or remove it completely.

[glenn-jocher]

You can try increasing the close mosaic value to 20 to see if it stabilizes the loss, or experiment with removing it to observe the effect on your model's performance.

[UranLajciSolaborate]

Hello @glenn-jocher thanks for the Q&A interaction.

I set the close_mosaic to 20 but the loss got worse as seen in the plot below:

I will train my model by removing it completely to see its effects.

[UranLajciSolaborate]

By setting the close mosaic to 0 we handled this case of the loss spike, as seen in the loss plots below:

[glenn-jocher]

@UranLajciSolaborate great to hear that setting close mosaic to 0 resolved the loss spike issue. If you have further questions, feel free to ask!