Dynamic soft margin loss & Ranked List Loss

[domenicoMuscill0]

I have implemented these two loss functions. About the DSM loss i have added the possibility to use labels in case the user wants to, but it can be easily used as a self-supervised loss function

[KevinMusgrave]

Ignore the crashing "trainers" workflow.

Are the losses tests passing on your system?

[domenicoMuscill0]

The losses's tests pass the majority of the time though sometimes i get the same assertion error; those times the difference between the correct loss and my loss is of 3e-4, but since other losses i created showed the same behavior (provided that the workflow tests passed everytime without any problem) i kept pushing to github.
I have checked the actual value of the tensors with the debugger and besides some small (approximation) errors the quantities are equal.

[KevinMusgrave]

Thank you @domenicoMuscill0!

[domenicoMuscill0]

Greetings, may i point out that the two issues related to this pull request are still open? Also, is there any high priority issue/algorithm that you would prefer me to put effort on? I was looking at the ProxyNCA++ loss function, but if there is something else i can change quite easily. Best regards, Domenico Muscillo Il Mar 12 Dic 2023, 08:34 Kevin Musgrave ***@***.***> ha scritto:

…

Thank you @domenicoMuscill0 <https://github.com/domenicoMuscill0>! — Reply to this email directly, view it on GitHub <#659 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYI7GXRQLD7PWYIOLZULRWTYJACIFAVCNFSM6AAAAAA4AHQKLSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJRGQ2DKMRXGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[KevinMusgrave]

Greetings, may i point out that the two issues related to this pull request
are still open?

Thanks for reminding me. I now uploaded v2.4.0 and closed those issues.

Also, is there any high priority issue/algorithm that you
would prefer me to put effort on? I was looking at the ProxyNCA++ loss
function, but if there is something else i can change quite easily.

ProxyNCA++ might be good, but:

• I think Section 3.3 says they are using a softmax formulation, whereas in the original NCA/ProxyNCA, the denominator doesn't include the "positive" relationship. However, I think our current implementation of NCA/ProxyNCA is already doing the softmax approach:
  • https://github.com/KevinMusgrave/pytorch-metric-learning/blob/master/src/pytorch_metric_learning/losses/nca_loss.py
  • https://github.com/KevinMusgrave/pytorch-metric-learning/blob/master/src/pytorch_metric_learning/losses/proxy_losses.py
• Section 3.4 suggests adding temperature scaling, but this is already part of NCA/ProxyNCA as softmax_scale (the inverse of temperature).
• Section 3.5 and 3.6 are about model architecture and learning rates.
• Section 3.7 is a batch sampling scheme that I think is equivalent to MPerClassSampler: https://kevinmusgrave.github.io/pytorch-metric-learning/samplers/#mperclasssampler