Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FR] Add mean average recall (mAR) to evaluate_detections #4972

Open
2 of 6 tasks
NicDionne opened this issue Oct 23, 2024 · 1 comment
Open
2 of 6 tasks

[FR] Add mean average recall (mAR) to evaluate_detections #4972

NicDionne opened this issue Oct 23, 2024 · 1 comment
Labels
feature Work on a feature request

Comments

@NicDionne
Copy link

NicDionne commented Oct 23, 2024

Proposal Summary

In a few sentences, provide a clear, high-level description of the feature request.
I propose to add a parameter to evaluate_detections. In the same way we can set compute_mAP=True, we should be able to set compute_mAR = True.

Motivation

  • What is the use case for this feature?
  • It is said that fiftyone evaluate using the COCO evaluation protocol https://cocodataset.org/#detection-eval. But this evaluation protocol also expose the mAR.
  • Why is this use case valuable to support for FiftyOne users in general?
  • mAR is a metric used in multiple contexts for evaluation.
  • Why is this use case valuable to support for your project(s) or organization?
  • This would facilate the use of Fiftyone to evaluate detections.

What areas of FiftyOne does this feature affect?

  • App: FiftyOne application
  • Core: Core fiftyone Python library
  • Server: FiftyOne server

Details

Use this section to include any additional information about the feature. If you have a proposal for how to implement this feature, please include it here.

Willingness to contribute

The FiftyOne Community welcomes contributions! Would you or another member of your organization be willing to contribute an implementation of this feature?

  • Yes. I can contribute this feature independently
  • Yes. I would be willing to contribute this feature with guidance from the FiftyOne community
  • No. I cannot contribute this feature at this time
@NicDionne NicDionne added the feature Work on a feature request label Oct 23, 2024
@brimoor
Copy link
Contributor

brimoor commented Oct 24, 2024

Hi @NicDionne 👋

Adding mAR makes sense to me! A PR that contributes this would be most welcome! 😄

The compute_mAP=True parameter was perhaps an unnecessarily specific naming choice. Functionally it just performs the necessary IoU sweep so that mAP and PR curves can be generated, but it also should provide the necessary data to compute mAR as well.

So, I believe that it would be cleanest to implement mAR by leaving this syntax unchanged:

dataset.evaluate_detections(..., compute_mAP=True)

and then simply adding an mAR() method analogous to the existing mAP() method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Work on a feature request
Projects
None yet
Development

No branches or pull requests

2 participants