MLCommons collaboration opportunities with MONAI Deploy

[dbericat]

MLPerf today measures model accuracy, not inference performance metrics. It felt like today, that part is a better fit for MONAI Core. @wyli . Also important to connect with Benchmarking WG. @lena Maier-Hein

For MONAI Deploy and how to connect bringing it to the clinic, here are couple of ideas to explore further in future meetings (in no particular order):

1. Expand model evaluation (Model archive at MONAI Core) to application evaluation (MAP). How would they compare?
   1.1. Is there an intermediate step where App SDK sends app to MedPerf server (via App Packager)?
   1.2. App SDK App Runner connection with MedPerf Server?
2. MLCube goals vs MONAI Deploy goals. Where does MLCube fit?

Please lay out others and let's discuss! :)

P..S. Slides presented for reference:

MLCommons MedPerf general slides
MedPerf Development Status

[msheller]

Thanks for this thread!

I expect that there are a number of interesting goals and ways to integrate. From the MedPerf perspective, our first thoughts are around how MAP containers could be used in MedPerf benchmark pipelines. Direct integration of the MAR and MedPerf server seems like a future step from a technical point of view, but perhaps there are reasons to start there?

Looking at the first goal of enabling use of MAP containers in MedPerf benchmarks, the obvious solutions in my mind are to:

1. have the MedPerf client use the MAR to run MAP containers.
2. provide a means to automatically convert/wrap MAP containers to/with MLCubes

Option 1 is what we do with MLCubes. It would also require backend changes on the MedPerf server to store sufficient metadata about the MAP container to tell the MedPerf client how to verify the container and call the MAR. I expect this would be simple enough. If not, however, option 2 might make more sense, as it would require no change to the MedPerf server.

Does the goal of enabling MedPerf to run MAP containers in benchmarks make sense as the right first step?