Skip to content

[ Back to index ]

MLCommons Task Force on Automation and Reproducibility

News (May 2024): our task force has successfully accomplished the first goal to provide a stable CM interface for MLPerf benchmarks and discussing the next steps with MLCommons - please stay tuned for more details!

Mission

Sponsors

We thank cKnowledge.org, cTuning.org, and MLCommons for sponsoring this project!

Citing CM

If you found CM useful, please cite this article: [ ArXiv ], [ BibTex ].

Current projects

  • Continue improving CM to support different MLCommons projects for universal benchmarking and optimization across different platforms.

  • Extend CM workflows to reproduce MLPerf inference v4.0 submissions (Intel, Nvidia, Qualcomm, Google, Red Hat, etc) via a unified interface.

  • Prepare tutorial for MLPerf inference v4.1 submissions via CM.

  • Discuss how to introduce the CM automation badge to MLPerf inference v4.1 submission similar to ACM/IEEE/NeurIPS reproducibility badges to make it easier for all submitters to re-run and reproduce each others’ results before the publication date.

  • Develop a more universal Python and C++ wrapper for the MLPerf loadgen with the CM automation to support different models, data sets, software and hardware: Python prototype; C++ prototype.

  • Collaborate with system vendors and cloud providers to help them benchmark their platforms using the best available MLPerf inference implementation.

  • Collaborate with other MLCommons working groups to autoamte, modularize and unify their benchmarks using CM automation recipes.

  • Use CM to modularize and automate the upcoming automotive benchmark.

  • Use MLCommons Croissant to unify MLPerf datasets.

Current tasks

  • Improving CM workflow automation framework: GitHub ticket
  • Updating/refactoring CM docs (framework and MLPef workflows): GitHub ticket
  • Improving CM scripts to support MLPerf: GitHub ticket
  • Adding profiling and performance analysis during benchmarking: GitHub ticket
  • Improving universal build and run scripts to support cross-platform compilation: GitHub ticket
  • Automate ABTF benchmarking via CM: GitHub ticket
  • Help automate MLPerf inference benchmark at the Student Cluster Competition'24: GitHub ticket

Completed deliverables

Resources

Acknowledgments

This task force was established by Grigori Fursin after he donated his CK and CM automation technology to MLCommons in 2022 to benefit everyone. Since then, this open-source technology is being developed as a community effort based on user feedback. We would like to thank all our volunteers, collaborators and contributors for their support, fruitful discussions, and useful feedback!