The "Collective Knowledge" project (CK) is motivated by the feedback from researchers and practitioners while reproducing results from more than 150 research papers and validating them in the real world - there is a need for a common and technology-agnostic framework that can facilitate reproducible research and simplify technology transfer to production across diverse and rapidly evolving software, hardware, models, and data. It consists of the following sub-projects:
-
Collective Mind scripting language (MLCommons CM) is intended to help researchers and practitioners describe all the steps required to reproduce their experiments across any software, hardware, and data in a common and technology-agnostic way. It is powered by Python, JSON and/or YAML meta descriptions, and a unified CLI. CM can automatically generate unified README and synthesize unified containers with a common API while reducing all the tedious, manual, repetitive, and ad-hoc efforts to validate research projects in production. It is used in the same way in native environments, Python virtual environments, and containers.
See a few real-world examples of using the CM scripting language:
- README to reproduce published IPOL'22 paper
- README to reproduce MLPerf RetinaNet inference benchmark at Student Cluster Competition'22
- Auto-generated READMEs to reproduce official MLPerf BERT inference benchmark v3.0 submission with a model from the Hugging Face Zoo
- Auto-generated Docker containers to run and reproduce MLPerf inference benchmark
-
Collective Mind scripts (MLCommons CM scripts) provide a low-level implementation of the high-level and technology-agnostic CM language.
-
Collective Knowledge platform (MLCommons CK playground) aggregates reproducible experiments in the CM format, connects academia and industry to organize reproducibility, replicability and optimization challenges, and help developers and users select Pareto-optimal end-to-end applications and systems based on their requirements and constraints (cost, performance, power consumption, accuracy, etc).
This open-source technology is being developed by the public MLCommons task force on automation and reproducibility led by Grigori Fursin and Arjun Suresh. The goal is to connect academia and industry to develop, benchmark, compare, synthesize, and deploy Pareto-efficient AI and ML systems and applications (optimal trade off between performance, accuracy, power consumption, and price) in a unified, automated and reproducible way while slashing all development and operational costs.
- Join our public Discord server.
- Join our public conf-calls.
- Check our news.
- Check our presentation and Forbes article about our development plans.
- Read about our CK concept (previous version before MLCommons).
2021-2023 MLCommons
This project is currently supported by MLCommons, cTuning foundation, cKnowledge and individual contributors. We thank HiPEAC and OctoML for sponsoring initial development.