Awesome-Robotics-Foundation-Model-How-To

A curated list of tools, needed to train robotics foundation models from scratch.

Table of Contents

• Awesome-Robotics-Foundation-Model-How-To
  • Table of Contents
  • How to build a foundation model from scratch?
    • Technical Background
    • Models (e.g. Vision Language Models, Robotics Foundation Models)
    • Datasets and Benchmarks
    • Training Pipeline and deployment
    • Simulators and Environments
    • Tools
  • Contribute

How to build a foundation model from scratch?

Technical Background

Date	keywords	Institute	Paper
2017-06	Transformers	Google	Attention Is All You Need
2018-06	GPT 1.0	OpenAI	Improving Language Understanding by Generative Pre-Training

Models (e.g. Vision Language Models, Robotics Foundation Models)

Date	keywords	Institute	Paper
2022-12	VLA	Google	RT-1: Robotics Transformer for Real-World Control at Scale
2023-07	VLA	Google DeepMind	RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
2023-10	VLA	Google et al.	Open X-Embodiment: Robotic Learning Datasets and RT-X Models
2024-05	VLA	UC Berkley et al.	Octo: An Open-Source Generalist Robot Policy

Datasets and Benchmarks

Date	keywords	Institute	Paper
2024-02	Dataset	Stanford et al.	Pushing the Limits of Cross-Embodiment Learning for Manipulation and Navigation
2024-05	Simulation	UCSD et al.	Evaluating Real-World Robot Manipulation Policies in Simulation
2024-03	Dataset	UCB et al.	DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
2023-10	Dataset	Google et al.	Open X-Embodiment: Robotic Learning Datasets and RT-X Models
2023-06	Simulation	Meta AI	HomeRobot: Open-Vocabulary Mobile Manipulation
2023-09	Dataset	Stanford et al.	Physically Grounded Vision-Language Models for Robotic Manipulation
2023-11	Dataset	Google DeepMind	RoboVQA: Multimodal Long-Horizon Reasoning for Robotics

Training Pipeline and deployment

Date	keywords	Institute	Paper
2018-06	GPT 1.0	OpenAI	Improving Language Understanding by Generative Pre-Training
2024-08	LLM	Lightning AI et al.	Build a Large Language Model (From Scratch)

Simulators and Environments

Date	keywords	Institute	Paper
	Mujoco	Google	Attention Is All You Need
	Calcin	OpenAI	Improving Language Understanding by Generative Pre-Training

Tools

Date	keywords	Institute	Paper
	Fog-RT-X	BerkleyAutomation	Fog-RT-X

Contribute

Contributions welcome! Read the contribution guidelines first.

This list was generated with the help of generator-awesome-list.