Open Source Self-Driving with Comma AI

Harold Schaefer, CTO at Comma AI, discusses the open-source autonomy stack, OpenPilot, which enhances vehicles with features like auto steer and advanced adaptive cruise control (ACC). He notes that when he joined Comma AI, the project was nascent, lacking a fully integrated product and requiring complex setups. OpenPilot now accounts for over 50% of miles driven autonomously, contrasting with Tesla's Full Self-Driving and Waymo's supervised robot taxi services.

Schaefer reflects on the evolution of self-driving technology, emphasizing that while advancements have been made, full autonomy is still out of reach. He highlights the appeal of self-driving as an applied robotics problem, offering immediate user benefits. The open-source philosophy of OpenPilot fosters community collaboration and innovation.

The architecture of OpenPilot includes a device with computing power, cameras, GPS, and IMU sensors, which runs machine learning models to determine driving actions based on video input. The system outputs commands for acceleration and steering, interfacing with various car models through reverse-engineered CAN messages. Training involves analyzing hundreds of millions of miles of human driving data, allowing the system to learn from mistakes.

OpenPilot aims to be a general robotics platform focused on driving, with components like a user interface and a localizer for motion estimation. The development approach prioritizes incremental progress, shipping useful features while navigating resource limitations compared to larger competitors.

The podcast discusses end-to-end solutions in autonomous driving, emphasizing the use of raw sensor data to train machine learning models. Schaefer notes that while competitors like Waymo and Tesla explore end-to-end methods, they still rely on classical detection techniques. Comma AI's focus on end-to-end solutions aligns with industry trends and represents a long-term strategy.

The concept of a world model is introduced, serving as a simulator for training and allowing for accurate control based on real video inputs. The importance of a high-quality, photorealistic simulator is emphasized, as previous classical simulators had limitations. The data center is dedicated to training and simulation, with real-time inference occurring on the device.

User experience is highlighted, with installation involving a connection to the car's canvas. The system is reliable on highways, with over 50% of user miles driven autonomously, and there is a focus on improving city driving behavior, particularly in handling red lights.

Schaefer expresses interest in enhancing existing cars with autonomy and exploring alternative applications, including a world model simulator for traffic light management and indoor robotics. He identifies three critical unsolved problems in robotics: controls, reinforcement learning (RL), and continual learning. The company faces challenges in how cars respond to steering and gas requests, often relying on classical control solutions.

The motivation for developing OpenPilot from scratch is discussed, highlighting its efficiency in inter-process communication. The decision to make OpenPilot open source is seen as vital for supporting a wide range of cars and fostering community contributions. The programming language usage in OpenPilot includes a mix of Python and C++, with Python favored for its simplicity in development.

The conversation reiterates the challenges of controls, reinforcement learning, and continuous learning. Schaefer emphasizes the desire for practical applications of technology that simplify daily tasks and the importance of open-source solutions for user control and privacy. The discussion concludes with appreciation for the insights shared and anticipation for future technological advancements.