Open models and the future of Physical AI with NVIDIA

Open code and models helped Mingyu Liu learn computer vision and machine learning, experiment, and build new systems, but he says they have become less available as the resources needed to develop models have grown. APIs can solve problems, but they provide less insight into how a model works and less opportunity to customize it, and may not meet needs for local computing or offline operation. Open models give developers a way to inspect and test capabilities, adapt them to their own applications, and potentially build on them.

Liu defines Physical AI as AI deployed in devices that act on the world and produce tangible results, including vehicles, factory automation, robots, agriculture, and construction. He describes current AI in physical devices as preliminary. A physical-AI policy model maps visual observations and user or machine interaction to actions, but devices differ in their sensors and operating conditions, so one model may not fit all. Vehicle configurations discussed include setups with 11 or seven cameras, and some vehicles also use LiDAR. Open models can be fine-tuned with device-specific data, such as adapting capabilities developed for one camera view to multiple views.

Physical AI also changes the computing requirements. Unlike an API that can aggregate requests from multiple users, an embedded system may need to process each request in real time, with batch-size-one inference and on-device computation. A transformer optimized to process aggregated API requests may use compute less efficiently on a device, while application needs and safety constraints can favor different architectures. Liu identifies improving “intelligence per watt” as a primary direction: systems need to deliver more capability within the power limits of their physical platforms.

Liu defines a world model broadly as a model of the world used to solve tasks, rather than by a particular set of inputs or outputs. He distinguishes world understanding, which represents what happened and can explain it in language, from world simulation, which predicts what will happen given observations and actions. He says Cosmos 3 combines both because they concern the same world and can share representations; it supports text, action, audio, and video inputs, and text, audio, video, and action outputs. Classical simulators encode physics through equations and programs, while neural simulation learns patterns from observations to predict outcomes.

For self-driving development, world models can help test policies in simulation before selected candidates are tried in real cars, allowing faster iteration. But simulation only narrows the hypotheses: promising candidates still need real-world testing. Liu says shared representations of the world can also connect predicted state changes to control signals and help regularize policy models when data is limited. The benefit therefore depends on using simulation to guide testing, not treating its predictions as a substitute for evidence from the physical world.

Liu expects Physical AI systems to combine specialized agents with robots, mobile platforms, self-driving trucks, and inspection systems, coordinating digital planning with physical tasks such as manufacturing. Agents will need task autonomy and the ability to correct themselves, but different tasks require different capabilities; intelligence per dollar matters alongside broad problem-solving ability. He cautions that multi-agent coordination across production may be decades away and that human-like robots are not ready for major commercial use.

Liu says developing Physical AI requires an ecosystem rather than the work of one company alone. NVIDIA’s open-model resources include open weights, training frameworks for adapting models with users’ data, and open-source data, as well as developer blogs, example recipes, and use cases. Openness can give builders more ways to customize models, but correctness still has to be verified; Liu describes supervisor agents monitoring factory scenes as a possible approach, while noting that the relevant scenes have not yet been identified.