NVIDIA launches Cosmos Framework to train and run physical AI world models

If you are building autonomous vehicles or robots, you know the physical AI headache: training a world model to understand your specific camera angles and LiDAR sensors is brutally hard, and serving it fast enough for real-time control is even harder.

NVIDIA stepped into the ring with the release of the Cosmos Framework alongside Cosmos NIM microservices. The goal is straightforward: give developers an end-to-end stack to train, fine-tune, simulate, and serve physical AI world models.

From multi-camera AVs to surgical robots

The highlight of the release is a suite of task-specific Cosmos 3 Nano checkpoints that target high-stakes simulation.

Take autonomous driving. Instead of waiting for a thunderstorm to test driving perception, Cosmos3-Nano-Transfer-Auto can take existing sunny driving footage and denoise it into a heavy downpour while keeping road boundaries and vehicle paths identical. In a four-GPU reference setup, it outputs seven synchronized 720p camera views and 99 LiDAR sweeps. For real-time driving, its distilled sibling, Cosmos3-Nano-Sim-Auto, generates streaming observations autoregressively as human or policy inputs change.

The framework also pushes into robotics and surgery. Cosmos3-Nano-Sim-Bimanual handles interactive two-arm simulations for YAM and AgiBot setups. In operating rooms, where modeling soft tissue deformation has historically broken physics engines, Cosmos3-H-Embodied-Nano bypasses manual geometry altogether. It learns physical dynamics directly from surgical video and robot kinematics, while Cosmos3-H-Dreams delivers live, closed-loop interactive feedback before running procedures on physical hardware.

Training loops and speculative speedups

NVIDIA made Cosmos Framework modular, supporting external open-source models like Qwen, pi0.5, and NVIDIA GR00T. It handles both LoRA and full supervised fine-tuning, feeding on formats like WebDataset and LeRobot.

Reinforcement learning pipelines are fully stocked: offline DPO, online GRPO with vLLM rollouts, PPO for embodied policies, on-policy distillation, and DiffusionNFT for diffusion world models. You can even hand the training workflow off to coding agents using built-in skills to run smoke tests and evaluate precision trade-offs within a fixed GPU budget.

On the deployment side, Cosmos NIM microservices package these models into containerized endpoints. By applying speculative decoding, NVIDIA clocks a 2–3x inference speedup on H200 GPUs. Streaming VLM inference also cuts event response latency by 10x, enabling sub-second alerts for live video.

Why it matters

For developers in robotics and autonomous systems, Cosmos lowers the barrier to synthetic edge-case generation. Instead of hand-crafting 3D assets in traditional physics engines, teams can generate photorealistic, synchronized camera-and-LiDAR data with text prompts.

Hardware demands remain serious, but manageable. While running Nano generation in BF16 demands 58 GiB of VRAM, FP8 with offloading brings that down to 31 GiB, putting realistic evaluation within reach of high-end single-GPU workstations.

The broader industry takeaway is clear: NVIDIA is locking in its moat. LLM serving is increasingly commoditized, but physical AI requires tight integration across sensor simulation, reinforcement learning, and low-latency inference. By wrapping the entire physical development loop inside NIM containers and Hopper/Blackwell-optimized kernels, NVIDIA is making sure that whenever an AI interacts with the real world, its chips remain the default operating system.