AWS Shows How to Build a Physical AI Model Factory with NVIDIA Cosmos 3

AWS has published a comprehensive technical guide showing how organizations can build what it calls a "Physical AI model factory" — a continuous pipeline for training robots and autonomous vehicles using NVIDIA's Cosmos 3 model on Amazon SageMaker HyperPod.

The approach addresses a core challenge in physical AI development: these systems cannot be built with a single training run. Instead, they require a perpetual loop of generating synthetic training data, post-training perception and policy models, and evaluating results in closed-loop simulation. The guide, authored by AWS engineers Nathan Arnold and Eric Saleh and first reported on the AWS Machine Learning Blog, demonstrates how to run this entire pipeline on shared infrastructure rather than provisioning separate compute for each stage.

The Cosmos 3 architecture

NVIDIA Cosmos 3, released under the Linux Foundation's OpenMDW-1.1 license, uses what the company calls a Mixture-of-Transformers design. Unlike traditional approaches that pair separate models for video generation and vision-language understanding, Cosmos 3 runs a single trunk that handles video, images, actions, and sound as one unified token stream.

The architecture employs two experts joined at every layer: a reasoner that predicts the next token and a generator that denoises video, audio, and actions. Critically, the model runs asymmetrically between training and deployment. During training it executes a full denoising schedule and decodes video to pixels. On an actual robot, it runs only a few denoise steps and skips video decoding entirely, outputting just the action tokens the robot needs to execute.

The same base checkpoint can operate in three modes by changing which tokens start as noise: forward dynamics for generating synthetic data, inverse dynamics for labeling actions in unlabeled video, and policy mode for deployment on physical systems.

Why it matters

The guide tackles a practical economic problem in physical AI development. Organizations typically provision separate GPU capacity for data generation, training, and evaluation — each with its own lifecycle overhead. When capacity is reserved whether or not it's actively training, the relevant metric becomes "GPU goodput": useful pipeline progress per reserved GPU-hour across the entire loop, not peak throughput of any single job.

By unifying three traditionally separate model classes into one architecture, Cosmos 3 enables time-sharing a single persistent GPU pool rather than fragmenting capacity stage by stage. AWS's SageMaker HyperPod on Amazon EKS provides the infrastructure layer: one cluster control plane, continuous health monitoring with automatic node recovery, and pre-configured Elastic Fabric Adapter networking for multi-node training.

Implementation details

The guide covers three representative workloads running on p5en.48xlarge instances (8x NVIDIA H200 GPUs): robot manipulation policy training on the public DROID dataset, video supervised fine-tuning on the 16-billion-parameter Cosmos3-Nano model, and LoRA fine-tuning on the 64-billion-parameter Cosmos3-Super model with context parallelism.

All stages share a single Amazon FSx for Lustre filesystem backed by Amazon S3, eliminating terabyte-scale data migrations between pipeline stages. The generation engine runs on vLLM-Omni, post-training uses NVIDIA's cosmos-framework with PyTorch FSDP2, and evaluation runs on a single-GPU policy server — all orchestrated as standard Kubernetes workloads.

The complete implementation, including infrastructure templates and job manifests, is available in AWS's awsome-distributed-ai GitHub repository. The guide was published by AWS on September 4, 2026.