NVIDIA Releases Cosmos 3 Edge: A 4B-Parameter Open World Model That Reasons and Generates Robot Actions On-Device
0
0
NVIDIA has released Cosmos 3 Edge, a 4-billion-parameter open world model built to run on-device. It helps robots and vision AI agents understand surroundings, reason in real time, and generate robot actions locally.
The Cosmos 3 family included Cosmos 3 Nano (16B) and Cosmos 3 Super (64B) shipped on May 31, 2026 at GTC Taipei. Edge is the third and smallest tier, at roughly one-sixteenth the size of Super.
The problem is specific. Machines operate at the edge in factories, warehouses, and hospitals. They need data center–level performance on memory-constrained systems. Cosmos 3 Edge targets that gap.
What does world model do here
A world model learns how an environment changes over time. It represents objects, motion, spatial relationships, and the effects of actions.
Consider a robot reaching for an object. Recognizing the object is only the first step. The robot must also track where the object is, how its gripper moves, and what happens on contact. A world model reasons about these relationships. It can predict the visual result of an action, infer the action that caused a change, or generate an action to reach a goal.
Cosmos 3 Edge brings these capabilities into one on-device model. Its shared representation lets a system understand the current world state, simulate possible futures, and connect those futures to actions.
Two transformer towers, one shared representation
Cosmos 3 uses a Mixture-of-Transformers architecture with two towers, described in the NVIDIA’s technical report.
The autoregressive tower processes vision and text tokens for understanding and reasoning. The diffusion tower processes vision, audio, and action tokens for prediction, generation, and neural simulation.
The two towers keep separate normalization layers and multilayer perceptrons. They share multimodal attention layers, which align information across language, video, audio, and action. This lets the model reason about a scene before it generates an output.
The attention pattern adapts to each modality. Language uses causal attention, where each token attends to earlier tokens. Diffusion tokens attend more broadly to the available context, supporting coherent prediction and generation. Depending on the task, the model emits reasoning tokens from the autoregressive tower, or denoised video and action tokens from the diffusion tower.
Cosmos 3 Edge uses a 2B dense transformer for its reasoner, and follows Qwen3-VL-compatible message conventions for image and video inputs, per the Cosmos GitHub repository.
One action representation across embodiments
Physical systems describe actions differently. A vehicle uses ego pose and movement. A camera uses camera motion. A robot arm uses the pose of its end effector, and a gripper adds grasp state.
Cosmos 3 maps these embodiments into a common action representation. Actions are encoded as compact geometric vectors that capture translation, rotation, and manipulation state.
This connects control to the visual structure of the world. The model associates pixel changes with physical motion and control inputs. Generated video then becomes more than a prediction. It represents how the world should change in response to an action.
Supported action dimensions depend on the embodiment. The Cosmos GitHub repository lists camera motion (9D), autonomous vehicle (9D), egocentric motion (57D), single-arm robot (10D), dual-arm robot (20D), and humanoid robot (29D).
Policy mode runs in both directions
As a policy, Cosmos 3 Edge predicts an action together with its expected visual consequence. Current state goes in; an action and its likely visual outcome come out.
Action flows in both directions. The model can predict the effect of an action, or infer the action from its effect. This connects world modeling directly to robot policy training and evaluation.
NVIDIA also released Cosmos 3 Edge Policy (DROID). It is a robot manipulation policy post-trained on the DROID dataset for pick-and-place tasks, with post-training scripts included. Developers can fine-tune on a small H100 cluster or an NVIDIA DGX Station before deployment.
Is it Deployable
Cosmos 3 Edge delivers memory-efficient inference across NVIDIA edge computers. Targets include NVIDIA RTX PRO GPUs, NVIDIA DGX, GeForce RTX GPUs, and NVIDIA Jetson, including the newly announced Jetson T2000 and T3000 modules.
As a post-trained world action model (WAM), the model operates at robot-control resolution of 640×360 observations. On NVIDIA Jetson Thor it generates 32 actions per inference, while achieving real-time control at 15 Hz. For generation, the Edge tier supports 256p and 480p resolutions, 12–30 fps, and 50–150 frames.
Using the open Cosmos framework, developers can post-train Cosmos 3 Edge for a specific embodiment and sensor set in about a day. NVIDIA positions a GeForce RTX 3070 or better as a local on-ramp for prototyping.
Key Takeaways
- Cosmos 3 Edge is a 4B open world model (2B dense reasoner) that runs on-device, released July 20 on Hugging Face.
- A Mixture-of-Transformers design pairs an autoregressive reasoner tower with a diffusion generator tower through shared multimodal attention.
- It hits 640×360 control resolution, 32 actions per inference, and 15 Hz real-time control on NVIDIA Jetson Thor.
- Actions map to a common translation/rotation/manipulation representation, spanning camera, vehicle, single-arm, dual-arm, and humanoid embodiments.
- Benchmarks (#1 on VANTAGE-Bench at 4B) are internally claimed; the model ships under Linux Foundation OpenMDW-1.1.
Sources: Hugging Face launch post, Cosmos3-Edge model card, Cosmos 3 collection, NVIDIA technical report (PDF), Cosmos GitHub, NVIDIA developer blog, Jetson Thor blog and NVIDIA Newsroom: Japan coalition
The post NVIDIA Releases Cosmos 3 Edge: A 4B-Parameter Open World Model That Reasons and Generates Robot Actions On-Device appeared first on MarkTechPost.
MarkTechPost
