Hesai Launches Kosmo 3D World Data Engine to Solve Real-World Data Scarcity for Physical AI

THE ESSENTIALS

Kosmo captures physical spaces using lidar, imaging and motion data, then applies AI to generate labeled 3D scenes suitable for robot simulation and world model training.
A single Kosmo capture can produce more than 10,000 training trajectories, addressing the real-world data scarcity that limits Physical AI development.
In a demo with robot maker Sharpa, a 15-minute Kosmo scan replaced approximately 300 hours of manual 3D modeling, and the robot achieved a 99% task-success rate in the reconstructed environment.
Kosmo integrates with NVIDIA Isaac Sim, Unreal Engine, RenderMan and Houdini, and supports open access to raw capture data and 3D assets.

Data Infrastructure for Physical AI

Unlike digital AI models that train on vast online datasets, robots operating in the physical world face a persistent shortage of accurate, scalable real-world data. Hesai, a Shanghai-headquartered lidar and sensor company, launched Kosmo at its Kosmo AI Day event on Oct. 14 to address this gap by turning physical environments into reusable 3D digital assets.

Kosmo combines lidar, imaging and motion sensors to capture real spaces, then uses AI to complete missing geometry and label objects within the resulting 3D scenes. The system adapts these scenes for robot simulation and can generate over 10,000 training trajectories from a single capture session.

Demonstrated Efficiency Gains

Hesai showcased practical applications across robotics, visual effects, cultural tourism and spatial commerce. In a robotics demonstration conducted in a Dairy Queen store with partner Sharpa, a 15-minute Kosmo scan replaced approximately 300 hours of manual 3D modeling. Sharpa's robot subsequently achieved a 99% task-success rate when operating in the digitally reconstructed environment.

In the visual effects domain, Pixar's RenderMan tutorial incorporated a Kosmo scan of its campus plaza containing 1.25 million splats, which was compatible with both Houdini and RenderMan pipelines.

Ecosystem Integration and Partnerships

Kosmo supports open access to raw capture data and 3D assets, with integrations spanning NVIDIA Isaac Sim, Isaac Lab, Unreal Engine, RenderMan and Houdini. Hesai has also partnered with Chinese companies including mapping service Amap, whose Flying Street View feature now includes Kosmo-generated 3D reconstructions of venues such as the Gravity Art Museum and PIFO Gallery.

Additional partnerships with Galbot and Dexmal target embodied AI advancement, while Hesai offers hardware-and-software bundles alongside spatial-data services for commercial users.