Based on first-person (Ego) video, EgoTwin accurately reconstructs 3D hand keypoints and hand mesh models, recovering the motion details and spatial relationships of real interactions. When aligning Ego data to the robot state space, EgoTwin achieves a significant improvement in data scaling efficiency compared with real-robot teleoperation and UMI approaches.
The large-scale development of embodied intelligence currently faces significant data bottlenecks: scarce real hand interaction data, difficult cross-embodiment action alignment, high real-robot debugging costs, and low iteration efficiency. These constraints have long limited the training optimization and real-world deployment of embodied intelligence.
Through technical collaboration with Baidu AI Cloud, EgoTwin has achieved a key breakthrough in 3D hand alignment, greatly improving the efficiency of scaling physical interaction data and providing a solid data foundation for the commercial deployment of Physical AI.