HandEdit: an open-source dataset that turns first-person human video into robot dexterous-hand training data

by

HandEdit, a dataset and benchmark built by Inspire-Robots together with Shanghai Jiao Tong University and Fudan University, has been open-sourced. The project tackles first-person human-to-robot dexterous-hand image editing: replacing the human hand or arm in a frame with a specified robot configuration while preserving the original scene, object, viewpoint and interaction as much as possible. Around that task it supplies paired data, a unified evaluation protocol and open tools for training and comparison.

This is not an ordinary “hand swap” visual effect. The edit must fully remove the original hand while keeping object state, contact relation and background unchanged, and the generated robot must match the target URDF’s structure, material and joint form.

Comparison of human first-person hand manipulation and the corresponding robot dexterous-hand render from the HandEdit dataset
HandEdit pairs human first-person manipulation with a corresponding robot dexterous-hand render. (Source: OFweek Robotics)

Data sources and composition

Robot dexterous-operation data is usually collected through teleoperation, motion capture or real robot execution, and is tightly bound to a specific robot body, making acquisition and scaling costly. First-person human videos record grasping, opening, twisting, cutting and two-hand collaboration with rich object and scene information, but human hands differ from robot dexterous hands in appearance, joint structure, scale and motion, so the videos cannot map directly to a specified robot configuration.

HandEdit assembles these resources:

  • More than 200 million image-edit samples.
  • Over 300,000 video clips.
  • Five first-person operation datasets aggregated: EgoDex, ARCTIC, OakInk2, HOI4D and HO-Cap.
  • 26 URDF configurations covered, including 13 standalone dexterous hands and 13 hand-arm integrated configurations.
  • More than 600 scenes, 1,100-plus objects and 400-plus task types.

The 13 standalone hands span multiple brands and structures, including Inspire-Robots’ own RH56DFX and RH5DG2 series. the hand-arm configurations combine manipulator platforms such as JAKA, KUKA, Panda, RM65/75, UR5 and xArm.

How the data is generated

To produce reference images for target robot configurations, HandEdit built a pipeline combining vision processing, geometric retargeting and simulation rendering:

  1. Segmentation: SAM3 locates the hand or arm region, covering fingers, palm, wrist and visible forearm.
  2. Background repair: ProPainter restores the occluded region with cross-frame consistency.
  3. Motion retargeting: MANO or 3D hand pose is mapped into the target URDF joint space, refined against each body’s kinematic chain.
  4. Arm solving: Hand-Arm data first retargets the hand, then solves arm joints via inverse kinematics.
  5. Same-view render and composite: render the robot foreground under the matched first-person camera and composite back into the repaired scene.
  6. Quality screening: drop samples with foreground residue, broken background, bad pose or severe clipping.

How edits are evaluated

Evaluation checks not just visual quality but whether the hand was fully removed, the target robot structure and identity are consistent, and object interaction is preserved. HandEdit sets two tracks, Hand-only and Hand-Arm, each with 1,000 images and 13 target configurations. Three metric layers apply: general similarity (PSNR, SSIM, LPIPS, FID). vision-language-model judgement of semantic consistency and perceptual quality. and embodied-task metrics for hand removal, structure consistency, robot identity and interaction preservation.

The benchmark tested 11 representative open-source and commercial image-editing methods, including GPT-Image, Nano Banana, FLUX, Seedream, Qwen-Image-Edit, HunyuanImage, FireRed-Image-Edit and OmniGen2. Three findings stand out: GPT-Image-2 is balanced across general similarity and embodied-task metrics. a high VLM score does not guarantee correct interaction (GPT-Image-1.5). and a natural picture does not mean task completion (FireRed-Image-Edit-1.1). That is why HandEdit keeps all three metric classes.

The HandEdit dataset and benchmark are open to global researchers and developers. Project page: https://handedit.github.io/ . Paper: https://arxiv.org/abs/2608.12122 . Code: https://github.com/HandEdit/HandEdit . Data: https://huggingface.co/datasets/HandEdit/HandEdit .

Editor’s note: This is an adapted translation of the original OFweek Robotics report. It has been trimmed and restructured for readability for an international business audience. The full original (in Chinese) is at https://robot.ofweek.com/2026-09/ART-898890-8120-30701741.html.

Translated and adapted from OFweek Robotics (https://robot.ofweek.com/2026-09/ART-898890-8120-30701741.html).

Leave a comment