Correcting Persistent Failures in Point-Cloud Grasping Policies using Human Demonstrations in Virtual Reality
Abstract
A human-in-the-loop failure-mining pipeline turns persistently unsolved grasps of a reinforcement-learning point-cloud policy into virtual-reality human demonstrations, improving hard-case success without reducing generalization to unseen objects.
Robots in homes, warehouses and farms must pick up objects they have never seen, usually without a 3D model or a category label. A grasping policy that works directly on the point cloud from a depth sensor needs only the geometry the robot observes, so it applies to any new object. Offline grasp generators such as Contact-GraspNet work this way, but they rank grasps by learned plausibility rather than by whether the grasp actually lifts the object. Reinforcement learning (RL) can optimize for that directly and make a point-cloud grasping policy much better on average, yet it leaves specific grasps persistently unsolved. We present a human-in-the-loop failure loop that targets those grasps: situations an RL policy keeps failing are mined automatically, sent to a virtual-reality application where a person picks the object up with their own hands, and returned to training as two grasp points. The person contributes only where to grasp; the robot chooses its own approach. In simulation, RL raises success on unseen objects from 55% with Contact-GraspNet to 83%. Adding 58 human demonstrations nearly doubles success on the hard cases RL kept failing, from 24% to 45%, and recovers performance that RL alone loses on similar cases that were never demonstrated, without reducing success on unseen objects.
Community
We found that RL makes a point-cloud grasping policy much better on average, but can leave particular object–pose combinations persistently unsolved. So we built a failure loop: mine those cases, show them to a person in VR, then use their two fingertip contacts to retrain the policy. The robot chooses its own approach, and each demonstration is checked with its gripper before training.
In simulation, RL improved lift success on unseen flat-lying objects from 55% with Contact-GraspNet to 83%. Adding 58 verified demonstrations collected in 32 minutes raised success on demonstrated hard cases from 24% to 45%, without a measurable loss on held-out objects. We also saw recovery on undemonstrated cases involving similar objects. The next step is to separate the effects of human supervision and targeted replay, then test on hardware.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Steerable and Reactive Grasping Through Modular Design with a Three-Point Interface (2026)
- HuG: Learning Humanoid Whole-Body Affordance-Aware Grasping with Active Perception (2026)
- Grasp-Taxonomy Conditioning as a Mode Switch for Dexterous Grasping of Climbing Holds (2026)
- Learning Grasp Targeting from Point Clouds for Log Pile Clearing on a Hydraulic Crane (2026)
- Dexterous In-Hand Object Pose Reaching with Learned Safety Filters (2026)
- GraspTune: Tactile-Driven Execution Refinement for Robust Grasping (2026)
- WorldSketch: Sketching World Models for Zero-Shot Mobile Manipulation (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.18919 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper