aravindsuresh16/smolvla_cathlab_fr3_instruments_v2_5

SmolVLA on simulated FR3 Duo "Hand me the … with the left/right arm" (v2.5)

SmolVLA fine-tuned on aravindsuresh16/cathlab_fr3_instruments_v2_5: 640 scripted-expert demonstrations of a dual-arm Franka FR3 (Robotiq 2F-85 grippers). The arm on the instrument's side picks up scissors, a trocar or a puncture device and places it on a central tray. The scene is simulated and rendered in NVIDIA Newton (MuJoCo-Warp). Every instruction names the acting arm.

Code (simulation, data generation, evaluation) and the full analysis: aravinds1610/mujoco_cathlab_vla, see docs/V2_RESULTS.md.

Status: research checkpoint, rarely completes the task

Closed loop in Newton, 48 episodes: 4 test sets (in-distribution, held-out layout, held-out table colour, both) × 12.

Test	Result
Correct arm chosen	48 / 48 (the same model trained without the arm in the instruction: 22 / 48)
Instrument placed on the tray	2 / 48 (both on held-out layouts)
Closest approach to the instrument (median)	9.1 cm
Started 6–18 cm above the instrument	0 / 48 placed

Main failure: the reach. The model closes the gripper at about the time the expert does (frame ~156 vs the expert's ~145), but a median 9 cm from the instrument. It has learned the motion pattern of a pick, not to aim it at the instrument it sees. Success varies between runs because actions are sampled from noise.

Inputs and outputs

• observation.images.{external_camera, left_wrist_camera, right_wrist_camera}: 224×224 RGB (mapped to camera1..3 by the saved preprocessor)
• observation.state: float32[16] = left joints 1–7, left gripper, right joints 1–7, right gripper (rad; gripper 1 = open)
• task: instruction text naming the arm, e.g. "Hand me the trocar with the left arm". The arm is the one on the instrument's side of the table
• action: chunk of 50 × 16 absolute joint targets at 30 Hz. The evaluation ran 25 per call and cross-faded 10 (blend=10 in cathlab_sim/smolvla_policy.py)

from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
policy = SmolVLAPolicy.from_pretrained("aravindsuresh16/smolvla_cathlab_fr3_instruments_v2_5")

Training

Base	lerobot/smolvla_base (vision encoder frozen, action expert trained)
Data	aravindsuresh16/cathlab_fr3_instruments_v2_5 (640 episodes, 230,558 frames)
Steps / batch	19,200 / 12 (1 epoch)
Hardware / time	RTX 4060 Laptop 8 GB, 3 h 36 min, 5.8 GB peak
Loss	1.44 → 0.010
LeRobot	0.6.1

Licence

CC BY-NC 4.0 (non-commercial, attribution), because the training data renders NVIDIA Isaac for Healthcare / Lightwheel instrument models under that licence. The base model, SmolVLA, is Apache-2.0.