Vision-Force ACT: Multimodal Action Chunking for Robotic Bimanual Peg-in-Hole Assembly

Abstract

Robotic bimanual peg-in-hole assembly requires accurate geometric alignment and compliant manipulation. Vision-based imitation learning is widely used in such kind of bimanual manipulation, which can efficiently locate and approach the target. However, visual observations could be ambiguous when the peg, hole or contact surface is occluded by the manipulators. This paper presents Vision-Force ACT, a multimodal extension of Action Chunking with Transformers (ACT) that incorporates bilateral wrist force/torque observations into both the conditional latent inference and the policy representation. The proposed method successfully outputs temporally coherent bimanual action chunks from synchronized images, joints state, and wrist F/T signals. A Mujoco-based simulated bimanual assembly platform equipped with 6-axis force/torque (F/T) sensor at both wrists is used in the experiment, and the proposed method is compared with the vision-only ACT baseline under identical training settings. The results show that Vision-Force ACT improves the task success rate from 40% to 68% with 50 demonstrations and from 60% to 78% with 200 demonstrations. The recorded F/T curves are also smoother than those of the vision-only policy and show closer agreement with the force demonstrations. These preliminary results support force sensing as an effective complement to visual observations for contact-rich bimanual imitation learning, while also motivating broader multi-task and real-robot validation.

Mingqi Chen

Postdoctoral Researcher

Qiang Li

Professor and header of AG

My research interests include underwater robot, collaborative robots, humanoid robots.