How does robot programming work today, and what is the problem?
Most industrial robots today are programmed in two ways: either offline on a computer using 3D models and simulations—which takes time and requires a lot of expertise—or online, for example with a teach pendant (a type of tablet with a joystick). This involves manually moving the robot, which is more direct but tedious and doesn’t allow for pre-testing.
Our mixed reality approach combines the best of both worlds: With goggles like the Meta Quest 3, users can see both the real robot and a virtual twin at the same time. First, the virtual robot is programmed, its movements are tested in a simulation, and possible collisions are checked. If everything works smoothly, the real robot takes over the task exactly as planned.
What did you research specifically?
We compared three different controller-based programming methods:
- Classical jogging: The robot is moved using controller sticks and buttons, similar to the classic teach pendant.
- Direct control: The controller is moved directly, and the virtual robot arm follows these movements (6-DoF) in real time.
- Gripper control: Based on direct control, this method adds a 3D-printed gripper to the controller that is identical to the robot gripper. Physical objects can be gripped and moved directly.
Thirty test subjects performed two tasks: a simple pick-and-place operation and a more complex swapping task involving two objects.
What were the most important results?
The Gripper Control method was clearly the best: participants took 68% less time than with the classic approach, made 65% fewer programming errors, and rated this method as the most natural and precise. Physical interaction with real objects during programming made all the difference.
What is the practical significance of this research?
This is a breakthrough for small and medium-sized companies without expertise in complex robot programming. With our approach, a person without programming experience can set up a robot for repetitive tasks in just a few minutes. The combination of intuitive operation, instant simulation, and precise execution finally makes robotics accessible to everyone—not just experts.
Publication
Optimising Robot Programming: Mixed Reality Gripper Control
Maximilian Rettinger, Leander Hacker, Philipp Wolters, Gerhard Rigoll
Technical University of Munich, ICRA 2025
Video: https://www.youtube.com/watch?v=83kWr8zUFIQ
Interview: Andreas Schmitz