The researchers now use pretrained generative AI to teach robots how to follow rules, or “policy”, and can perform multiple complex tasks.
The models learn the correct movement path or trajectory for robots by being exposed to only feasible robot actions. These trajectories are not always suited to the needs of users in actual situations.
To fix these problems, it is often necessary to collect new data and train the model again. This can be expensive, time-consuming, and require advanced machine learning skills.
Imagine being able to correct the action of a robot in real-time with a single interaction.
The new technology is a perfect mimic of nature, allowing robots to move with ease.
Researchers at MIT and NVIDIA have developed a framework that could make this scenario a real possibility. The new framework developed by MIT and NVIDIA researchers could help make robots easier to use, more user-friendly. It would allow people to change a robot’s behavior in real time with simple interaction.
The robot can be trained to respond intuitively and in real-time by the human. It allows robots to react to real-time, intuitive human feedback and select an action that is closely in line with user intent.
A framework with a 21-percent higher success rate than an alternative testing method.
Felix Yanwei Wang is a graduate student in electrical engineering and computers science and the lead author on a recent paper. We can’t expect untrained people to collect data and refine a model of a neural net. Consumers will want the robot to be ready to use right from the start. If it isn’t, then they need an easy way to customise it. This is what we set out to do in our work.”
We want the user to be able to interact with robots without making these kinds of errors, and we also need a behavior which is valid and possible.
By using simple interaction, MIT researchers have developed an interactive system that allows anyone to correct a robot’s behavior. As Felix Yanwei Wang, graduate student, demonstrated, users can direct a robot by pointing at an object displayed on the screen.
Credit: Melanie Gonick, MIT
Users can guide robot actions in three simple ways. Users can either point at the object they want to see in the robot camera, follow a route, or move the arm of the robot. The most accurate method is to physically move the robot, since it prevents information loss when translating a 2D picture into a 3D motion.
Scientists developed a technique called sampling to stop the robot from performing invalid movements, such as colliding into objects. The robot can choose the most appropriate action out of a list that includes all valid actions. This is in line with what the user wants.
In order to avoid imposing instructions directly, the robot uses the feedback from the user in conjunction with the learned behavior of the robot. The robot can adapt to the user’s feedback while still staying safe.
This approach was found to be superior in tests conducted in simulations, and even with real robot arms in a kitchen. It may not complete the task instantly, but it lets users correct errors in real time, without having to wait until it is completed to give new instructions.
The researchers are now looking forward to optimising the sampling process while maintaining or increasing its performance.
Journal Reference
- Yanwei Wang, Lirui Wang, Yilun Du, Balakumar Sundaralingam, Xuning Yang, Yu-Wei Chao, Claudia Perez-D’Arpino, Dieter Fox, Julie Shah. Human Interactions for Inference-Time Steering of Policy. arXiv: 2411.16627v1
