Researchers developed a robot system that actively explores objects to learn their properties
The ROMA system uses large language models to decide which physical actions, like touching or lifting, are needed to understand an object.
Humans naturally explore the world by touching and shaking items when they cannot tell what something is just by looking. A new system called ROMA brings this active process to robots by integrating vision, audio, touch, and force sensing.
Researchers designed ROMA as a reasoning loop that uses large language models to figure out what information is missing. The system decides which physical interaction, such as pushing or lifting an object, will best provide the needed sensory feedback. It then executes the action and updates its understanding based on what it senses.
ROMA integrates vision, audio, tactile, and force sensing into a reasoning-interaction-feedback loop.
To make this possible, the team created a dataset named ROMI-2K. This dataset includes nearly 2,000 real-world objects and covers 6 basic physical interactions, each paired with synchronized sensor data. This allowed the team to train the model to assess when it has enough evidence to form a complete understanding.
Imagine you place a mysterious container in front of a robot. Instead of just staring at it, the robot might decide to tilt it to hear if something rattles inside or press against it to gauge its firmness. This chain of perception helps the robot solve tasks that are too complex for simple visual observation alone.
While the system handles complex, long-horizon tasks well, it currently relies on specific physical interfaces for its interactions. The researchers have not yet tested how ROMA performs when tasked with extremely fragile or delicate items that require precise human-like force control.
Source: arXiv cs.RO