“This is a more intuitive way to command the machine,” says Hermano Krebs, principal research scientist in MIT’s Department of Mechanical Engineering. “With this new interface, we can eliminate a lot of the mental maps that an operator would need to build in order to operate an excavator.”
Krebs sees the interface as a faster way to train excavator operators, as well as a new way to physically operate the machines, both on-site and remotely.
“Instead of having joysticks, you might have this miniature arm on the side, where the operator would place their own arm, kind of like an exoskeleton, which would allow them to operate the excavator in the cab,” Krebs says. “If work has to be done in a difficult or unsafe environment, you could have an operator sitting off-site in a trailer and using this arm to remotely tele-operate the excavator.”
The team reports its open-access results this week in the Journal of Computing and Civil Engineering. MIT co-authors include Moises Alencastre-Miranda, Joao Buzzatto, and Eran Beeri Bamani, along with collaborators from Sumitomo Heavy Industries, an industrial machinery manufacturer based in Japan.
A machine mimic
At MIT, Krebs’ group works on human-robot interactions, with a longtime focus on physical rehabilitation. Through this work, the team has accumulated knowledge about the ways in which humans control their limbs and how they can most intuitively interact with machines.
In 2018, Krebs struck up a collaboration with researchers at Sumitomo Heavy Industries, who were looking for a faster way to train excavator operators. They noted that in Japan, the population of heavy machinery operators is aging rapidly; training their replacements takes time.
Operators typically learn by driving actual excavators on a controlled driving course. As they operate the machine, novices must learn to relate the actions of the excavator’s joysticks with the movements of the arm, bucket, and cab. Coordinating these actions to carry out actual tasks adds another level of complexity that can take months to years to master.
The team reasoned that if they could eliminate the need for this mental map, they might significantly shorten the training process. To do so, they looked for a more natural way to control the machine, as an alternative to the traditional joysticks. They soon landed on the mechanical arm design, reasoning that the physical resemblance to the digger’s own arm and bucket could enable operators to mimic and control the excavator’s movements directly, without much mental translation.
Over the next few years, the researchers worked to build the mechanical arm, along with the software to pair its movements with a virtual simulation of an excavator. The combination of the mechanical arm and the virtual simulator constitutes a new training and control platform for digger operators, which the team has named the “World-Space Interface.”
“‘World-space’ refers to everything in the world that is outside of yourself, or in this case, outside of the excavator’s cab,” Krebs explains. “Normally, operators have to build a mental map of how to manipulate things in the world-space. But now, we can just mime picking up rocks or dirt, and the computer will do that translation to the world-space for us.”
Construction on day one
For their new study, the team ran training experiments with volunteers who used the World-Space Interface (WSI) as well as a more traditional, joystick-based excavator simulator. The researchers developed virtual simulations of 15 realistic excavation environments, including construction sites, highways, forest roads, riverbanks, mining areas, and urban and rural settings. Each virtual environment was associated with various excavation tasks, such as scooping and dumping sand or gravel, digging and grading trenches, clearing debris from roads, removing tree branches from water edges, and breaking up rocks.
The team designed the experiment to resemble the tasks that an operator typically performs during a weeklong excavator driving course. For one hour each day for seven days, volunteers — both expert and novice — operated the WSI and the joystick counterpart, training on tasks with increasing difficulty.
The researchers then compared the volunteers’ performance before and after the training period. For the joystick simulator, they found that novices were consistently worse than experts, though they did improve over the training period. In comparison, the team found that with the new World-Space Interface, novices were just as good as experts from the start.
“In this case, joysticks are a non-intuitive way to control and coordinate the machine,” says study co-author and MIT postdoc Joao Buzzatto. “This is the first interface that does not require me to command the excavator with joysticks.”
The team is now working to add haptics, or feeling to the WSI’s physical arm. The idea is that, as an operator uses the arm to mime an action such as picking up a pile of rocks, the arm will generate a force in response, as if the operator can feel the heaviness of the rocks, as confirmation that the excavator is indeed picking them up.
“Haptics would make this an even more intuitive system,” says co-author and visiting engineer Solmon Jeong.
The team says the new training interface can be a more natural alternative to excavator simulators that the construction industry is currently exploring. Companies such as Caterpillar, Hyundai, and Komatsu are developing virtual simulators, both to help train operators before they go on-site, and to one day remotely control excavators from a distance. However, these simulators are largely based on traditional joystick controllers that still take time to learn.
If the team’s new arm-and-bucket controller were incorporated, as an appendage in an excavator cab, or in a virtual, teleoperational simulator, the researchers envision that even first-time operators could get to work, from day one.
This research was supported, in part, by Sumitomo Heavy Industries.
