A new chip developed by MIT researchers could help tiny, low-power UAVs avoid obstacles as they zip around tight corners inside an industrial HVAC system to check for gas leaks.
The chip allows small autonomous robots and other battery-limited devices to construct detailed 3D maps of their environments in real-time using only about as much power as a single LED. A robot could use such a map to plan a collision-free path to reach its goal.
Typically, generating such thorough maps requires power-hungry systems and a great deal of memory to build and store 3D representations of the obstacles in a robot’s environment.
The MIT researchers took a different approach by combining an extremely efficient mapping algorithm with specialized hardware designed to accelerate its workload, which minimizes memory and power consumption.
This system-on-a-chip consumes only about 6 milliwatts of power, a fraction of the power required by other systems.
This low-power operation could also make the chip well-suited for lightweight augmented reality headsets that can be worn for extended periods, for applications like educational medical simulation or detailed repair and assembly work.
“This paper showcases a key example of how you can leverage co-design of the algorithm and hardware to really push energy efficiency. While there has been a lot of work looking into compact 3D maps, what stands out about this work is that it also ensures that the process to generate those maps is as efficient as possible. Our chip allows you to store very large maps in a very small space, and do it in a very energy efficient manner,” says Vivienne Sze, a professor in the Department of Electrical Engineering and Computer Science (EECS), a member of the Research Laboratory of Electronics (RLE), and senior author of a paper on the chip.
She is joined on the paper by co-lead authors and MIT graduate students Zih-Sing Fu and Peter Zhi Xuan Li as well as Sertac Karaman, a professor of aeronautics and astronautics and the director of LIDS. The work was recently presented at the IEEE Very Large-Scale Integrated Circuits Symposium.
A more compact map
For a robot, generating a 3D map that includes the obstacles in its environment usually demands a lot of power because it must store images captured by its camera, and process all the 3D pixels in each image multiple times.
Instead of representing the environment using 3D pixels, which are cubes called voxels, the MIT researchers utilized a technique that maps the obstacles in space using ellipsoid blobs called Gaussians.
The size, shape, and thickness of these ellipsoids can be smoothly adapted, so they match the shape of curved objects more efficiently than if one uses rigid, cube-shaped voxels.
Importantly, the map captures the obstacles and free space around the robot, and together these let the robot plan a safe, collision-free path. Mapping obstacles and free space with voxels typically consumes a lot of memory, which makes traditional methods power-hungry. Because Gaussians can flexibly fit the geometry, a single elongated ellipsoid can represent a region that would take many voxels, so occupied surfaces and free space are captured far more compactly.
For their new system-on-a-chip, called Gleanmer, the researchers employed an algorithm their lab developed called GMMap that efficiently generates a 3D map of the robot’s environment using Gaussians to represent obstacles.
With traditional approaches, a robot would need to load and process each depth image several times to adjust the size and shape of the ellipsoids. The system would usually construct Gaussians by comparing all the pixels in an image to each other. But the amount of memory and power needed to do this remains too high for many edge devices.
To solve this problem, the MIT researchers invented a technique that can generate highly accurate Gaussians from depth images with only one pass, after which they can discard the images, so the chip never has to store an entire image at once.
Instead of comparing each pixel to every other pixel in the 3D image, their algorithm assumes that nearby pixels belong in the same Gaussian, so it only needs to compare each pixel to its neighbors.
“At any point in time, we only need to store a few pixels in memory, which significantly reduces the memory footprint our algorithm requires,” Li says.
Leveraging co-design
But as the robot moves through the space, it usually sees the same object from different viewpoints. When it generates Gaussians, some will overlap because they represent the same object. This can make the 3D map too large to store on an edge device.
Fusing overlapping Gaussians makes the map more compact, but doing so typically requires the algorithm to process many raw pixels stored in memory. The researchers developed a novel technique to perform this fusion process directly on overlapping Gaussians, without needing to revisit the original pixels. Since Gaussians are more compact than pixels, this significantly reduces memory and power requirements.
The same principle runs through their algorithm — most computations operate directly on compact Gaussians rather than the original pixels, enabling energy efficiency.
The researchers exploit this principle to design a chip that keeps the Gaussians it is actively working on within small, fast on-chip memory right beside the computational units. This is only possible because the Gaussian map is so compact.
The Gaussians the robot needs to work on next are waiting in the on-chip memory units, so they don’t need to be fetched from more distant, power-hungry, off-chip storage.
“By having a dedicated memory that just stores the objects you’ve seen in the previous few frames, you can access the data much more efficiently,” Fu explains.
They tested the system-on-a-chip by reconstructing a range of diverse, pre-existing 3D environments. The chip can also reconstruct obstacles and free space directly from live data streamed from an iPhone camera.
Gleanmer generated detailed 3D maps in real-time while consuming about 6 milliwatts of power. It required only about 2.5 percent of the power that the best existing chip for map construction would need.
By reusing compact Gaussians along the path as it plans, the chip lets a robot chart a safe trajectory using only about 20 percent of the energy it would otherwise need.
“We reduce the memory consumption by making sure the algorithm is efficient. Then we accelerate the workload that is performed by that efficient algorithm, so in the end, our chip is as efficient as possible,” Li says.
The researchers plan to further improve energy efficiency by moving the processing units on the chip closer to the sensors that gather environmental data. They could also explore additional applications, such as the use of Gaussians to represent schematics. This could help AI systems reason about complex blueprints more efficiently.
“Real-time 3D mapping has been the missing piece for small autonomous systems. A drone inspecting a pipeline or a pair of AR glasses navigating a room both need to understand the space around them — instantly, continuously, and at almost no power cost. Gleanmer makes that possible for the first time in a chip you can hold between your fingers,” says Karaman.
This work is supported, in part, by the MIT-MathWorks Fellowship, Amazon, the U.S. National Science Foundation, and Intel.
