The Chief Scientist of Reality Labs Describes a New Compute Architecture for True AR Glasses | Technology

The Chief Scientist of Reality Labs Describes a New Compute Architecture for True AR Glasses | Technology
The Chief Scientist of Reality Labs Describes a New Compute Architecture for True AR Glasses 

Reality Labs Chief Scientist Outlines a New Compute Architecture for True AR Glasses

Speaking at the IEDM conference late last year, Meta Reality Labs chief scientist Michael Abrash described the company's analysis of how contemporary compute architectures will need to evolve to make the AR glasses of our sci-fi concepts possible.

While there are a few AR 'glasses' on the market today, none of them are actually the size of a normal pair of glasses (even a bulky pair).  The best AR headsets available today—the likes of the HoloLens 2 and Magic Leap 2—are still closer to goggles than glasses and are too heavy to be worn all day (not to mention the look you'll get from the crowd).

READ MORE: The Top Priorities for Artificial Intelligence in Banking in 2022 | Technology

If we're going to build AR glasses that are actually shaped like glasses, with all-day battery life and the features needed to compelling AR experiences, it'll need a series of radical improvements—and  Paradigm shifts in some cases—both in hardware [...] and software," says Michael Abrash, chief scientist at Reality Labs, Meta's XR organization.

That is to say: Meta doesn't believe that its current technology—or anyone's for that matter—is capable of delivering the sci-fi specs that every AR concept video imaginable.

But, the company thinks it knows where things need to go for this to happen.

Speaking at the IEDM 2021 conference late last year, Abrash laid out the case for a new compute architecture that could actually meet the needs of glasses-sized AR devices.

READ MORE: Why Hybrid Intelligence is the Future of Artificial Intelligence at McKinsey | Technology

Follow the Power

The main reason for rethinking how computing should be handled on these devices is the need to reduce power consumption enough to meet battery life and heat requirements.

"How can we radically improve the power efficiency [of mobile computing devices] by a factor of 100 or 1,000?"  he asks.  “This will require a thorough system-level rethinking of the full stack, along with end-to-end co-design of hardware and software. And the place to start that rethink is to look at where the power is going today.” 

To this end, Abrash generated a graph comparing the power consumption of low-level computing operations.

READ MORE: Artificial Intelligence and the Origin of Digital Twins | Technology

As the chart highlights, the most energy intensive computing operations are in data transfer.  And it doesn't just mean wireless data transfer, but also data transfer from one chip to another inside the device.  In addition, the chart uses a logarithmic scale;  According to the chart, moving the data to RAM uses 12,000 times the power of the base unit (which in this case is adding two numbers together).

Bringing all of these together, the circular graphs on the right show that the technologies needed for AR—SLAM and hand-tracking—use most of their power only to move data to and from RAM.

“Clearly, for low power applications [such as in lightweight AR glasses], it is important to reduce the amount of data transfer as much as possible,” Abrash says.

To do this, he says, would require a new compute architecture – that instead of shuffling large amounts of data between centralized computing hubs – to reduce wasteful data transfers computing operations would be more widespread throughout the system.  evenly distributes.

READ MORE: How fintech firms are utilising AI and machine learning to generate alternative loan scores | Technology

Calculate where you least expect it

A starting point for a distributed computing architecture, Abrash says, could start with the many cameras that AR glasses require to understand the world around the user.  This would involve doing some preliminary calculations on the camera sensor itself before sending only the most important data to the power-hungry data transfer lane.

To make this possible, Abrash says it will take co-designed hardware and software, such as hardware designed with a specific algorithm in mind, which is essentially the hardware embedded in the camera sensor itself – any  Allows to take care of some operations before data.  Even the sensor drops.

"The combination of requirements for lowest power, best requirements, and smallest possible form-factor make the XR sensor the new frontier in the image sensor industry," says Abrash.

READ MORE: Deep Learning: An in-depth look at AI-powered Technology | Technology

Domain Specific Sensor

He also revealed that Reality Labs has already started working in this direction, and has even built a prototype camera sensor that is specifically designed for the low power, high performance requirements of AR glasses.  has been done. 

The sensor uses an array of so-called digital pixel sensors that simultaneously capture digital light values ​​at three different light levels at each pixel.  Each pixel has its own memory to store the data, and can decide which of the three values ​​to report (instead of sending all the data to another chip to do that job).

This not only reduces power, Abrash says, but also significantly increases the sensor's dynamic range (the ability to capture dim and bright light levels in the same image).  He shared a sample image captured with the company's prototype sensor compared to a typical sensor to demonstrate a wider dynamic range.

READ MORE: InWorld introduces impressive AI-based character generation and interaction | Technology

In the image on the left, the bright bulb washes out the image, leaving the camera not able to capture much of the scene.  On the other hand, the image on the right can see not only the extreme brightness of the lightbulb's filament in detail, but it can also see other parts of the scene.

This wide dynamic range is essential for the sensor for future AR glasses, which will need to function well in low-light indoor conditions such as sunny days.

Even with the HDR benefits of Meta's prototype sensor, Abrash says it's significantly more power efficient at 30 fps using just 5mW (less than 25% of what he claims is a typical sensor).  And it scales well too;  While it will take more power, he says the sensor can capture up to 480 frames per second.

READ MORE: A Virtual Reality Business is testing a chronic pain therapy on a simulated patient group | Technology

But, Meta wants to go even further, with even more complex calculations taking place on the sensor itself.

"For example, a shallow part of deep neural networks—segmentation and classification for XR workloads such as eye-tracking and hand-tracking—can be applied to sensors."

But that may not happen, Abrash says, before more hardware innovation, such as the development of ultra-dense, low-power memory that will be needed for "true on-sensor ML computing."

While the company is experimenting with these technologies, Abrash notes that the industry at large needs to come together to do it at scale.  In particular they state "the development of MRAM technologies by [chip makers] is an important element for developing AR glasses."

READ MORE: Researchers create strange magnetic particles with lasers-quantum can revolutionize computers | Technology

"Combined together in an end-to-end system, our proposed distributed architecture, and the related technology I describe, has the potential for vast improvements in power, area, and form-factor," Abrash said.  "Improvements that are needed to become comfortable and functional enough to be part of the daily lives of a billion people."

Source: Ben Lang, ROADTOVR, Direct News 99