Researchers in the emerging field of spatial computing have developed a prototype augmented reality headset that uses holographic imaging to overlay full-color, 3D moving images on the lenses of what would appear to be an ordinary pair of glasses. Unlike the bulky headsets of present-day augmented reality systems, the new approach delivers a visually satisfying 3D viewing experience in a compact, comfortable, and attractive form factor suitable for all-day wear.“Our headset appears to the outside world just like an everyday pair of glasses, but what the wearer sees through the lenses is an enriched world overlaid with vibrant, full-color 3D computed imagery,” said <a href="https://web.stanford.edu/~gordonwz/" data-extlink="" target="_blank">Gordon Wetzstein</a>, an associate professor of electrical engineering and an expert in the fast-emerging field of spatial computing. Wetzstein and a team of engineers introduce their device in a new paper in the journal <a href="https://www.nature.com/articles/s41586-024-07386-0" data-extlink="" target="_blank">Nature</a>. Additional information about this advance is available at <a href="https://www.computationalimaging.org/publications/holographicAR/" data-extlink="" target="_blank">this website</a>, created by the research team.Though only a prototype now, such a technology, they say, could transform fields stretching from gaming and entertainment to training and education – anywhere computed imagery might enhance or inform the wearer’s understanding of the world around them.“One could imagine a surgeon wearing such glasses to plan a delicate or complex surgery or airplane mechanic using them to learn to work on the latest jet engine,” <a href="https://manugopa.github.io/" data-extlink="" target="_blank">Manu Gopakumar</a>, a doctoral student in the Wetzstein-led <a href="https://www.computationalimaging.org/" data-extlink="" target="_blank">Stanford Computational Imaging lab</a> and co-first author of the paper said.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1715819863917-2_20240430_holographic_ar_displays_n6a0965_0.png.webp" style="width: 100%px;" class="fr-fic fr-dib">New holographic augmented reality system that enables more compact 3D displays | Image courtesy of Andrew Brodhead<h2 id="isPasted">Barriers overcome</h2>The new approach is the first to thread a complex maze of engineering requirements that have so far produced either ungainly headsets or less-than-satisfying 3D visual experiences that can leave the wearer visually fatigued, or even a bit nauseous at times.“There is no other augmented reality system out there now with comparable compact form factor or that matches our 3D image quality,” said <a href="https://gunyeal.github.io/" data-extlink="" target="_blank">Gun-Yeal Lee</a>, a postdoctoral researcher in the Stanford Computational Imaging lab and co-first author of the paper.To succeed, the researchers have overcome technical barriers through a combination of AI-enhanced holographic imaging and new nanophotonic device approaches. The first hurdle was that the techniques for displaying augmented reality imagery often require the use of complex optical systems. In these systems, the user does not actually see the real world through the lenses of the headset. Instead, cameras mounted on the exterior of the headset capture the world in real time and combine that imagery with computed imagery. The resulting blended image is then projected to the user’s eye stereoscopically.“The user sees a digitized approximation of the real world with computed imagery overlaid. It’s sort of augmented virtual reality, not true augmented reality,” explained Lee.These systems, Wetzstein explains, are necessarily bulky because they use magnifying lenses between the wearer’s eye and the projection screens that require a minimum distance between the eye, the lenses, and the screens, leading to additional size.“Beyond bulkiness, these limitations can also lead to unsatisfactory perceptual realism and, often, visual discomfort,” said <a href="https://choisuyeon.github.io/" data-extlink="" target="_blank">Suyeon Choi</a>, a doctoral student in the Stanford Computational Imaging lab and co-author of the paper.<h2>Killer app</h2>To produce more visually satisfying 3D images, Wetzstein leapfrogged traditional stereoscopic approaches in favor of holography, a Nobel-winning visual technique developed in the late-1940s. Despite great promise in 3D imaging, more widespread adoption of holography has been limited by an inability to portray accurate 3D depth cues, leading to an underwhelming, sometimes nausea-inducing, visual experience.The Wetzstein team used AI to improve the depth cues in the holographic images. Then, using advances in nanophotonics and waveguide display technologies, the researchers were able to project computed holograms onto the lenses of the glasses without relying on bulky additional optics.A waveguide is constructed by etching nanometer-scale patterns onto the lens surface. Small holographic displays mounted at each temple project the computed imagery through the etched patterns which bounce the light within the lens before it is delivered directly to the viewer’s eye. Looking through the glasses’ lenses, the user sees both the real world and the full-color, 3D computed images displayed on top.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1715819889720-3_20240430_holographic_ar_displays_n6a1169_0.png.webp" style="width: 100%px;" class="fr-fic fr-dib">Using nanophotonic technologies called metasurface optics, the researchers designed and fabricated a novel waveguide design that can relay 3D hologram information of RGB visible light into a single compact device with high transparency. These nanophotonic waveguide samples were fabricated in-house at Stanford Nanofabrication Facility and Stanford Nano Shared Facilities. | Image courtesy of Andrew Brodhead<h2 id="isPasted">Life-like quality</h2>The 3D effect is enhanced because it is created both stereoscopically, in the sense that each eye gets to see a slightly different image as they would in traditional 3D imaging, and holographically.“With holography, you also get the full 3D volume in front of each eye increasing the life-like 3D image quality,” said <a href="https://bchao1.github.io/" data-extlink="" target="_blank">Brian Chao</a>, a doctoral student in the Stanford Computational Imaging lab and also co-author of the paper.The ultimate outcome of the new waveguide display techniques and the improvement in holographic imaging is a true-to-life 3D visual experience that is both visually satisfying to the user without the fatigue that has challenged earlier approaches.“Holographic displays have long been considered the ultimate 3D technique, but it’s never quite achieved that big commercial breakthrough,” Wetzstein said. “Maybe now they have the killer app they’ve been waiting for all these years.”Additional authors are from The University of Hong Kong and NVIDIA. Wetzstein is also member of&nbsp;<a href="http://biox.stanford.edu/" data-extlink="" target="_blank">Stanford Bio-X</a>, the&nbsp;<a href="https://humanperformance.stanford.edu/" data-extlink="" target="_blank">Wu Tsai Human Performance Alliance</a>, and the&nbsp;<a href="https://neuroinstitute.stanford.edu/" data-extlink="" target="_blank">Wu Tsai Neurosciences Institute</a>.

Prototype of the compact augmented reality glasses. Through holography and AI, these glasses can display full-color, 3D moving images over an otherwise direct view of the real world. | Image courtesy of Andrew Brodhead

Combining advances in display technologies, holographic imaging, and artificial intelligence, engineers at Stanford say they have produced a leap forward for augmented reality.

AI and holography bring 3D augmented reality to regular glasses

Holographic images have real depth because they are three dimensional, whereas monitors merely simulate depth on a 2D screen. Because we see in three dimensions, holographic images could be integrated seamlessly into our normal view of the everyday world.The result is a virtual and augmented reality display that has the potential to be truly immersive, the kind where you can move your head normally and never lose the holographic images from view. “To get a similar experience using a monitor, you would need to sit right in front of a cinema screen,” said <a href="https://engineering.princeton.edu/faculty/felix-heide" target="_blank">Felix Heide</a>, assistant professor of <a href="http://cs.princeton.edu/" target="_blank">computer science</a> and senior author on <a href="https://www.nature.com/articles/s41467-024-46915-3" target="_blank">a paper</a> published April 22 in Nature Communications.And you wouldn’t need to wear a screen in front of your eyes to get this immersive experience. Optical elements required to create these images are tiny and could potentially fit on a regular pair of glasses. Virtual reality displays that use a monitor, as current displays do, require a full headset. And they tend to be bulky because they need to accommodate a screen and the hardware necessary to operate it.“Holography could make virtual and augmented reality displays easily usable, wearable, and ultrathin,” said Heide. They could transform how we interact with our environments, everything from getting directions while driving, to monitoring a patient during surgery, to accessing plumbing instructions while doing a home repair.One of the most important challenges is quality. Holographic images are created by a small chip-like device called a spatial light modulator. Until now, these modulators could only create images that are either small and clear or large and fuzzy. This tradeoff between image size and clarity results in a narrow field of view, too narrow to give the user an immersive experience. “If you look towards the corners of the display, the whole image may disappear,” said Nathan Matsuda, research scientist at Meta and co-author on the paper.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1713912832170-Neural_Etendue_Expander_3_v2-1-2048x1152.png" style="width: 100%px;" class="fr-fic fr-dib">At left, an image produced using current holographic display technology. At right, the same image created using the new optical element invented by researchers at Princeton and Meta.Heide, Matsuda and&nbsp;<a href="https://ethan-tseng.github.io/" target="_blank">Ethan Tseng</a>, doctoral student in computer science, have&nbsp;<a href="https://light.princeton.edu/publication/neural-etendue-expander/" target="_blank">created a device</a> to improve image quality and potentially solve this problem. Along with their collaborators, they built a second optical element to work in tandem with the spatial light modulator. Their device filters the light from the spatial light modulator to expand the field of view while preserving the stability and fidelity of the image. It creates a larger image with only a minimal drop in quality.Image quality has been a core challenge preventing the practical applications of holographic displays, said Matsuda. “The research brings us one step closer to resolving this challenge,” he said.The new optical element is like a very small custom-built piece of frosted glass, said Heide. The pattern etched into the frosted glass is the key. Designed using AI and optical techniques, the etched surface scatters light created by the spatial light modulator in a very precise way, pushing some elements of an image into frequency bands that are not easily perceived by the human eye. This improves the quality of the holographic image and expands the field of view.Still, hurdles to making a working holographic display remain. The image quality isn’t yet perfect, said Heide, and the fabrication process for the optical elements needs to be improved. “A lot of technology has to come together to make this feasible,” said Heide. “But this research shows a path forward.”The paper, “Neural Etendue Expander for Ultra-Wide-Angle High-Fidelity Holographic Display” was published April 22 in Nature Communications. In addition to Heide and Tseng, co-authors from Princeton include Seung-Hwan Baek and Praneeth Chakravarthula. In addition to Matsuda, co-authors from Meta Research are Grace Kuo, Andrew Maimone, Florian Schiffers, and Douglas Lanman. Qiang Fu and Wolfgang Heidrich from the Visual Computing Center at King Abdullah University of Science and Technology in Saudi Arabia also contributed. The work was supported by Princeton University’s&nbsp;<a href="https://iac.princeton.edu/home.html" target="_blank">Imaging and Analysis Center</a> and the King Abdullah University of Science and Technology’s Nanofabrication Core Lab.

Researchers at Princeton and Meta have created a tiny optical device that makes holographic images larger and clearer. Small enough to fit on a pair of eyeglasses, the device could enable a new kind of immersive virtual reality display. Illustration by Liz Sabol, photo by Nathan Matsuda

Setting the stage for a new era of immersive displays, researchers are one step closer to mixing the real and virtual worlds in an ordinary pair of eyeglasses using high-definition 3D holographic images, according to a study led by Princeton University researchers.

Holographic displays offer a glimpse into an immersive future

In this episode, we discuss research from Harvard that tackles the blurriness and lack of disruption recovery with most holograms which prevents users from having an immersive experience.