Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
AR/MR Combiners Part 2 – Hololens
Microsoft’s Hololens is perhaps the most most well known device using flat “waveguide” optics to “combine” the real world with computer graphics. Note there are no actual “holograms” anywhere in Hololens by the scientific definition.
At left is a picture from the Verge Teardown of a Hololens SDK engine and a from a US Patent Application 2016/0231568 I have added some red and green dots to the “waveguides” in the Verge picture to help you see their outlines.
A diffraction grating is a type of Diffractive Optical Element (DOE) and has a series of very fine linear structures with a period/repeated spacing on the order of the wavelengths of light (as in extremely small). 
A diffraction grating acts like a lens/prism to bend the light and as an unwanted side effect the light also is split separated by wavelength (see top figure at left) as well has affecting the polarization of the light. If it were a simple grating, the light would symmetrically split the light in two directions (top figure at left) but as the patent points out if the structure is tilted then more of the light will go in the desired direction (bottom figure at left). This is a very small structure (on the order of the wavelength of the light) must be formed on the surface of the flat waveguide.
Optical waveguides use the fact that once light enters glass or clear plastics at a certain angle or shallower, it is will totally reflect, what is known as Total Internal Reflection or TIR. The TIR critical angle is around 45 degrees for the typical glass and plastics with their coatings used in optics.
Hololens use the diffraction grating (52 in Fig 3B above) to bend or “incouple” the light or the light so that it will TIR (see figure at right). The light then TIR’s off of the flat surfaces around within the glass and hits off a triangular “fold zone” (in Fig. 3B above) which causes light to turn ~90 degrees down to the “exit zone” DOE (16 in Fig. 3B). The exit zone DOE causes the angle of the light to be reduced so it will no longer TIR so it can exit the glass toward the eye.
Another function of the waveguides, particularly the exit waveguide 16 is to perform “pupil expansion” or slightly diffusing the light so that the image can be viewed from a wider angle. Additionally, it is waveguide 16 that the user sees the real world through and invariably it has to have some negative effect from seeing the world through a slightly diffuse diffraction grating.
Hololens is far from the first to use DOE’s to enter and exit a flat waveguide (there are many examples) and they appear to have acquired the basic technology from Nokia’s efforts of about 10 years ago. Other’s have used holographic optical elements (HOE) which perform similar functions to DOEs and still others have use more prismatic structure in the waveguides, but each of these alternatives solves some issues as the expense of others.
A big issue for the flat combiners I have seen to date has been chroma aberrations, the breaking up of white light into colors and out of focus and haze effects. In bending the light at about 45 degrees is like going through a “prism” and the color separate, follow slightly different paths through the waveguide and are put back together by the exit grating. The process is not perfect and thus there is some error/haze/blur that can be multiple pixels wide. Additionally as pointed out earlier, the user is invariably looking at the real world through the structure meant to cause the light to exit the from the waveguide toward the eye and it has to have at least some negative effect.
There is a nice short 2013 article on flat combiners by (one author being a Google employee) that discusses some of the issues with various combiners including the Nokia one on which Hololens is base. In particular they stated:
“The main problems of such architecture are the complexity of the master fabrication and mass replication as well as the small angular bandwidth (related to the resulting FOV). In order to mimic the holographic Bragg effect, sub-wavelength tilted structures with a high aspect ratio are needed, difficult to mass replicate for low cost volume production”
Base on what I have heard from a couple of sources, the yield is indeed currently low and thus the manufacturing cost is high in making the Hololens combiner. This may or may not be a solvable (in terms of meeting a consumer acceptable price) problem with volume production.
While the Hololens combiner is a marvel of optical technology, one has to go back and try and understand why they wanted a thin flat combiner rather than say the vastly simpler (and less expensive maybe by over 10X) tilted flat combiner that say Osterhout Design Group (ODG), for example, is currently using. Maybe it is for some planned greater advantage in the long term, but when you look at the current Hololens flat combiner, the size/width of the combiner would seem to have little effect on the overall size of the resulting device. Interestingly, Microsoft has spent about $150 million in licensing fees to ODG.
Conclusions
Now step back and look at the size of the whole Hololens structure with the concentric bands going around the users head. There is inner band to grip the user’s head while the electronics is held in the outer band. There is a large nose bridge to distribute the weight on the persons nose and a big curve shield (usually dark tinted) in front of the combiner. You have to ask, did the flat optical combiner make a difference?
I don’t know reasons/rational/advantages of why Hololens has gone with a vastly more complex combiner structure. Clearly at the present, it does not give a significant (if any) size advantage. It almost looks like they had this high tech combiner technology and decided to use it regardless (maybe it was the starting point of the whole program).
Microsoft is likely investing several billion dollars into Hololens. Google likely spent over $1 billion on the comparatively very simple Google Glass (not to mention their investment in Magic Leap). Closely realated, Facebook spent $2b to acquire Oculus Rift. Certainly big money is being thrown around, but is it being spent wisely?
Side Comments: No Holograms Anywhere to be Found
What Microsoft calls “Holograms” are the marketing name Microsoft has given to Mixed Reality (MR). It is rather funny to see technical people that know better stumble around saying things like “holograms, but not really holograms, . . .” Unfortunately due to the size and marketing clout of Microsoft others such as Metavision has started calling what they are doing “holograms” too (but this does not make is true).
Then again probably over 99% of what the public thinks are “holograms” are not. Usually they are simple optical combiner effects cause by partial reflections off of glass or plastic.
Perhaps ironically, while Microsoft talks of holograms and the product as the “Hololens” there are as best I can find no holograms used even static ones that could have been used in the waveguide optics (they use diffraction gratings instead).
Also interestingly, the patent application is assigned to Microsoft Technology Licensing, LLC., a recently separated company from Microsoft Inc. This would appear to be in anticipation of future patent licensing/litigation (see for example).
Next Time on Combiners
Next time on this subject, I plan on discussing Magic Leap the $1.4 Billion invested “startup” and what it looks like they may be doing. I was originally planning on covering it with Hololens, but it became clear that it was too much to try and cover in one article.
Karl ,
MSFT out with a new patent today :
Waveguide eye tracking employing switchable diffraction gratings
http://pdfpiw.uspto.gov/.piw?PageNum=0&docid=09494799&IDKey=253E8128B46C&HomeUrl=http%3A%2F%2Fpatft.uspto.gov%2Fnetacgi%2Fnph-Parser%3FSect1%3DPTO2%2526Sect2%3DHITOFF%2526p%3D1%2526u%3D%25252Fnetahtml%25252FPTO%25252Fsearch-bool.html%2526r%3D1%2526f%3DG%2526l%3D50%2526co1%3DAND%2526d%3DPTXT%2526s1%3Demagin%2526OS%3Demagin%2526RS%3Demagin
There are different image generation technologies that can be used to implement micro display 120. For example, micro display 120 can be implemented using a transmissive projection technology where the light source is modulated by optically active material, backlit with white light. These technologies are usually implemented using LCD type displays with powerful backlights and high optical energy densities. Micro display 120 can also be implemented using a reflective technology for which external light is reflected and modulated by an optically active material. Digital light processing (DLP), liquid crystal on silicon (LCOS) and Mirasol.RTM. display technology from Qualcomm, Inc. are all examples of reflective technologies. Additionally, micro display 120 can be implemented using an emissive technology where light is generated by the display, see for example, a PicoP.TM. display engine from Microvision, Inc. Another example of emissive display technology is a micro organic light emitting diode (OLED) display. Companies eMagin and Microoled provide examples of micro OLED displays.
Thanks for the info on the patent. There is nothing that Magical about Magic Leap in this regards, a lot of companies are looking at things like this. I don’t believe we are going to see this type switchable waveguides of this type anytime soon in a production product. It goes back to the problem that you have to look through it. Putting many layers of structure (gratings, holograms, mirrors) is going to degrade the real world view and it gets worse if you add whatever makes them switchable.
I’m about 98% sure that Magic Leap is using a single plane combiner and are varying the focus of the input to the waveguide. Looking at all the evidence of the patents and the videos displayed to date as well as their ramping up some kind of manufacturing in the next year or two, and using some rational engineering judgement, you can eliminate a lot of things and you are left with a small set of what they could be doing.
The same goes for this Microsoft patent. It would be much easier to look at the eye and determine the focus distance and adjust the input and have a much simpler combiner. Patents are not constrained by practical realities.
Hi Karl,
I really enjoy reading your blog, your reports are well written and easy to read. Keep up the good job.
I have a very basic question about HUD for cars and also for motorcycles. Since you have worked on Navdy you are the person best suited to answer this question.
I’ve been reading about HUD for motorcycles and at least three products/prototypes claim that they can project a virtual image more than 2m beyond the combiner. One is the BMW HUD helmet made in collaboration with DIgiLens. I contacted DigiLens and they told me that the virtual image appears more than a couple of meters in front. Another one was Skully Helmet which I believe used a technology very similar to google glass. According to people who tested the prototype the virtual image was projected 5m in front. At last another one is LiveMap helmet and they claim that the virtual image appears 4 meters in front.
For car HUDs, such as Navdy or other HUDs (aftermarket or OEM built into the car), the virtual image is projected 1 or 1.5meters beyond the combiner. If motorcycle HUDs which are tiny can project a virtual image at 4meters or more why can’t car HUD project a virtual image at 6meters or even more (road infinity)? Does the fact that motorcycle HUDs are mounted near they eye play a role? Or is it that manufacturers of car HUDs are not interested in projecting a virtual image more than 2 meters?
Navdy used a simple spherical semi-mirror as the combiner and the optical element that “virtualizes” (changes the apparent focus) of the image that is projected onto a screen. A more curved combiner that would have made the image appear further out in space but it would have also distorted the image more. To make it work, the “screen” with the “real” image is slightly off-axis and would this further increases distortion. We also had to deal with the user being a radically different distances from the combiner based on the car design, the device placement and the user’s seating/eye position. These factors led to the design being more conservative in the virtual focus distance; but there is nothing keeping it from being further out. The built-in HUD designs have multiple mirrors to do various corrections and they don’t have the severe space issues putting a device on top of the dash.
Optically, Google Glass used a very similar design, only they use a mirror on the end and have a flat beam splitter for the combiner whereas in Navdy the mirror also acts as a combiner. Like Navdy, the mirror in their system also virtualizes the image (moves the apparent focus point). https://www.kguttag.com/wp-content/uploads/2017/02/Google-Glass-Birdbath-001.png
I have seen built-in HUD in both a BMW and a Mini Cooper. In the first the windshield is used as the combiner while in the second they use a separate piece of glass located on the dashboard. In both cases the virtual image appears to be projected close to the hood of the car, which would be 1m or 1.5m beyond the windshield.
Some head mounted displays (HMD), for example, the Vuzix glasses claim that the virtual image is projected at infinity. How come a head mounted display with the most severe space issues of all systems can project the virtual image at infinity and a built-in HUD which is 10times bigger can’t project the image more than a couple of meters? Do you know if HMD have inherent advantages when projecting a virtual image far away related to the proximity of the optics to the pupil of the eye?
I’m not a trained optics person and there are some holes in my knowledge. It’s not really necessary to move the focus point to infinity and I doubt that the near eye designs do (and most people couldn’t measure it), you just want to get it into the person’s far vision. One thing the near eye designs can do is add refractive optics.
I know that a number of near eye designs are optically very similar to what we did at Navdy, namely they use a spherical mirror to move the image out in space. Flat mirrors/beam splitters have no effect on the focus other than the linear distance of the light path and many of the near eye designs only have a single mirror. Many near eyes displays use a “birdbath” optical path (see pages 11 and 12 of http://www.kessleroptics.com/wp-content/pdfs/Optics-of-Near-to-Eye-Displays.pdf). But things are scaled up differently and the HUDs don’t use a beam splitter and thus are slightly off-axis. HUD’s have the viewer’s eye very far away and they need a HUGE eyebox/pupil compared to near eye as they have to deal with head versus just eye movement.
I know my trade-off with a simple single spherical mirror HUD was linearity of the image versus moving the focus further out. It also ties into image quality as magnification and apparent focus are linked. If you want to move the focus further out, you need to project a smaller image that gets magnified more and any imperfections in the screen become magnified (the HUD projects a very tiny image compared to a normal direct view front projectors).
Sorry for this but, The Icis glasses that are to be released for alpha later this year. They are using a holographic wave-guide combiner correct?
Is there a way to purchase HWC glass without purchasing the glasses for development projects?
I have been aware of Laforge (ICIS) since December of 2015 we another reader asked about them. They had a FAILED Indiegogo campaign for the ICIS way back in early 2014 or about 3.5 years ago (https://venturebeat.com/2014/02/24/laforge-optical-pulls-its-stylish-smart-glasses-indiegogo-campaign/).
What I wrote in 2015 still holds. In the over 1.5 years since, Laforge has put out nothing to convince me that they are not yet another crowdfunding scam (maybe not, but they have done nothing on-line that I can find that is not FAKE). I have not seen any credible reviewer use their product. If you know differently and have proof, let me know and I will print a full retraction.
By the way, when I checked out the Laforge ISCIS video it said that this was their “Old product” and they have moved on to the new “Shima” model. The only progress they seem to have made in over 3.5 years is to change the name of the product.
Oh by the way, I think they were planning embed a prism/mirror in the glasses and not a hologram.
Below is a copy of my reply from December 2, 2015.
Michael,
I was not familiar with Laforge before, but from what I could see on-line, it is yet one more near eye display CONCEPT with almost no details. I could find nothing about what it actually looks like other than the “Photoshop perfect” concept images. Have you actually seen it and worn it?
Your blog and podcasts are incredibly helpful! You talk in other posts about the optical efficiencies of birdbath and bug eye reflectors, but I can’t find any information on the real world efficiencies of diffractive and reflective waveguides like Hololens and Lumus. Do you have any information on this?
I have not been able to get exact numbers but both Lumus (multi-reflective) and Hololens (diffractive) output less than 10% of the input light and it is probably closer to 5% and perhaps less. Lumus has claimed to be more efficient than Hololens, but I have not been able to verify the claim. Lumus has an issue of the light having to pass through multiple layers of reflectors. Diffractive waveguides (Hololens, Digilens, Vuzix, and Magic Leap) have to deal with diffraction efficiencies and going through 3 sets of diffraction gratings (and thus 3X multiplicative losses).
This is one reason why Lumus, Hololens, and Magic Leap use LCOS and Digilens and Vuzix use DLP. Using an LED illuminator, DLP and LCOS can be arbitrarily bright (over 100,000 nits if necessary) given a bright enough LED and good collimation. I think Hololens is only about 300 nits on the output where Lumus products are spec’ed all the way up to 6,500 nits for their PD-16. Some of this difference efficiency, but much it is due to using a brighter light source.