AR/MR Combiners Part 2 – Hololens

hololens-combiner-with-patent

Microsoft’s Hololens is perhaps the most most well known device using flat “waveguide” optics to “combine” the real world with computer graphics. Note there are no actual “holograms” anywhere in Hololens by the scientific definition.

At left is a picture from the Verge Teardown of a Hololens SDK engine and a from a US Patent Application 2016/0231568 I have added some red and green dots to the “waveguides” in the Verge picture to help you see their outlines.

diffraction grating is a type of Diffractive Optical Element (DOE) and has a series of very fine linear structures with a period/repeated spacing on the order of the wavelengths of light (as in extremely small). hololens-diffraction-gratingA diffraction grating acts like a lens/prism to bend the light and as an unwanted side effect the light also is split separated by wavelength (see top figure at left) as well has affecting the polarization of the light. If it were a simple grating, the light would symmetrically split the light in two directions (top figure at left) but as the patent points out if the structure is tilted then more of the light will go in the desired direction (bottom figure at left).   This is a very small structure (on the order of the wavelength of the light) must be formed on the surface of the flat waveguide.

Optical waveguides use the fact that once light enters glass or clear plastics at a certain angle or shallower, it is will totally reflect, what is known as Total Internal Reflection or TIR.  The TIR critical angle is around 45 degrees for the typical glass and plastics with their coatings used in optics.

hololens-patent-sideviewHololens use the diffraction grating (52 in Fig 3B above) to bend or “incouple” the light or the light so that it will TIR (see figure at right).   The light then TIR’s off of the flat surfaces around within the glass and hits off a triangular “fold zone” (in Fig. 3B above) which causes light to turn ~90 degrees down to the “exit zone” DOE (16 in Fig. 3B).  The exit zone DOE causes the angle of the light to be reduced so it will no longer TIR so it can exit the glass toward the eye.

Another function of the waveguides, particularly the exit waveguide 16 is to perform “pupil expansion” or slightly diffusing the light so that the image can be viewed from a wider angle.   Additionally, it is waveguide 16 that the user sees the real world through and invariably it has to have some negative effect from seeing the world through a slightly diffuse diffraction grating.

Hololens is far from the first to use DOE’s to enter and exit a flat waveguide (there are many examples) and they appear to have acquired the basic technology from Nokia’s efforts of about 10 years ago.   Other’s have used holographic optical elements (HOE) which perform similar functions to DOEs and still others have use more prismatic structure in the waveguides, but each of these alternatives solves some issues as the expense of others.

A big issue for the flat combiners I have seen to date has been chroma aberrations, the breaking up of white light into colors and out of focus and haze effects.   In bending the light at about 45 degrees is like going through a “prism” and the color separate, follow slightly different paths through the waveguide and are put back together by the exit grating.  The process is not perfect and thus there is some error/haze/blur that can be multiple pixels wide. Additionally as pointed out earlier, the user is invariably looking  at the real world through the structure meant to cause the light to exit the from the waveguide toward the eye and it has to have at least some negative effect.

There is a nice short 2013 article on flat combiners by (one author being a Google employee) that discusses some of the issues with various combiners including the Nokia one on which Hololens is base.  In particular they stated:

“The main problems of such architecture are the complexity of the master fabrication and mass replication as well as the small angular bandwidth (related to the resulting FOV). In order to mimic the holographic Bragg effect, sub-wavelength tilted structures with a high aspect ratio are needed, difficult to mass replicate for low cost volume production”  

Base on what I have heard from a couple of sources, the yield is indeed currently low and thus the manufacturing cost is high in making the Hololens combiner.   This may or may not be a solvable (in terms of meeting a consumer acceptable price) problem with volume production.

hololens-odg-comparisonWhile the Hololens combiner is a marvel of optical technology, one has to go back and try and understand why they wanted a thin flat combiner rather than say the vastly simpler (and less expensive maybe by over 10X) tilted flat combiner that say Osterhout Design Group (ODG), for example, is currently using.   Maybe it is for some planned greater advantage in the long term, but when you look at the current Hololens flat combiner, the size/width of the combiner would seem to have little effect on the overall size of the resulting device.  Interestingly, Microsoft has spent about $150 million in licensing fees to ODG.

Conclusions

Now step back and look at the size of the whole Hololens structure with the concentric bands going around the users head.  There is inner band to grip the user’s head while the electronics is held in the outer band.  There is a large nose bridge to distribute the weight on the persons nose and a big curve shield (usually dark tinted) in front of the combiner.  You have to ask, did the flat optical combiner make a difference?

I don’t know reasons/rational/advantages of why Hololens has gone with a vastly more complex combiner structure.   Clearly at the present, it does not give a significant (if any) size advantage.   It almost looks like they had this high tech combiner technology and decided to use it regardless (maybe it was the starting point of the whole program).

Microsoft is likely investing several billion dollars into Hololens. Google likely spent over $1 billion on the comparatively very simple Google Glass (not to mention their investment in Magic Leap). Closely realated, Facebook spent $2b to acquire Oculus Rift. Certainly big money is being thrown around, but is it being spent wisely?

Side Comments: No Holograms Anywhere to be Found

What Microsoft calls “Holograms” are the marketing name Microsoft has given to Mixed Reality (MR).   It is rather funny to see technical people that know better stumble around saying things like “holograms, but not really holograms, . . .”  Unfortunately due to the size and marketing clout of Microsoft others such as Metavision has started calling what they are doing “holograms” too (but this does not make is true).

Then again probably over 99% of what the public thinks are “holograms” are not.  Usually they are simple optical combiner effects cause by partial reflections off of glass or plastic.

Perhaps ironically, while Microsoft talks of holograms and the product as the “Hololens” there are as best I can find no holograms used even static ones that could have been used in the waveguide optics (they use diffraction gratings instead).

Also interestingly, the patent application is assigned to Microsoft Technology Licensing, LLC., a recently separated company from Microsoft Inc.  This would appear to be in anticipation of future patent licensing/litigation (see for example).

Next Time on Combiners

Next time on this subject, I plan on discussing Magic Leap the $1.4 Billion invested “startup” and what it looks like they may be doing.   I was originally planning on covering it with Hololens, but it became clear that it was too much to try and cover in one article.

2 comments

  1. Frankenberry says:

    Karl ,

    MSFT out with a new patent today :

    Waveguide eye tracking employing switchable diffraction gratings

    http://pdfpiw.uspto.gov/.piw?PageNum=0&docid=09494799&IDKey=253E8128B46C&HomeUrl=http%3A%2F%2Fpatft.uspto.gov%2Fnetacgi%2Fnph-Parser%3FSect1%3DPTO2%2526Sect2%3DHITOFF%2526p%3D1%2526u%3D%25252Fnetahtml%25252FPTO%25252Fsearch-bool.html%2526r%3D1%2526f%3DG%2526l%3D50%2526co1%3DAND%2526d%3DPTXT%2526s1%3Demagin%2526OS%3Demagin%2526RS%3Demagin

    There are different image generation technologies that can be used to implement micro display 120. For example, micro display 120 can be implemented using a transmissive projection technology where the light source is modulated by optically active material, backlit with white light. These technologies are usually implemented using LCD type displays with powerful backlights and high optical energy densities. Micro display 120 can also be implemented using a reflective technology for which external light is reflected and modulated by an optically active material. Digital light processing (DLP), liquid crystal on silicon (LCOS) and Mirasol.RTM. display technology from Qualcomm, Inc. are all examples of reflective technologies. Additionally, micro display 120 can be implemented using an emissive technology where light is generated by the display, see for example, a PicoP.TM. display engine from Microvision, Inc. Another example of emissive display technology is a micro organic light emitting diode (OLED) display. Companies eMagin and Microoled provide examples of micro OLED displays.

    • KarlG says:

      Thanks for the info on the patent. There is nothing that Magical about Magic Leap in this regards, a lot of companies are looking at things like this. I don’t believe we are going to see this type switchable waveguides of this type anytime soon in a production product. It goes back to the problem that you have to look through it. Putting many layers of structure (gratings, holograms, mirrors) is going to degrade the real world view and it gets worse if you add whatever makes them switchable.

      I’m about 98% sure that Magic Leap is using a single plane combiner and are varying the focus of the input to the waveguide. Looking at all the evidence of the patents and the videos displayed to date as well as their ramping up some kind of manufacturing in the next year or two, and using some rational engineering judgement, you can eliminate a lot of things and you are left with a small set of what they could be doing.

      The same goes for this Microsoft patent. It would be much easier to look at the eye and determine the focus distance and adjust the input and have a much simpler combiner. Patents are not constrained by practical realities.

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.