Magic Leap Display Developments Revealed in 2017 – Part 1

Introduction

It has been sometime since I wrote about Magic’s Leap’s technical activity. So I thought in light of the recent Rolling Stone reveal, it was time to discuss what Magic Leap is really doing and separating out the marketing hype.

I have a lot of information to share and so I am going to have to break it up into a few parts (not sure how many yet). For this first part, I am going to primarily discuss patent application  2017/0276948 (‘948) filed March 24, 2017 and published September 28, 2017. Perhaps nothing sums up Magic Leap’s more recent intentions in display technology as much as this application.

LCOS Shown as Magic Leap’s Prime Display Example In 2017 Applications

Eleven (11) of the 2017 of Magic Leap’s applications have the figure shown (left) that explicitly shows an LCOS display engine with a typical beam splitter configuration and LED illumination. The Fiber Scanning Display (FSD), prominent in earlier Magic Leap applications has been relegated to being “some embodiments.” Fig. 6 shows the LCOS display system 250 being connected by a line to “image injection devices” (360-400) connect by an unnumbered line.

Some Patent Application “Archeology”

It is interesting to contrast with the figure 6 from 2017 above with Magic Leap’s 2015 ‘939 application (right) as shown on the left. “A plurality of displays (200, 202, 204, 206, 208), or in another embodiment a single multiplexed display . . .” became “image injection device” in the 2017 applications. But the cylindrical shape still looks like a smaller drawing of the FSD shown in Fig 13 in the 201 in the application such as in Fig. 13D on the (below). In the 2015 applications.

While much attention was lavished on fiber scanning display in the 2015 application mentioning it over 40 times, the application also mentioned DLP/DMD about 40 times. LCOS, however, was only mentioned once in the ‘939 application and not as a display device per say but for use in optical occlusion/masking (I will have to cover occlusion and Magic Leap in a future article).

Doing some “archeology” and peeling back the layers, it looks like the figures from 2015 (from filings in 2014 and provisional applications filed in late 2013) were drawn assuming FSD was the primary display. But at some point about the time of filing they were switching toward DLP for doing focus planes with the “optical multiplexing” being the DLP doing sequential images.

But by 2016 applications, LCOS is starting to show up as more than a passing mention, most significantly in ‘789 shown at left and discussed on this blog over a year ago, but on other applications, LCOS is still ending up on a list that starts with FSD.

As a side note. The “injection optics” 2060 are possibly the “image injection devices” 360-400 in Fig 8A above.

By 2017, we now see an LCOS engine being shown and mentioned ahead of FSD. And many of these same applications don’t even mention DLP.

Fiber Scanning Display Myth Continues

Magic Leap is still maintaining the pretext, at least in their patents, that fiber scanning display (FSD) might actually happen someday.  I am wondering who they are still trying to fool? I gave some of the reasons “No Fiber Scan Display (FSD)” in Nov. 2016 and some very simple math can prove that the speed of the fiber becomes impossibly high as the resolution increases (from zero to many times the speed of sound hundreds of times a second for 720p or above resolution).  Yet it never ceases to amaze me that people still believe in this technical snake oil.

It is more a sign of poor due diligence and gullibility that Magic Leap was not laughed out of the room when they presented it as one of their “Core Technologies” (see my article on Magic Leaps 2013’s “Confidential” Presentation). There are many people at Magic Leap that must know or should know how impossible it is to increase the resolution of FSD, yet they keep putting it in patent applications, like it is some sort of religious idol (which appears to have become at Magic Leap).

Two Focus Planes For VAC

Everything Magic Leap has done to date, both publicly and from my sources, say Magic Leap are trying to address the well-known 3-D stereo vision issue of “Vergence/Accommodation Conflict” (VAC) (see Figures 10A and 10 B above) with just two (2) “focus planes”.  I discussed VAC and Magic Leap’s general approach back on Nov 26th 2016. that also supports two focus planes. If anything, the more recent patent applications further confirms this. The figures above and the flow chart on the right (combining Figs. 29 to 21A) seem to summarize Magic Leap’s current approach and the issues they are trying to solve.

While the stack from Fig. 8A above suggests there are 5 layers of waveguide, there needs to be 3 waveguides per color (or 2 if you allow more blurring of blue). See the 2016 ‘789 Fig. 6 above for an example of using 3 waveguides per depth plane or 6 for two depth planes.

The ‘948 application recounts the VAC issue (Fig. 10A and 10B) and has a graph (“Fig.” 15 above) of how two focus planes reduce the amount of visual error measured in diopters (1/focal-length). The specification along with Fig. 19 to 21A show their approach which is pretty simple and very roughly appears to comes down to

  • based on where the eye’s aim, select a focus plane for display
  • if aim of the eyes move, wait for a saccade or blink and then change the plane or if it takes too long, change anyway
  • “modify the image” (assume blur)  if there is content on the selected plane that exceeds a “threshold”

While this tends to confirm the direction Magic Leap is taking, it was for me,  lacking in terms of sophistication. In essence, this approach tracks the eye and its behavior and then generates an image on a single focus plane and then renders the parts of the image that are supposed to be out of focus as blurry.

This is consistent with Magic Leap moving from DLP that could support multiple simultaneous focus/depth planes to LCOS which is not typically fast enough to support both field sequential color and focus planes without having objectionable color sequential breakup). By only displaying one depth plane per frame, it would support LCOS being used.

Quoting the paper:

Utilizing the techniques described herein, the perceived presentation quality of virtual content may be improved. For example, perceptible visual artifacts, such as flicker caused by switching content between different depth planes, may be reduced, particularly when the display system is operating in a vari-focal mode.”

This VAC approach seems to stand out in contrast with Avegant’s VAC method which uses DLP and which Avegant told me did not use eye tracking and rather they let the eye in real time choose the focus/depth plane. While a year ago I was given a brief chance to look through the Avegant headset prototype, I was not able to do an extensive evaluation and the image quality, but I did notice what could be describes as flickering image issues (but also note it was an early prototype). One thing to take away from Magic Leap’s application is that the depth/focus plane method is prone to causing temporal artifacts; and in this application at least, they are trying to mitigate the problems, not totally solve them.

Waveguides = Magic Leap’s “Photonic Lightfield Chip” Hype

On the far left is what Magic Leap in their patents describes (direct quote from the patent),  “With reference now to FIG. 9B, a perspective view of an example of the plurality of stacked waveguides.” This is what Magic Leap renames/hypes as a “Photonic LIghtfield Chip.” They look remarkably similar to the waveguide used by Microsoft’s Hololens shown to the right beside it. Hololens uses stacked/layered waveguides too (not shown in the figure below) for the various colors of light as well. It’s also widely rumored that it is the waveguide manufacturing that has been a major cost problem to Hololens and Magic Leap likely has twice as many.

As mentioned earlier, Fig. 9B above shows using 3 waveguides, one per color. To support two depth planes, there must be 6 waveguides (as shown in Magic Leap application 2017/0329075).  I will discuss more in future articles. This becomes a lot of layers of high precision optical devices to manufacture, yield, and look through (with negative effects).

Summary

As we add all the information up, it points to Magic Leap using LCOS for the main display device and waveguides similar to Microsoft’s Hololens. The big difference is that Magic Leap will have two focus planes and only show one at a time and then use software to blur virtual objects that should be out of focus. They are using a stack of 4 to 6 waveguides, divided into two focus planes.

There are also many known limitations in terms of image quality with a field sequential LCOS device going through a waveguide. Namely, you should expect something similar to Hololens.

As Magic Leap went from a marketing presentation to having to actually build a product, they had to live within the bounds of physics and look more like existing devices. Coming up with new names for what everyone else has already done, does not make it new.

 

18 comments

  1. Maybe not a right place for my question, but do like to your opinion about the potential of this technology: “https://www.nature.com/articles/srep09532” . Do you think a nice glasses free 3D display system can be built base on it? Thanks!

    1. Thanks for the question. The fundamental problem/trade-off with this or any other glasses-free 3D-display is that you end up trading a large amount of resolution just to get a little 3-D. The amount you trade, among other things is a function of the size of the sweet-spot. While you can do “interesting things” this type of display is it unlikely to make something suitable for a large market.

      1. I am guilty of connecting some dots here. I am basing this on how I saw things unfold in chronilogical order:
        1. MS releases Hololens Dev. Ed. Chief complaint is narrow FOV.
        2. MS cancels ver. 2 of Hololens to work on ver 3 in improve (among other things) the narrow FOV.
        3. MS releases a paper https://www.microsoft.com/en-us/research/wp-content/uploads/2017/05/holo_author.pdf to show they know a way to potentially fix the narrow FOV. The paper is all about Phase controlled LCOS as a technique for NED.
        4. Himax, MS partner (for all intents and purposes) puts this on their website shortly after MS releases the above paper: http://www.himax.com.tw/products/microdisplay-products/lc-adaptive-lens-products/. Under applications, it says this: “Augmented reality and Virtual reality – Image plane registration, Vergence/Accommodation conflict compensation, and vision correction.”

        I am theorizing that Himax was trying to demonstrate (indirectly) that they are still involved in the future of Hololens, and that it may involve phase controlled LCOS with MS.
        Feel free to dismantle my linkage.

        1. I think you may be connecting dots that don’t really connect. The fact that people are doing research work on Holograms with Phase LCOS does not mean that it in some way helps FOV. The issues are in some ways orthogonal.

          You might be interested in the Oculus focus surface work that used Phase LCOS for creating “focus surfaces” https://www.oculus.com/blog/oculus-research-to-present-focal-surface-display-discovery-at-siggraph/. But once again using Phase LCOS does not help the FOV and I don’t see it as being practical for broad use anytime soon. This effect is related to Himax’s used of Phase LCOS for focus control, but there are lots of companies that make Phase LCOS, in part because, Phase LCOS in the past has been used primarily for optical switching

          1. I connected that dot because of what is said in the paper, like this:
            “We use this optical correction ability not only to fix minor aberrations
            but to enable truly compact, eyeglasses-like displays with wide fields
            of view (80◦) that would be inaccessible through conventional means.”

            This would be a significant improvement over version 1.
            I am not the only one who made this association. This one is quite direct:
            http://www.techradar.com/news/wearables/this-is-how-microsoft-s-hololens-will-address-its-biggest-flaw-1322596

            Thanks for the dialog.

          2. I can not agree more with the FOV issu of the phase only LCOS.As a diffraction element or in CGH,origin pixels of LCOS have diffract orders,FOV will limit in +-1 diffract orders, smaller pixel size or substructure in pixel may improve it.Another issue is the frame rate of phase only LCOS,it is hard to improve because phase only LCOS has 2pi phase deepth and thicker than common LCOS. Jasper SLM and Holoeye SLM are 60hz frame rate. For two focus layer,30hz in single color and 10hz in CS-RGB. About the muli-focus with waveguide,in my understand, it can just transfor light which focus infinity,does magic leap use other element to switch the focus distance in the outcouple area of the waveguide?

    1. Himax seems to be the odds on favorite for being the LCOS supplier, but there are others making LCOS including, Syndiant, RaonTech, Jasper, Sony (sort of), and I hear some Chinese companies (but I have not seen them).

  2. Again it may not be the right place to ask, but is it theoritically possible to create a black colour in see through lenses?

    1. The KEY problem is FOCUS. You can put a black dot on the front of a lens and it will have next to no effect (just darken the image very slightly). The trick is to get the occlusion mask to be in focus with the light it is blocking from the real world. If the reals world is “complex” (like the true real world) with objects at a variety of focus distancing, the whole things blows up.

  3. Fascinating as always. Avegant have a Dev kit available. I assume they’ll be at CES? Will you be blogging while you’re there?

    It will be pretty disappointing if Magic leap isnt significantly better than hololens. Is there any chance Magic Leap One could have something not in any patent application yet?

    1. I don’t think Avegant is demo’ing this year at CES.

      I will probably report after CES as it is usually too crazy when I am there. I tend to have 4 to 5 meetings a day plus trying to see stuff and getting from place to place.

      While there is always a chance Magic Leap could go “Patent Bare” and either not patent stuff or wait and risk someone else beating them to the patent office. The America Invents Act (actually the anti-American Inventor Act, this act is despicable) instituted in 2011 is just terrible for companies wanting to keep something secret while protecting their patent rights.

      I don’t think there is something fundamentally new that Magic Leap is doing that will turn up as as Magic Leap appears to be far behind their original plans and they have filed about 300 patents. Why would they leave out something really important?

      1. Yes,Avegant is not going to demo in CES show as they‘ve been told not going to present by their investor.

        Somehow I believe the way of digital light field from Magic Leap, is very much similar to the way that Avegant has done.

        I check that maybe DLP or FLCoS with high frame rate performance so far can make such concept of light field, and numbers of focal plane could be generated are 2 at minimum or 4 at maximum if the GPU is very powerful.

        Do you think 2 focal plane would be enough?

        1. The optics Avegant and Magic leap are using for VAC are different. Magic Leap imparts the focus plane with the exiting of the light from the waveguide whereas Avegant changes the focus before it. It takes a lot of optical complexity for Magic Leap to support even two focus depths were Avegant could support more.

          F-LCOS is still not nearly as fast in this application.

          No, I don’t think 2 focus planes is enough. I think Magic Leap ended up with just 2 as all they could do with a conventional display and waveguides after they figured out that Fiber Scanning Display (FSD) was not going to work. I think it is going to be a lousy compromise; as in it will not really solve the problem they were after while adding a lot of cost and making the “normal” image worse.

          1. If Magic Leap chose to make the hologram with multiple-stacking optical element. It is true that you can make lots focal plane, but to do this, i can image it will extramly difficult to do. The waveguide optical element will thicker, the alignment for each element must acturate enough. Those challenges to make me think even if they can do this, but how to supress the retail price under 1000USD ? I think 1000USD is really a margin for consumer electronics.

Leave a Reply

Your email address will not be published. Required fields are marked *