Introduction

It has been sometime since I wrote about Magic’s Leap’s technical activity. So I thought in light of the recent Rolling Stone reveal, it was time to discuss what Magic Leap is really doing and separating out the marketing hype.

I have a lot of information to share and so I am going to have to break it up into a few parts (not sure how many yet). For this first part, I am going to primarily discuss patent application  2017/0276948 (‘948) filed March 24, 2017 and published September 28, 2017. Perhaps nothing sums up Magic Leap’s more recent intentions in display technology as much as this application.

LCOS Shown as Magic Leap’s Prime Display Example In 2017 Applications

Eleven (11) of the 2017 of Magic Leap’s applications have the figure shown (left) that explicitly shows an LCOS display engine with a typical beam splitter configuration and LED illumination. The Fiber Scanning Display (FSD), prominent in earlier Magic Leap applications has been relegated to being “some embodiments.” Fig. 6 shows the LCOS display system 250 being connected by a line to “image injection devices” (360-400) connect by an unnumbered line.

Some Patent Application “Archeology”

It is interesting to contrast with the figure 6 from 2017 above with Magic Leap’s 2015 ‘939 application (right) as shown on the left. “A plurality of displays (200, 202, 204, 206, 208), or in another embodiment a single multiplexed display . . .” became “image injection device” in the 2017 applications. But the cylindrical shape still looks like a smaller drawing of the FSD shown in Fig 13 in the 201 in the application such as in Fig. 13D on the (below). In the 2015 applications.

While much attention was lavished on fiber scanning display in the 2015 application mentioning it over 40 times, the application also mentioned DLP/DMD about 40 times. LCOS, however, was only mentioned once in the ‘939 application and not as a display device per say but for use in optical occlusion/masking (I will have to cover occlusion and Magic Leap in a future article).

Doing some “archeology” and peeling back the layers, it looks like the figures from 2015 (from filings in 2014 and provisional applications filed in late 2013) were drawn assuming FSD was the primary display. But at some point about the time of filing they were switching toward DLP for doing focus planes with the “optical multiplexing” being the DLP doing sequential images.

But by 2016 applications, LCOS is starting to show up as more than a passing mention, most significantly in ‘789 shown at left and discussed on this blog over a year ago, but on other applications, LCOS is still ending up on a list that starts with FSD.

As a side note. The “injection optics” 2060 are possibly the “image injection devices” 360-400 in Fig 8A above.

By 2017, we now see an LCOS engine being shown and mentioned ahead of FSD. And many of these same applications don’t even mention DLP.

Fiber Scanning Display Myth Continues

Magic Leap is still maintaining the pretext, at least in their patents, that fiber scanning display (FSD) might actually happen someday.  I am wondering who they are still trying to fool? I gave some of the reasons “No Fiber Scan Display (FSD)” in Nov. 2016 and some very simple math can prove that the speed of the fiber becomes impossibly high as the resolution increases (from zero to many times the speed of sound hundreds of times a second for 720p or above resolution).  Yet it never ceases to amaze me that people still believe in this technical snake oil.

It is more a sign of poor due diligence and gullibility that Magic Leap was not laughed out of the room when they presented it as one of their “Core Technologies” (see my article on Magic Leaps 2013’s “Confidential” Presentation). There are many people at Magic Leap that must know or should know how impossible it is to increase the resolution of FSD, yet they keep putting it in patent applications, like it is some sort of religious idol (which appears to have become at Magic Leap).

Two Focus Planes For VAC

Everything Magic Leap has done to date, both publicly and from my sources, say Magic Leap are trying to address the well-known 3-D stereo vision issue of “Vergence/Accommodation Conflict” (VAC) (see Figures 10A and 10 B above) with just two (2) “focus planes”.  I discussed VAC and Magic Leap’s general approach back on Nov 26th 2016. that also supports two focus planes. If anything, the more recent patent applications further confirms this. The figures above and the flow chart on the right (combining Figs. 29 to 21A) seem to summarize Magic Leap’s current approach and the issues they are trying to solve.

While the stack from Fig. 8A above suggests there are 5 layers of waveguide, there needs to be 3 waveguides per color (or 2 if you allow more blurring of blue). See the 2016 ‘789 Fig. 6 above for an example of using 3 waveguides per depth plane or 6 for two depth planes.

The ‘948 application recounts the VAC issue (Fig. 10A and 10B) and has a graph (“Fig.” 15 above) of how two focus planes reduce the amount of visual error measured in diopters (1/focal-length). The specification along with Fig. 19 to 21A show their approach which is pretty simple and very roughly appears to comes down to

• based on where the eye’s aim, select a focus plane for display
• if aim of the eyes move, wait for a saccade or blink and then change the plane or if it takes too long, change anyway
• “modify the image” (assume blur)  if there is content on the selected plane that exceeds a “threshold”

While this tends to confirm the direction Magic Leap is taking, it was for me,  lacking in terms of sophistication. In essence, this approach tracks the eye and its behavior and then generates an image on a single focus plane and then renders the parts of the image that are supposed to be out of focus as blurry.

This is consistent with Magic Leap moving from DLP that could support multiple simultaneous focus/depth planes to LCOS which is not typically fast enough to support both field sequential color and focus planes without having objectionable color sequential breakup). By only displaying one depth plane per frame, it would support LCOS being used.

Quoting the paper:

“Utilizing the techniques described herein, the perceived presentation quality of virtual content may be improved. For example, perceptible visual artifacts, such as flicker caused by switching content between different depth planes, may be reduced, particularly when the display system is operating in a vari-focal mode.”

This VAC approach seems to stand out in contrast with Avegant’s VAC method which uses DLP and which Avegant told me did not use eye tracking and rather they let the eye in real time choose the focus/depth plane. While a year ago I was given a brief chance to look through the Avegant headset prototype, I was not able to do an extensive evaluation and the image quality, but I did notice what could be describes as flickering image issues (but also note it was an early prototype). One thing to take away from Magic Leap’s application is that the depth/focus plane method is prone to causing temporal artifacts; and in this application at least, they are trying to mitigate the problems, not totally solve them.

Waveguides = Magic Leap’s “Photonic Lightfield Chip” Hype

On the far left is what Magic Leap in their patents describes (direct quote from the patent),  “With reference now to FIG. 9B, a perspective view of an example of the plurality of stacked waveguides.” This is what Magic Leap renames/hypes as a “Photonic LIghtfield Chip.” They look remarkably similar to the waveguide used by Microsoft’s Hololens shown to the right beside it. Hololens uses stacked/layered waveguides too (not shown in the figure below) for the various colors of light as well. It’s also widely rumored that it is the waveguide manufacturing that has been a major cost problem to Hololens and Magic Leap likely has twice as many.

As mentioned earlier, Fig. 9B above shows using 3 waveguides, one per color. To support two depth planes, there must be 6 waveguides (as shown in Magic Leap application 2017/0329075).  I will discuss more in future articles. This becomes a lot of layers of high precision optical devices to manufacture, yield, and look through (with negative effects).

Summary

As we add all the information up, it points to Magic Leap using LCOS for the main display device and waveguides similar to Microsoft’s Hololens. The big difference is that Magic Leap will have two focus planes and only show one at a time and then use software to blur virtual objects that should be out of focus. They are using a stack of 4 to 6 waveguides, divided into two focus planes.

There are also many known limitations in terms of image quality with a field sequential LCOS device going through a waveguide. Namely, you should expect something similar to Hololens.

As Magic Leap went from a marketing presentation to having to actually build a product, they had to live within the bounds of physics and look more like existing devices. Coming up with new names for what everyone else has already done, does not make it new.