Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
Hololens 2 Display Evaluation (Part 4: LBS Optics)
Introduction
I received an email from David Kessler, an expert in optics and who has designed several laser beam scanning displays. In that email, David wrote that my speculation Hololens 2 (HL2) might be using the pupil expansion method shown in their US10,025,093 patent (right) was incorrect. David said that the HL2 was not using a screen-like pupil expander (EPE 306).
Microsoft’s Hololens 2 announcement video, a still from which is on the left, showed the scanning engine with mirror optics. More recently, there was a teardown with a video by a Microvision stock investor “u/s2upid” user on Reddit who tore apart a Trimble version HL2 just to find the Microvision logos inside.
I thought that optics in the 10,024,093 patent might be hiding behind the sizeable slow-scan mirror in the figure from the video above to expand the pupil, but this proves not to be the case. I went back to search the Microsoft patents the explanation of how the HL2 pupil expansion optics works
After going through the HL2’s pupil expansion patent, I will discuss some other tops, including the size of the engine and some display artifacts.
Microsoft’s “Increasing Pupil Size In A Display System” Patent Application
Patent Application US2019/0278076, “Increasing Pupil Size In A Display System,” matches the optics in the video and the teardown quite closely. Below are some still frames from the teardown video and the key figure in the patent application. Dimensions have been added (in blue).
As the patent explains, the pupil of the fast scan mirror must be small to enable the fast-scan mirror to be small enough that it can faster without distorting. Quoting the application in reference to FIG. 5-2 above:
As the fast scan arc 338 increases, the forces on the fast scan mirror 310 necessary to move through the fast scan arc 338 with a frequency greater than 10 kHz, 20 kHz, or 30 kHz may begin to distort the fast scan mirror 310. Decreasing the size of the fast scan mirror 310 through magnification of the pupil size may limit and/or prevent distortion of the fast scan mirror 310.
Figure 6 shows a simplified side view of the optics in figure 5-2. The figure has been rotated 90 degrees from the patent to orient it the same as both Fig. 5-2 and in the HL2 with the laser output shooing up.
On the HL2, there are three sets of converging optical elements (440). The laser module (right) is on the opposite side of the teardown pictures above. Each of the red, green, and blue colors has dual lasers with shared converging optics (440 in application Fig. 6 above). The lasers go to a conventional dichroic color combiner (not shown in FIG 6) and then to a turning prism with mirror surfaces to direct the light upward toward the fast scan mirror (410/310 above).
Fig. 7 from the patent application (below) shows how the pupil size changes as it goes through the optics of figures 5-2 and 6 above. Note that the output pupil is taller than it is wide. Going back to the teardown pictures, the output port on the optics is also taller than it is wide, suggesting that the display image is also being anamorphically compressed in the horizontal direction.
Pupil Shaping and Anamorphic Image Distortion
On the left is a front view of the laser engine attached to the waveguide from the teardown with added dimensions in blue. Note that the engine is roughly a rectangular solid that is 24.4mm by 29mm by 19mm (approximately an inch on a side) or about 13.4 cubic centimeters, very large for an AR/MR headset display engine.
Note in the picture of the HL2 waveguide that the output in the waveguide is taller than it is wide. You might also notice the somewhat triangular-shaped left (cyan reflected light) and right (blue-reflected light) intermediate DOE expander’s inputs are similarly taller than they are wide.
I have not seen the anamorphic distortion of the image itself being discussed in the Microsoft patents/applications. It appears that the HL2’s butterfly waveguide ends up roughly doubling the width of the image, as about half of the image propagated down each side of the diffractive optical elements (DOEs), as shown in Microsoft’s US2017/0363871 application (below). Figures 13 and 14 from the patent are combined with the parts of figure 14 that were different in figure 13 in red.
Figure 16 from the same patent shows how the left and right portions combine, in effect, roughly doubling the resultant image width. While not explicitly discussed in the patents that I could find, there appears to be an anamorphic horizontal stretching inherent in the butterfly waveguide. The horizontal stretching by the waveguide then driven the need horizontally to compress the output from the laser engine. And thus the taller than wide output window as seen in the pictures above.
HL2 Optics are Large for AR/MR
As shown in the pictures above, the laser combining and pupil expansion optics results in a large optical engine of about 13.4 cubic centimeters.
For comparison below, I have included a few recent AR headset display engines for diffractive waveguides. Compound Photonics has an engine with less than a quarter the volume with a similar FOV and much higher resolution using LCOS. Waveoptics has a DLP based engine with slightly better resolution but a somewhat smaller FOV that is about 70% the volume. At CES this year, Vuzix and JadeBird (using Waveoptics) had prototypes of green (only) MicroLEDs with engines that are about 1cc (~10mm on a side) that they claim can be much smaller in production to show the future potential of MicroLEDs
Trapezoidal Bright Area Possible Explanation
Looking back and the patent’s combined figures 13 and 14 something interesting to note is how the light progresses diagonally through the left and right intermediate DOE.
The diagonal propagation seems to explain the trapezoidal shape I noted in Hololens Display Evaluation (Part 3: Color Uniformity) as seen in the picture (left).
Thin, 1 or 2 Scan Lines wide, Diagonal Lines
High-resolution pictures such as the one on the left, show diagonal lines in the display. In the last article, I speculated that they might be a screen-like exit pupil expander (EPE), but the evidence above, shows there is not an EPE “screen.”
The figure below shows crops from the left, center, and right side of the display as indicated by the rectangles in the picture of the whole display.
- The lines slant from right to left, going top to bottom on the left side of the display and right to left on the bottom (see left and right above).
- As shown, the lines transition from slanting left to slanting right toward the middle of the display (see center image below).
- In some areas, the lines are spaced apart horizontally by about the same distances as two scan lines. The thickness of the lines varies.
- In some areas, the lines are spaced apart by the width of a single scan line. In a transition area, the lines will get thinner, and then a line in between the two lines will start. (left)
- The slope of the lines is about 60 degrees.
- The lines are “sharp” with longer exposures (8+ fields), suggesting they don’t move.
- The lines are even evident in “black” parts of the image suggesting they are picking up scattered light (see above right)
- While the image has chroma aberrations (see left), the lines don’t, this suggests that the lines are optically near the output (perhaps the output grating).
- The lack of chroma aberrations suggests that whatever is causing the lines is near or on the output grating.
- If the lines are from the output grating, then they are on the order of 12 to 13 microns for more tightly spaced lines and 24 to 26 microns for the line with lower periodicity (~854 scan lines per 29 degrees vertical FOV and the waveguide about 20mm from the eye). So it does not seem like the gratings themselves, which should be nearer 1 micron (wavelength of light times about 1.4x).
Conclusions
The HL2 is using lenses to aim the beam onto a small, fast-moving mirror. It then uses other mirror optics to shape the pupil for entrance into the waveguide. The use of mirrors to shape the pupil is likely much more energy-efficient than using a screen-like pupil expander, but like most mirror optics, it takes up a larger volume. It’s not clear how much this type of pupil expander could shrink.
I think there is an answer for the trapezoidal area in the HL2’s display that is brighter and more uniform. It is where both the left and right DOEs contribute to the output image as a result of the diagonal traveling of light.
The thin roughly 60 degree diagonal lines remain a bit of a mystery. If you have a reason or even a functional theory, let me know.
Can you see the diagonal lines when you are wearing the display?
Yes, but it takes concentration. There is flicker/rippling and every other line occasionally disappears due to the interlaced scanning.
Thank you (as always!) for your analysis. About the mysterious diagonal lines, did you also see it with your eyes? If it only showed up in pictures taken by a camera but not visible with naked eyes, I think it may be a kind of “moire” which are caused by optical interference between your camera CCD pattern and some patterns in the HL2 optics. But I’m not sure because there’s no definable “pixel” (which is a pattern with a fixed frequency) in a LBS image as in a DLP or LCoS.
Thanks, Rex,
It takes concentration but I can see both the lines and can see with my eyes the moire pattern where the lines change slants. I can see the thinner lines in the center and left side of the display but not so much the thicker one on the right. The pictures in the article were taken with 45mm lens and there were about 7.5 pixels per scan line in the image. The lines are also visible, and where I first discovered them, with a 25mm lens and ~3.5 pixels per scan line. Either way and particularly with the 45mm lens, the pictures are well above the Nyquist rate even with a Bayer filter on the camera. The patterns also don’t move if the camera is moved.
Hi, Karl, this is Liang, i have been worked on AR optics for many years. you can clear see that these lines are also infocus, that means they are not from waveguide or mirrors, they are from the image source, if this is oled etc, it will be easier, for this, since each pixel is directly scaned, then it will be related with the actuation, there are two directions scannnig for MV, single drive input contains both slow scan ramp and fast scan sinusoidal drive signals, thus it explains why at the edge of the display, the lines are thickner and less dense than center field. If MV is using Lissajous modulation for scanning, i am quite sure they will be Lissajous pattern.
i hope you find this help.
Thank, Liang,
It is a good point about them being in-focus. There is a lens between the waveguide and the eye/camera that move the focus point to about 2 meters from near infinity. BTW, there is a compensating lens on the other side of the waveguide so it does not defocus the real-world. As the lens is making the collimated image move form to about 2 meters, I would ten to agree that it should not make the exit grating become in-focus.
I’m sure it is not a laser scanning Lissajous pattern. You can see in the “transition area” figure (https://www.kguttag.com/wp-content/uploads/2020/07/Transition-Diagonal-LInes-Lower-Left-Corner-P7150118-45mm_f6.3_15th-ISO100-C.jpg) where lines are discretely being added. Also, note that you can see the lines continuing the “dark area” image (https://www.kguttag.com/wp-content/uploads/2020/07/Lines-in-black-image-P7150118-45mm_f6.3_15th-ISO100-1.jpg ) suggesting that it is a surface that is picking up the scattered light and not directly from the laser scanning.
If MV is using Lissajous modulation for scanning, i am quite sure they will be Lissajous pattern.
i hope you find this help.
what I mean is that MV is not Lissajous, so that’s why we see they are lines instead of Lissajous pattern.
if it is a surface for scatter light, it should not be in focus.
Hi Karl,
Since this is a kind of analog device could this pattern be some kind of high-frequency noise coming from electronics that is steering the fast mirror and influencing other parts of the scanner electronics (e.g. laser source)?
That would be my silly guess.
Hi Karl,
Thanks for the article. Really great explanations and breakdowns on everything.
I had a question about the steering between the fast and slow mirrors. In Figure 6, what function do the cylinder mirrors and planar mirror serve for the assembly?
Thanks.
The cylindrical mirrors stretch the pupil and the image as shown in Fig. 7. The flat mirrors simply add distance by folding the path of the light. They start by concentrating the laser beams with lenses to be small using lenses so the fast mirror can be small. Mirrors 434-1 and 434-4 (the horizontal curve mirrors after the horizontal/fast mirror) shapes the light coming off the horizontal mirror in the horizontal direction. It is both going to affect the beam shape and horizontal dimension of the image. Mirror 434-3 is going to shape the beam vertically (but not the image as it has not been spread vertically yet). Refractive elements 336 (Fig 5-2). do some final bean shaping and horizontal image shaping.
There does not appear to be image shaping after the slow scan mirror. The pupil/beam and horizontal image is all shaped before it.
Curve mirrors are inexpensive ways to get to relatively good image quality with no chroma aberrations (color separation). The big downside of mirrors is that they generally take more space than refractive optics and you have to deal with the light reflecting back in the direction from which it came. There either needs to be a “beam splitter” to be “on-axis,” or the light has to be off-axis (as is done with the HL2 which introduces some distortion. I don’t know if the distortion in this case, as it is off-axis, is considered acceptable or if they try to remove the distortion with the refractive doublet (336 in fig 5-2 and seen in the teardown pictures).
Karl，
thank you for this informative post. To my understanding, any imaging optical device like mirror, lens can NOT work as an exit pupil expander (EPE). yes, they could strech the pupil slightly but not enough limited by Lagrange invariant.
ALL EPEs need some mechnism like scatering, multiple reflection, etc. multi reflections in a wave guide act as EPE.
I was expecting to see something like the EPE as shown in the 10,025,093 patent figure, but the teardown shows otherwise. The HL2 is using optics to shape the pupil as shown in patent application US2019/0278076. “Shape” rather than “expand” may be a better way to put it.
Hi Karl,
My guess is that a diffraction waveguide acts a little bit like an interferometer. When a beam is diffracted out of the waveguide, it splits, one part goes in the eye (or sensor) the rest is bouncing into the waveguide before being split again and go into the eye again with a small optical path difference (a little bit like this but with a thicker layer, hence the thinner lines: https://en.wikipedia.org/wiki/Thin-film_interference). Then, if one use lasers into waveguide, this is possible that the interference fringes can be visible, even in the white, just like one can see “white” speckle when using LBS displays. As the fringes are formed on the retina (or camera sensor), they are in focus even thought there are not generated by the display itself. This would also explain the slanting of the lines that follow the light propagation into the waveguide. Also, the interference line spacing will change in relation to the FOV as the distance between the multiple beams generated is changing with the FOV. Using 2 lasers per color could be an attempt to mitigate this effect.
A way to test this could be to project a pure red image (by making sure that only the red lasers are turned on) and then a pure blue image and comparing the line spacing at the same location in the image. It should change with wavelength.
Cheers,
[…] and optics from Lumus’s website. I compared it to a teardown picture for the Hololens 2 that I wrote about in Hololens 2 Display Evaluation and adjusted them to the same scale. The Maximus’s display engine appears to be about 1/4th […]
hi karl
These lines are produced by laser interference. They may be caused by the reflection and diffraction of the 0-order light in the coupling area, or the interference between other non-propagating orders and propagating orders.
[…] Oqmented and Dispelix glasses show much smaller engines than the Hololens 2 (see: Hololens 2 Display Evaluation (Part 4: LBS Optics)). But then again, they are much lower resolution with much smaller fields of view (FOV). They are […]
[…] ML2’s optical stack (after the projector) is shown on the right. On the surface, it looks a lot like the stack of the Hololens 1 other than the dimmer layer. Nothing very surprising. The ML2 uses very high index 2.0 glass (only recently available), which helps support a wider FOV than the Hololens 2 without resorting to the complex and image compromising “butterfly” design of the Hololens 2. […]