Varjo Foveated Display (Part 1)

Introduction

The startup Varjo recently announced and did a large number of interviews with the technical press about their Foveated Display (FD) Technology. I’m going to break this article into multiple parts, as currently planned, the first part will discuss the concept and the need for and part 2 will discuss how well I think it will work.

How It Is Suppose to Work

Varjo’s basic concept is relatively simple (see figure at left – click on it to pop it out). Varjo optically combines a OLED microdisplay with small pixels to give high angular resolution over a small area (what they call the “foveated display“), with a larger OLED display to give low angular resolution over a large area (what they call the “context display“). By eye tracking (not done in the current prototype), the foveated display is optically moved to be in the center of the person’s vision by tilting the beam splitter. Varjo says they have thought of and are patenting other ways of optically combining and moving the foveated image other than a beam splitter.

The beam splitter is likely just a partially silvered mirror. It could be 50/50 or some other ratio to match the brightness of the large and microdisplay OLED. This type of combining is very old and well understood. They likely will blend/fade-in the image in the rectangular boarder where the two display images meet.

The figure above is based on a sketch by Urho Konttori, CEO of Varjo in a video interview with Robert Scoble combined with pictures of the prototype in Ubergismo (see below), plus answers to some questions I posed to Varjo. It is roughly drawn to scale based on the available information. The only thing I am not sure about is the “microdisplay lens” which was shown but not described in the Scoble interview. This lens(es) may or may not be necessary based on the distance of the microdisplay from the beam combiner and could be used to help make the microdisplay pixels appear smaller or larger. If the optical path though the beam combiner to large OLED (in the prototype from an Oculus headset) would equal the path from to the microdisplay via reflecting off the combiner, then the microdisplay lens would not be necessary. Based on my scale drawing and looking at the prototype photographs it would be close to not needing the lens.

Varjo is likely using either an eMagin OLED microdisplay with a 9.3 micron pixel pitch or a Sony OLED microdisplay with a 8.7 micron pixel pitch. The Oculus headset OLED has ~55.7 micron pixel pitch. It does not look from the configuration like the microdisplay image will be magnified or shrunk significantly relative to the larger OLED. Making this assumption, the microdisplay image is about 55.7/9 = ~6.2 time smaller linearly or effectively ~38 times the pixels per unit area. This ~38 times the area means effectively 38 times the pixels over the large OLED alone.

The good thing about this configuration is that it is very simple and straightforward and is a classically simple way to combine two image, at least that is the way it looks. But the devil is often in the details, particularly in what the prototype is not doing.

Current Varjo Prototype Does Not Track the Eye

The Varjo “prototype” (picture at left from is from Ubergismo) is more of a concept demonstrator in that it does not demonstrate moving the high resolution image with eye tracking. The current unit is based on a modified Oculus headset (obvious from the picture, see red oval I added to the picture). They are using the two Oculus larger OLED displays the context (wide FOV) image and have added an OLED microdisplay per eye for the foveated display. In this prototype, they have a static beam splitter to combine the two images. In the prototype, the location of the high resolution part of the image is fixed/static and requires that the user look straight ahead to get the foveated effect. While eye tracking is well understood, it is not clear how successfully they can make the high resolution inset image track the eye and whether the a human will notice the boundary (I will save the rest of this discussion for part 2).

Foveated Displays Raison D’être

Near eye display resolution is improving at a very slow rate and is unlikely to dramatically improve. People quoting “Moore’s Law” applying to display devices are simply either dishonest or don’t understand the problems. Microdisplays (on I.C.s) are already being limited by the physics of diffraction as their pixels (or color sub-pixels) get withing 5 times the wavelengths of visible light. The cost of making microdisplays bigger to support more pixels drives the cost up dramatically and this not rapidly improving; thus high resolution microdisplays are still and will remain very expensive.

Direct view display technologies while they have become very good at making large high resolution display, they can’t be make small enough for lightweight head-mounted displays with high angular resolution. As I discussed the Gap in Pixel Sizes (and for reference, I have included the chart from that article) which I published before I heard of Varjo, microdisplays enable high angular resolution but small FOV while adapted direct view display support low angular resolution with a wide FOV. I was already planning on explaining why Foveated Displays are the only way in the foreseeable future to support high angular resolution with a wide FOV: So from my perspective, Varjo’s announcement was timely.

Foveated Displays In Theory Should Work

It is well known that the human eye’s resolution falls off considerably from the high resolution fovea/center vision to the peripheral vision (see the typical graph at right). I should caution, that this is for a still image and that the human visual system is not this simple; in particular it has sensitivity to motion that this graph can’t capture.

It has been well proven by many research groups that if you can track the eye and provide variable resolution the eye cannot tell the difference from a high resolution display (a search for “Foveated” will turn up many references and videos). The primary use today is with Foveated Rendering to greatly reduce the computational requirements of VR environment.

Varjo is trying to exploit the same foveated effect to gives effectively very high resolution from two (per eye) much lower resolution displays. In theory, it could work but will in in practice?  In fact, the idea of a “Foveated Display” is not new. Magic Leap discussed it in their patents with a fiber scanning display. Personally, the idea seems to come up a lot in “casual discussions” on the limits of display resolution. The key question becomes: Is Varjo’s approach going to be practical and will it work well?

Obvious Issues With Varjo’s Foveated Display

The main lens (nearest the eye) is designed to bring the large OLED in focus like most of today’s VR headsets. And the first obvious issues is that the lens in a typical VR headset is designed resolve pixels that are more than 6 times smaller. Typical VR headsets lenses are, well . . ., cheap crap with horrible image quality. To some degree, they are deliberately blurring/bad to try and hide the screen door effect of the highly magnified large display. But the Varjo headset would need vastly better, and much more expensive, and likely larger and heavier optics for the foveated display; for example instead of using a simple cheap plastic lens, they may need a multiple element (multiple lenses) and perhaps made of glass.

The next issue is that of the tilting combiner and the way it moves the image. For simple up down movement of the foveated display’s image will follow a simple path up/down path, but if the 45 degree angle mirror tilts side to side the center of the image will follow an elliptical path and rotate making it more difficult to align with the context image.

I would also be very concerned about the focus of the image as the mirror tilts through of the range as the path lengths from the microdisplay to the main optics changes both to the center (which might be fixable by complex movement of the beam splitter) and the corners (which may be much more difficult to solve).

Then there is the general issue of will the user be able to detect the blend point between the foveated and context displays. They have to map the rotated foveated image match the context display which will loose (per Nyquist re-sampling) about 1/2 the resolution of the foveated image. While they will likely try cross-fade between the foveated and context display, I am concerned (to be addressed in more detail in part 2) that the visible/human detectable particularly when things move (the eye is very sensitive to movement).

What About Vergence/Accommodation (VAC)?

The optical configuration of Varjo’s Foveated Display is somewhat similar to that of Oculus’s VAC display. Both leverage a beam splitter, but then how would you do VAC with a Foveated Display?

In my opinion, solving the resolution with wide field of view is a more important/fundamentally necessary problem to solve that VAC at the moment. It is not that VAC is not a real issue, but if you don’t have resolution with wide FOV, then VAC is not really necessary?

At the same time, this points out how far away headsets that “solve all the world’s problems” are from production. If you believe that high resolution with a wide field of view that also address VAC, you may be in for a many decades wait.

Does Varjo Have a Practical Foveated Display Solution?

So the problem with display resolution/FOV growth is real and in theory a foveated display could address this issue. But has Varjo solved it? At this point, I am not convinced, and I will try and work though some numbers and more detail reasoning in part 2.

17 comments

  1. first, lol, nice write up, i was starting to get excited and started to catch the hype bug. hard to keep the imagination in check sometimes.

    • KarlG says:

      I think foveated displays is an important concept. But it is a different question as to whether this is ready for prime time anytime soon. I think the devils it in the details and I am particularly worried about the transition between the two regions being perceptible as well as whether the central image movement will work (and stay in focus).

      Foveated displays are likely to be more important than say vergence/accommodation which seems to have gotten more press. But neither is likely to be a mass market product in the next few years.

  2. Jack H (@Halo_AR_ltd) says:

    Am I right in thinking if they place a first steering mirror before the combining steerable mirror and have both rotate about their centroids then both varying path length and image stretching are avoided?
    If the above is the case then I suppose having the steering pair as above the combiner would be better since they would only need to be as big as the microdisplay image.
    Another option they might try is shifting the microdisplay directly, on its x and y axes.
    Also the varying transparency and slight image shift of the peripheral display as that on-axis combiner moves would be annoying if not fully corrected.

    David Kessler had his own foveated display invention:
    http://www.kessleroptics.com/portfolio/near-to-eye-displays/foveated-laser-display/

    I’ll send you on a sneak peak of my own optical foveation design in a while.

    -Jack

    • KarlG says:

      I don’t know if I totally follow you. I’m was going by Varjo’s CEO description. Are you saying to have one mirror scan horizontally and the other vertically? I would think this would still have a distortion in it but less (sort of like with single versus dual mirror laser beam scanning). I would think there would be a focus problem is the path lengths from side to side are different. Sometimes when you add things such as moving the microcisplay, you don’t really solve the problem so much a move it to a different problem.

      I’m not sure if I followed you varying transparency comment, but it made me think that the beam splitters optical characteristics might change as it tilts making it harder to match the two images (which will be tricky in both brightness and color, thus there will have to be a wide transition range.

      Yes, I saw David Kessler’s foveated display in researching the topic before the article and meant to mention it but forgot to. There are others working on foveated display. I will be interested to see what you come up with.

  3. Nick says:

    David Kessler should not be publishing this.

    • KarlG says:

      It would be interesting to know why, but I found it readily in a web (Google) search. I make it a practice to not publish anything I think should be private.

  4. KirkM says:

    No need to eye-track if the detailed image size captures small eye movements, and there are some good rules-of-thumb on this number. Also not convinced that 50/50 is automatically the right BS number. I would want to see a comparison with a brighter central field and also maybe a custom BS with variable T that would do the blending. Finally, this general concept can also be implemented as a binocular solution with some interesting trade-offs.

    • KarlG says:

      I can’t follow/understand your comment. Perhaps you could give some more information. Varjo is making binocular near eye display system. The whole concept of a Foveated Display is to provide high resolution only where the eye is looking, so I don’t understand at all your comment that no eye tracking is needed.

      I would agree that the ratio of transmissive/reflective for the beam splitter is dependent on the relative brightness of the two displays. Not sure about the variable transmission (T?) as that would introduce other issues, particular moving the display image.

  5. Hawkeye says:

    Hi Karl- First time Commenter- love your in-depth posts about different micro-display technologies On a slightly different topic- I was wondering what your opinion was of the Raon Tech R-vista 50 LCOS micro-display (searched and saw that you hadn’t discussed it). It has a 40 degree FOV, 1280×720 but it is much cheaper than other displays. The reason bringing it up is because it seems to be on par with other displays in the major tech categories but is actually affordable… How would you rate it compared to others ie. the ODG LCOS in their R7 ?

    • KarlG says:

      I have heard of Raon LCOS (from Korea) being used in some near eye displays, but have not had the opportunity to seriously evaluate it. I have not personally seen the R-Vista 50 optics, so my comments below are based on what I see of the design and my general knowledge of display systems. From the pictures and videos they appears to be using a “birdbath” type design similar to newer ODG’s R8/R9 but with a polarizing beamspitter as it uses LCOS.

      It is really hard to know without comparing them side by side. The ODG R7 is also an LCOS design and appears looks to use refractory optics (inside the top) with a single pass through a simple plate beamsplitter. The biggest downside to the birdbath is that with two passes through the beam splitter (reflective and transmissive) there are a lot more changes for reflections. All things being equal, the ODG R7 might have better image quality, at least in terms of less reflections, but it depends on the optics they used. The R7 will for the same illumination be brighter AND will let through much more of the “real world” light. The R7 only has a beamsplitter between the real world and the eye, where the R-Vista 50 has a beamsplitter and the spherical combiner which then throws away part of both the display light and the real world light. All in all I would expect the R-Vista 50 optics to cost less but the R7 to have better display and view of the real world image quality. I suspect this is why did the ODG built the R7 this way rather than use a birdbath.

      BTW: Comparing to the ODG’s R8/9 is using OLED, ODG is likely used a non-polarizing beam splitter but is otherwise similar in terms of the optical path for the R-Vista 50. Since LCOS works with/requires polarized light the polarizing birdbath design does add as much loss as you would get with a OLED not using polarized light -or else you have to take the hit on polarizing the OLED’s light). LCOS has a big advantage in NITs (candelas per meter squared) because the LED illumination is highly collimated light and LEDs can be very bright; you can easily get more than 10,000 NITs out of an LCOS/LED system where even 2,000 NITS is more than pushing OLED microdisplays. Because the LCOS display is so much brighter they can allow the spherical beamsplitter in the LCOS system to much more transparent than the ODG system. LCOS will be field sequential color which will cause color breakup occasionally. I really don’t thing the “birdbath” design works well for a see-through OLED application.

      • Hawkeye says:

        Hi Karl,
        Thanks for this response- I will have to see just how much better quality/brightness the ODG has compared to the rvista50 (& whether that quality difference is worth the price differential in optics)

        One question I had was how the lens color/gradient level effects the display optics…i.e. odg cleverly chose a tinted lens (probably to hide some of their hardware and make the smartglass look less bulky). But does a tinted lens effect how the image is displayed vs a pure clear, white lens? Also what would the effect be if other gradients/colors are chosen (like a red sunglass lens). I am interested to know whether/how these different lens types affects how the display looks at all..

        https://www.framesdirect.com/landing/a/lens-color-guide.html?affiliate=73

        • KarlG says:

          The ODG lenses are not so much tinted as partial mirrors. Typically this is done with aluminum alloys. They are not trying to “hide” the optic’s but rather all the light blocking is to cause light to reflect. With a partial mirror, if you pass X% then you reflect 1-X%. Then on top of this ODG has an option outer lens which may be tinted, but you want this to be a very neutral/non-coloring tint otherwise you tint the “real world.”

          • Hawkeye says:

            Ok thanks- yes I was referring to the outer lens..because there are options to choose what colors you want the outer lens to be but was confused because if we chose a red outer lens, would that distort the display…though I wonder just how how much the different colored outer lenses would distort the view…

          • KarlG says:

            The Human Visual System has does quite a bit of “automatic color balance”. The eye’s sense of color is highly relative and it is well known that if you wear color tinted glasses, it will affect your sense of color both when wearing them and if you wear them long enough when you take them off. So wearing color tinted lenses will definitely affect the color perception. With a combined image, it will also directly tint the colors. So you would have direct/”objective tinting” of the display image and you would have “subjective color changing” as the human visual system adjusts to the color tinted background. I don’t known how to quantify the effect without human studies (it is at least partially a “subjective” effect).

            There are many articles on how the eye sees color relatively, in particular look at the color cubes in the two articles below:
            http://brainden.com/color-illusions.htm
            http://retouchingacademy.com/the-fundamentals-of-color-theory/

            Also, do a search on: “Wearing color tinted glasses affect on color perception”

  6. […] discussed in Part 1, the basic concept of foveated display in theory should work to provide high angular resolution with […]

  7. NIKOLAOS BERATLIS says:

    I would like to add a comment on a previous post (Near Eye AR/VR and HUD Metrics For Resolution, FOV, Brightness, and Eyebox/Pupil) but there is no comment box. I added a comment a few months ago and I want to reply but there is no option. Can you help me?

Leave a Reply

Your email address will not be published. Required fields are marked *