Archive for June 28, 2017

Texas Instruments 99/4A and TMS9918 History

A little break from displays today to go back into my deep dark history. For my first 20 years in the industry, I was an I.C. designer and led led the architecture a number of CPUs and graphics devices.

I got a “shout out” of sorts in an IEEE article on the 99/4 computer by Wally Rhines, CEO of Mentor, about my work on the TMS9918 graphics unit which was my first design (started in 1977). Contrary to what the article states, I was NOT the only designer, back then it took 7 “whole engineers” (quite a few less than today) to design a graphics chip and I was the youngest person on the program. I think the 9918 took less than 1 year from raw concept to chip. Wally gave things from his perspective as a high level manager and he may be off in some details.

The 9918 coined the word “Sprites” and was used in the TI 99/4A, Colecovision, and the MSX computer in Japan. It was the first consumer chip to directly interface to DRAMs (I came up with the drive scheme). Pete Macourek and I figured out how to make the make the sprites work and then I did all the Sprite logic and control design.

A “Z80-like” register file compatible superset clone of the 9918 was used in both the Nintendo (Nintendo was a software developer for Coleco) and Sega Game systems among others.

After working on the TMS9918, I led the architecture and early logic design of the TMS9995 (which resulted in my spending 6 months in Bedford England) which is also mentioned in Wally’s article. If the TI Home Computer was not cancelled, I would have had a major part in the design of both the CPU and the Graphics chip on the 99/8 and 99/2.

Back in 1992 in the I was interviewed about the home computer in the days of BBS Bulletin Boards. This was only about 10 years after the events so they were more fresh in my mind. At the time of the 1992 interview, I was working on the first fully programmable media processor (and alluded to it in the interview) that integrated 4 DSP CPUs and a RISC processor on a single device (call the TMS320C80 or MVP). Another “little thing” that came out of that program was the Synchronous DRAM. You see I had designed the DRAM interface on the 9918 and the TMS340 graphics processor family and had worked on the Video DRAM (predecessor of today’s Graphics DRAMs) and was tired of screwing with the analog interface of DRAMs; so in a nutshell, I worked with TI’s memory group to define the first SDRAM (one of the patents can be found here). The 320C80 was the first processor to directly interface with SDRAM because it was co-designed with them.

For anyone interest, I wrote some more about my TI Home Computer and 9918 history on this blog back in the early days of this blog in 2011.

Varjo Foveated Display (Part 1)

Introduction

The startup Varjo recently announced and did a large number of interviews with the technical press about their Foveated Display (FD) Technology. I’m going to break this article into multiple parts, as currently planned, the first part will discuss the concept and the need for and part 2 will discuss how well I think it will work.

How It Is Suppose to Work

Varjo’s basic concept is relatively simple (see figure at left – click on it to pop it out). Varjo optically combines a OLED microdisplay with small pixels to give high angular resolution over a small area (what they call the “foveated display“), with a larger OLED display to give low angular resolution over a large area (what they call the “context display“). By eye tracking (not done in the current prototype), the foveated display is optically moved to be in the center of the person’s vision by tilting the beam splitter. Varjo says they have thought of and are patenting other ways of optically combining and moving the foveated image other than a beam splitter.

The beam splitter is likely just a partially silvered mirror. It could be 50/50 or some other ratio to match the brightness of the large and microdisplay OLED. This type of combining is very old and well understood. They likely will blend/fade-in the image in the rectangular boarder where the two display images meet.

The figure above is based on a sketch by Urho Konttori, CEO of Varjo in a video interview with Robert Scoble combined with pictures of the prototype in Ubergismo (see below), plus answers to some questions I posed to Varjo. It is roughly drawn to scale based on the available information. The only thing I am not sure about is the “microdisplay lens” which was shown but not described in the Scoble interview. This lens(es) may or may not be necessary based on the distance of the microdisplay from the beam combiner and could be used to help make the microdisplay pixels appear smaller or larger. If the optical path though the beam combiner to large OLED (in the prototype from an Oculus headset) would equal the path from to the microdisplay via reflecting off the combiner, then the microdisplay lens would not be necessary. Based on my scale drawing and looking at the prototype photographs it would be close to not needing the lens.

Varjo is likely using either an eMagin OLED microdisplay with a 9.3 micron pixel pitch or a Sony OLED microdisplay with a 8.7 micron pixel pitch. The Oculus headset OLED has ~55.7 micron pixel pitch. It does not look from the configuration like the microdisplay image will be magnified or shrunk significantly relative to the larger OLED. Making this assumption, the microdisplay image is about 55.7/9 = ~6.2 time smaller linearly or effectively ~38 times the pixels per unit area. This ~38 times the area means effectively 38 times the pixels over the large OLED alone.

The good thing about this configuration is that it is very simple and straightforward and is a classically simple way to combine two image, at least that is the way it looks. But the devil is often in the details, particularly in what the prototype is not doing.

Current Varjo Prototype Does Not Track the Eye

The Varjo “prototype” (picture at left from is from Ubergismo) is more of a concept demonstrator in that it does not demonstrate moving the high resolution image with eye tracking. The current unit is based on a modified Oculus headset (obvious from the picture, see red oval I added to the picture). They are using the two Oculus larger OLED displays the context (wide FOV) image and have added an OLED microdisplay per eye for the foveated display. In this prototype, they have a static beam splitter to combine the two images. In the prototype, the location of the high resolution part of the image is fixed/static and requires that the user look straight ahead to get the foveated effect. While eye tracking is well understood, it is not clear how successfully they can make the high resolution inset image track the eye and whether the a human will notice the boundary (I will save the rest of this discussion for part 2).

Foveated Displays Raison D’être

Near eye display resolution is improving at a very slow rate and is unlikely to dramatically improve. People quoting “Moore’s Law” applying to display devices are simply either dishonest or don’t understand the problems. Microdisplays (on I.C.s) are already being limited by the physics of diffraction as their pixels (or color sub-pixels) get withing 5 times the wavelengths of visible light. The cost of making microdisplays bigger to support more pixels drives the cost up dramatically and this not rapidly improving; thus high resolution microdisplays are still and will remain very expensive.

Direct view display technologies while they have become very good at making large high resolution display, they can’t be make small enough for lightweight head-mounted displays with high angular resolution. As I discussed the Gap in Pixel Sizes (and for reference, I have included the chart from that article) which I published before I heard of Varjo, microdisplays enable high angular resolution but small FOV while adapted direct view display support low angular resolution with a wide FOV. I was already planning on explaining why Foveated Displays are the only way in the foreseeable future to support high angular resolution with a wide FOV: So from my perspective, Varjo’s announcement was timely.

Foveated Displays In Theory Should Work

It is well known that the human eye’s resolution falls off considerably from the high resolution fovea/center vision to the peripheral vision (see the typical graph at right). I should caution, that this is for a still image and that the human visual system is not this simple; in particular it has sensitivity to motion that this graph can’t capture.

It has been well proven by many research groups that if you can track the eye and provide variable resolution the eye cannot tell the difference from a high resolution display (a search for “Foveated” will turn up many references and videos). The primary use today is with Foveated Rendering to greatly reduce the computational requirements of VR environment.

Varjo is trying to exploit the same foveated effect to gives effectively very high resolution from two (per eye) much lower resolution displays. In theory, it could work but will in in practice?  In fact, the idea of a “Foveated Display” is not new. Magic Leap discussed it in their patents with a fiber scanning display. Personally, the idea seems to come up a lot in “casual discussions” on the limits of display resolution. The key question becomes: Is Varjo’s approach going to be practical and will it work well?

Obvious Issues With Varjo’s Foveated Display

The main lens (nearest the eye) is designed to bring the large OLED in focus like most of today’s VR headsets. And the first obvious issues is that the lens in a typical VR headset is designed resolve pixels that are more than 6 times smaller. Typical VR headsets lenses are, well . . ., cheap crap with horrible image quality. To some degree, they are deliberately blurring/bad to try and hide the screen door effect of the highly magnified large display. But the Varjo headset would need vastly better, and much more expensive, and likely larger and heavier optics for the foveated display; for example instead of using a simple cheap plastic lens, they may need a multiple element (multiple lenses) and perhaps made of glass.

The next issue is that of the tilting combiner and the way it moves the image. For simple up down movement of the foveated display’s image will follow a simple path up/down path, but if the 45 degree angle mirror tilts side to side the center of the image will follow an elliptical path and rotate making it more difficult to align with the context image.

I would also be very concerned about the focus of the image as the mirror tilts through of the range as the path lengths from the microdisplay to the main optics changes both to the center (which might be fixable by complex movement of the beam splitter) and the corners (which may be much more difficult to solve).

Then there is the general issue of will the user be able to detect the blend point between the foveated and context displays. They have to map the rotated foveated image match the context display which will loose (per Nyquist re-sampling) about 1/2 the resolution of the foveated image. While they will likely try cross-fade between the foveated and context display, I am concerned (to be addressed in more detail in part 2) that the visible/human detectable particularly when things move (the eye is very sensitive to movement).

What About Vergence/Accommodation (VAC)?

The optical configuration of Varjo’s Foveated Display is somewhat similar to that of Oculus’s VAC display. Both leverage a beam splitter, but then how would you do VAC with a Foveated Display?

In my opinion, solving the resolution with wide field of view is a more important/fundamentally necessary problem to solve that VAC at the moment. It is not that VAC is not a real issue, but if you don’t have resolution with wide FOV, then VAC is not really necessary?

At the same time, this points out how far away headsets that “solve all the world’s problems” are from production. If you believe that high resolution with a wide field of view that also address VAC, you may be in for a many decades wait.

Does Varjo Have a Practical Foveated Display Solution?

So the problem with display resolution/FOV growth is real and in theory a foveated display could address this issue. But has Varjo solved it? At this point, I am not convinced, and I will try and work though some numbers and more detail reasoning in part 2.

VAC By Oculus and Microsoft . . . Everywhere and Nowhere

Technically Interesting New Papers At Siggraph 2017

Both Oculus (Facebook) and Microsoft’s are presenting interesting technical research  papers at Siggraph 2017 (July 30th to August 3rd) that deal with Vergence/Accommodation (VAC).  Both have web pages (Oculus link and Microsoft link) with links to relatively easy to follow videos and the papers. But readers should take to heed the words on the Microsoft Page (which I think is applicable to both): “Note that this Microsoft Research publication is not necessarily indicative of any Microsoft product roadmap, but relates to basic research around holographic displays.” I can’t hope to try and get into all the technical details here, but they both have a lot well explained information with figures and for those that are interested, you can still learn a lot from them even if you have to skip over some of the heavy duty math. One other interesting thing is that both Oculus and Microsoft used phase controlled LCOS microdisplays at the heart of their technologies.

Briefly, VAC is the problem with stereoscopic 3-D where the apparent focus of objects does not agree with were they seem to appear with binocular vision. This problem can cause visual discomfort and headaches. This year I have been talking a lot about VAC thanks first to Magic Leap (ML article) and more recently Avegant (Avegant VAC article ) making big deals about and both raising a lot of money (Magic Leap over $1B) as a result. But least you think Magic Leap and Avegant are the only ones, there are dozens of research groups over the last decade working on VAC. Included in that number is Nvidia with a light field approach that they presented a paper in 2013 also at Siggraph (The 2013 Nvidia Paper with links embedded at the bottom of the Abstract to more information and a video)

The Oculus paper has a wealth of background/education information about VAC and figures that help explain the concepts. In many ways it is a great tutorial. They also have a very lengthy set of references that among other things confirm how many different groups have worked on VAC and this is only a partial list. I also recommend papers and videos on VAC by Gordon Wetzstein of Stanford. There is so much activity that I put “Everywhere” in the title.

I particularly liked Oculus’s Fig. 2 which is copied at the top of this article (they have several other very good figures as well as their video). They show the major classes of VAC, from a) do nothing, b) change focus (perhaps based on eye tracking), to c) Multifocal which is what I think Magic Leap and Avegant are doing, to d)&e) Oculus’s “focal surfaces(s), to f) light fields (ex. Nvidia’s 2013 paper). But light fields are in a way a short cut compared to real/true holograms which is what Microsoft’s 2017 paper is addressing (not shown in the table above but discussed in Oculus’s paper and video).

I put the “real” in front of the work “hologram” because confusingly Microsoft, for what appears to be marketing purposes, has chosen to call stereoscopic merged reality objects “holograms” which scientifically they are not. Thanks to Microsoft’s marketing clout and others choosing “if you can’t beat them joint them” in using the term, we now have the problem of what to call real/true holograms as discussed in Microsoft’s 2017 Siggraph paper.

High Level Conceptually:
  • Light Fields are a way to realize many of the effects of holograms such such as VAC and being able to see around objects. But light fields have piece-wise discontinuities. They can only reduce the discontinuities by massively trading off resolution; thus they need massive amounts of processing and native display resolution for a given visual resolution. Most of the processing and display resolution never makes it do the eye as based on where the eye is looking and focused, all but a small part of the generated image information is never seen. The redundancy with light fields tends to grow with a square law (X and Y).
  • Focus planes in effect try and cut down the Light Field square law redundancy problem by having the image redundancy grow linearly. They need multiple planes and then rely on your eye to do the blending between planes. Still the individual planes on “flat” and with a large continuous surface there would be discontinuities at the point where it would have to change planes (imagine a road going off in the distance).
  • Oculus Surfaces are in essence and improvement on focus planes where the surfaces try to conform more to the depth in the image and reduce the discontinuities. One could then argue whether it would be better to have more simple focus planes or fewer Focus Surfaces.
  • Holograms have at least an “n-cube” problem as they conceptually capture/display the image in X, Y, and Z. As the resolution increases the complexity grows extremely fast. Light fields have sometimes been described as “Quantized Holograms” at they put a finite limit on the computational and image content growth.
Oculus’s Focus Surface Approach

In a nutshell, Oculus is using an eMagin OLED to generate the image and a Jasper Display Phase Shift LCOS device to generate a “focus surface”. The focus changes focus continuously-gradually, and not on a per-pixel basis, which is why they call is a “surface”.  The figure on the right (taken from their video) shows the basic concept of a “focus surface” and how the surface roughly tracks the image depth. The paper (and video) go on to  discuss how having more than one surface and how the distance approximation “error” would compare with multi-focus planes (such as Magic Leap and Avegant).

While the hardware diagram above would suggest something that would fit in a headset, it is still at the optical breadboard stage. Even using microdisplays, it is a lot to put on a person’s head. Not to mention the cost of having in effect two displays (the LCOS one controlling the focus surface) plus all the additional optics. Below is a picture of the optical breadboard.

Microsoft (True This Time) Holograms

While Oculus’s hardware looks like something that could fit in a headset someday, Microsoft is much more of a research concept, although they did show a compact AR Prototype “glasses” (shown at right) that had a small subset of the capability of the larger optical breadboard.

Microsoft’s optical breadboard setup could support either Wide FOV or Multi-Focal (VAC) but not both at the same time (see picture below). Like other real time hologram approaches (and used by Oculus in their focal surface approach), Microsoft uses a Phase LCOS device.The Microsoft paper goes into some of the interesting things that can be done with holograms including correcting for aberrations in the optics and/or a person’s vision.

In many ways Holograms ultimate end game in display technology where comparatively everything else in with VAC is a hack/shortcut/simplification to avoid the massive computations and hardware complexities/difficulties of implementing real time holograms.

Resolution/Image Quality – Not So Much

The image quality in the Oculus Surface paper is by their admission very low both in terms of resolution and contrast. As they freely admit, it is a research prototype and not meant to be a product.

Some of these limitations are the nature of making a one-off experiment as the article points out but some of the issues may be more fundamental physics. One thing that concerns me (and pointed out in the article) in the Oculus design is that they have to pass all three colors through the same LC material and the LC’s behavior varies with wavelength. These problems would become more significant as resolution increases. I will give the Oculus paper props for both for is level of information and candor about many of the issues; it really is a very well done paper if you are interested in this subject.

It is harder to get at the resolution and image quality aspects of the the Microsoft Hologram paper as they show little images from different configurations. They can sort of move the problems around with Holograms; they can tune them and even the physical configuration for image quality, pupil size, or depth accommodation, but not all at the same time. Digital/real-time holograms can do some rather amazing things as as the Microsoft paper demonstrates but but they are still inordinately expensive both to compute and display and the image quality is inferior to more conventional methods. Solving for image quality (resolution/contrast), pupil/eyebox size, and VAC/image depth simultaneously makes the problems/cost tend to take off exponentially.

Don’t Expect to See These In Stores for Decades, If Ever

One has to realize that these are research projects going for some kind of bragging rights in showing the technical prowess, which both Oculus and Microsoft do impressively in their own ways. Note the Nvidia Light Field paper was presented at Siggraph 2013 years ago and supporting decent resolution with Light Fields is still a very far off dream. If their companies thought these concepts were even remotely practical and only a few years away, the companies would have kept them deep dark secrets. These are likely seen by their companies as so out in the future that there is no threat to letting their competition see what they are doing.

The Oculus Surface approach is conceptually better on a “per plane” than the “focus planes” VAC approaches, but then you have to ask are more simple planes better overall and/or less expensive? At a practical level I think the Oculus Surface would be more expensive and I would expect the image quality to be considerably worse. At best, the Oculus Surface would be a stop-gap improvement.

Real time high resolution holograms that will compete on image quality would seem to be even further out in time. This is why there are so many companies/researchers looking at short cuts to VAC with things like focus planes.

VAC in Context – Nowhere

VAC has been a known issue for a long time with companies and researchers working in head mounted displays. Magic Leap’s $1+B funding and their talk about VAC made it a cause célèbre in AR/VR and appears to have caused a number of projects to come out from behind closed doors (for V.C. funding or just bragging rights).

Yes, VAC is a real issue/problem particularly/only when 3-D stereoscopic objects appear to be closer than about 2 meters (6 feet) away. It causes not only perceptual problems, but can cause headaches and make people sick. Thus you have companies and researchers looking for solutions.

The problem IMO is that VAC is would be say about 20th (to pick a number) on my list of serious problems facing AR/VR. Much higher on the list are based image quality, ergonomic (weight distribution), power, and computing problems. Every VAC solution comes at some expense in terms of image quality (resolution/contrast/chromatic-abberations/etc).

Fundamentally, if you eye can pick what it focuses on, then there has to be a lot of redundant information presented to the eye that it will discard (not notice) as it focuses on what it does see. This translates into image information that must be displayed (but not seen), processing computations that are thrown away, and electrical power being consumed for image content that is not used.

I’m Conflicted

So I am conflicted. As a technologist, I find the work in VAC and beyond (Holograms address much more than VAC) fascinating. Both the Oculus and Microsoft articles are interesting and can be largely understood by someone without a PhD in the subject.

But in the end I am much more interested in technology that can reach a sizable market and on that score I don’t understand all the fuss about VAC.  I guess we will have to wait and see if Magic Leap changes the world or is another Segway or worse Theranos; you might be able to tell which way I am leaning based on what I understand.

Today, the image quality of headsets is pretty poor when compared to say direct view TVs and Monitors, the angular resolution (particularly of VR) is poor, the ergonomics are for the most part abysmal, and if you are going to wireless, the batteries are both too heavy and have too short a life. Anything that is done to address VAC makes these more basic problems not just a little worse, but much worse.

Near Eye Displays (NEDs): Gaps In Pixel Sizes

I get a lot of questions to the effect of “what is the best technology for a near eye display (NED).” There really is no “best” as every technology has its strengths and weaknesses. I plan to right a few articles on this subject as it is way too big for a single article.

Update 2017-06-09I added the Sony Z5 Premium 4K Cell Phone size LCD to the table. Their “pixel” is about 71% the linear dimension of the Samsung S8 or about half the area but still much larger than any of the microdisplay pixels. But one thing I should add is that most cell phone makers are “cheating” on what they call a pixel. The Sony Z5 Premium’s “pixel” really only has 2/3rds of an R, G, and B per pixel it counts. It also has them in a strange 4 pixel zigzag that causes beat frequency artifacts when displaying full resolution 4K content (GSMARENA’s Close Up Pixtures show of the Z5 Premium fails the show the full resolution in both directions). Note similarly Samsung goes with RGBG type patterns that only have 2/3rd the full pixels in the way they count resolution as well. These “tricks in counting are OK when viewed with the naked eye at beyond 300 “pixels” per inch, but become more problematical/dubious when used with optics to support VR. 

Today I want to start with the issue of pixel size as shown in the table at the top (you may want to pop the table out into a separate window as you follow this article). To give some context, I have also included a few major direct view categories of displays as well. I have grouped the technologies into the colored bands in the table. I have given the pixel pitch (distance between pixel centers) as well as the pixel area (the square of the pixel pitch assuming square pixels. Then to give some context for comparison I have compared the pitch and area relative to a 4.27-micron (µm) pixel pitch which is about the smallest being made in large volume. Finally there are columns showing how big the pixel would be in arcminutes when view from 25cm (250mm =~9.84inches) which is the commonly accepted near focus point. Finally there is a column showing how much the pixel would have to be magnified to equal 1-arcminute at 25cm which gives some idea about the optics required.

In the table, I tried to use smallest available pixel in a given technology that was being produced with the exception of “micro-iLED” for which I could not get solid information (thus the “?”). In the case of LCOS, the smallest field sequential color (FSC) pixel I know of is the 4.27µm one by my old company Syndiant used in their new 1080p device. For the OLED, I used the eMagin 9.3 pixel and for the DLP, their 5.4 micron pico pixel. I used the LCOS/smallest pixel as the baseline to give some relative comparisons.

One thing that jumps out in the table are the fairly large gaps in pixel sizes between the microdisplays versus the other technologies. For example you can fit over 100 4.27µm LCOS pixels in the area of a single Samsung S8 OLED pixel or 170 LCOS pixels in the area of a the pixel used in the Oculus CV1. Or to be more extreme you can fit over 5,500 LCOS pixels in one pixel of a 55-inch TV pixel.

Big Gap In Near Eye Displays (NEDs)

The main point of comparison for today are the microdisplay pixels which range from about 4.27µm to about 9.6µm in pitch to the direct view OLED and LCD displays in 40µm to 60µm that have been adapted with optics to be used in VR headsets (NEDs). Roughly we are looking at one order of magnitude in pixel pitch and two orders of magnitude in area. Perhaps the most direct comparison is the microdisplay OLED pixel at 9.3 microns versus the Samsung S8 at 4.8X linear and a 23x area difference.

So why is there this huge gap? It comes down to making the active matrix array circuitry to drive the technology. Microdisplays are made on semiconductor integrated circuits while direct view displays are made on glass and plastic substrates using comparatively huge and not very good transistor. The table below based on one in an article from 2006 by Mingxia Gu while at Kent State University (it is a little out of date, but gives lists the various transistors used in display devices).

The difference in transistors largely explains the gap. With the microdisplays using transistors made in I.C. fabs whereas direct view displays fabricate their larger and less conductive transistors on top of glass or plastic substrates at much lower temperatures.

Microdisplays

Within the world of I.C.’s, microdisplays used very old/large transistors often using nearly obsolete semiconductor processes. This is both an effort to keep the cost down and the fact that most display technologies need higher voltages than would be supported by smaller transistor sizes.

There are both display physics and optical diffraction reasons which limit making microdisplay pixels much smaller than 4µm. Additionally, as the pixel size gets below about 6 microns, the optical cost of enlarging the pixel to be seen by the human start to escalate so headset optics makers want 6+ micron pixels which are much more expensive to make. To a first order, microdisplay costs in volume are a function of area of the display so smaller pixels means less expensive devices for the same resolution.

The problem for microdisplays is even using old I.C. fabs, the cost per square millimeter is extremely high compared to TFT on glass/plastic, and yields drop as the size of the device grows so doubling the pixel pitch could result in an 8X or more increase in cost. While is sounds good to be using old/depreciated I.C. fabs, it may also mean they may not have the best/newest/highest yielding equipment or worse yet, they close down the facilities as being obsolete.

The net result is that microdisplays are no where near cost competitive with “re-purposed” cell phone technology for VR if you don’t care about size and weight. They are the only way to do a small lightweight headsets and really the only way to do AR/see through displays (save the huge Meta 2 bug-eye bubble).

I hope to pick up this subject more in some future articles (as each display type could be a long article in and of itself. But for now, I want to get onto the VR systems with larger flat panels.

Direct View Displays Adapted for VR

Direct View VR (ex. Oculus, HTC Vive, and Google Cardboard) have leveraged direct view display technologies developed for cell phones. They then put simple optics in front of the display so that people can focus the image when the display is put so near the eye.

The accepted standard for human “near vision” is 25cm/250mm/9.84-inches. This is about as close as a person can focus and is used for comparing effective magnification. With simple (single/few lens) optics you are not so much making the image bigger per say, but rather moving the display closer to the eye and then using the optics to enable the eye to focus. A typical headset uses a roughly 40mm focal length lens and then put the display at the focal lens or less (e.g. 40mm or less) from the lens.  Putting the display at the focal length of the lens makes the image focus at infinity/far away.

Without getting into all the math (which can be found on the web) the result is that with a 40mm focal length nets an angular magnification (relative to viewing at 25cm) of about 6X. So for example looking back at the table at the top, the Oculus pixel (similar in size to the HTC Vive) which would be about 0.77 arcminutes at 25cm end up appearing to cover about 4.7 arcminutes (which are VERY large/chunky pixels) and about a 95 degree FOV (depends on how close the eye gets to the lens — for a great explanation of this subject and other optical issues with the Oculus CV1 and HTC Vive see this Doc-Ok.org article).

Improving VR Resolution  – Series of Roadblocks

For reference, 1 arcminute per pixel is consider near the limit of human vision and most “good resolution” devices try to be under 2 arcminutes per pixel and preferably under 1.5. So let’s say we want to keep the ~95 FOV but improve the angular resolution by 3x linearly to about 1.5 arcminutes, we have several (bad) options:

  1. Get someone to make a pixel that is 3X smaller linearly or 9X smaller in area. But nobody makes a pixel this size that can support about 3,000 pixels on a side. A microdisplay (I.C. based) will cost a fortune (like over $10,000/eye if it could be made at all) and nobody makes transistors that a cheap and compatible with displays that are small enough. But let’s for a second assume someone figures out a cost effective display, then you have the problem that you need optics that can support this resolution and not the cheap low resolution optics with terrible chroma aberrations, god rays, and astigmatism that you can get away with 4.7 arcminute pixels
  2. Use say the Samsung S8 pixel size (a little smaller) and make two 3K by 3K displays (one for each eye). Each display will be about 134mm or about 5.26 inches on a side and the width of the two displays plus the gap between them will end up at about 12 inches wide. So thing in terms of strapping an large iPad Pro in front of your face only, it now has to be about 100mm (~4 inches) in front of the optics (or about 2.5X as far away at on the current headsets). Hopefully you are starting to get the picture, this thing is going to huge and unwieldy and you will probably need shoulder bracing in addition to head straps. Not to mention that the displays will cost a small fortune along with the optics to go with them.
  3. Some combination of 1 and 2 above.
The Future Does Not Follow a Straight Path

I’m trying to outline above the top level issue (there are many more). Even if/when you solve the display cost/resolution problem, lurking behind that is a massive optical problem to sustain that resolution. These are the problems “straight line futurists” just don’t get; they assume everything will just keep improving at the same rate it has in the past not realizing they are starting to bump up against some very non-linear problems.

When I hear about “Moore’s Law” being applied to displays I just roll my eyes and say that they obviously don’t understand Moore’s Laws and the issued behind it (and why it kept slowing down over time). Back in November 2016 Oculus Chief Scientist Michael Abrash made some “bold predictions” that by 2021 we would have 4K (by 4K) per eye and 140 degree FOV with 2 arcminutes per pixel. He upped my example above by 1.33x more pixels and upped the FOV by almost 1.5X which introduces some serious optical challenges.

At times like this I like to point out the Super Sonic Transport or SST of the 1960’s. The SST seemed inevitable for passenger trave, after all in less than 50 years passenger aircraft when from nothing to the jet age; yet today, over 50 years later, passenger aircraft still fly at about the same speed. Oh by the way, in the 1960’s they were predicting that we would be vacationing on the moon by now and having regular fights to Mars (heck, we made it to the moon in less than 10 years). We certainly could have 4K by 4K displays per eye and 140 degree FOV by 2021 in a head mounted display (it could be done today if you don’t care how big it is), but expect it to be more like the cost of flying supersonic and not a consumer product.

It is easy to play arm chair futurist and assume “things will just happened because I want them to happen. The vastly harder part is to figure out how it can happen. I lived through I.C. development in the late 1970’s through the mid 1990’s so I “get” learning curves and rates of progress.

One More Thing – Micro-iLED

I included in the table at the top Micro Inorganic LEDs, also known as just Micro-LEDs (I’m using iLED to make it clear these are not OLEDs). They are getting a lot of attention lately, particularly after Apple bought LuxVue and Oculus bought InfiniLED. These essentially use very small “normal/conventional” LEDs that are mounted (essentially printed) on a substrate. The fundamental issue is that red requires a very different crystal from blue and green (and even they have different levels of impurities). So they have to make individual LEDs and then combine them (or maybe someday grow the dissimilar crystals on the common substrate).

The allure is that iLEDs have some optics properties that are superior to OLEDs. They have tighter color spectrum, more power efficient, can be driven much brighter, less issues with burn in, and in some cases have less diffuse (better collimated) light.

These Micro-iLEDs are being used in two ways, one to make very large displays by companies such as Sony, Samsung, and NanoLumens or supposedly very small displays (LuxVue and InfiniLED). I understand how the big display approach works, there is lots of room for the LED and these displays are very expensive per pixel.

With the small display approach, they seem to have to double issue of being able to cut very small LEDs and effectively “print” the LEDs on a TFT substrate similar to say OLEDs. What I don’t understand is how these are supposed to be smaller than say OLEDs which would seem to be at least as easy to make on similar TFT or similar transistor substrates. They don’t seem to “fit” in near eye, but maybe there is something I am missing at this point in time.