Realistic virtual vision with dynamic foveated rendering

Tobii Spotlight Technology

Over the past two decades, Tobii has become known as the world’s leading eye tracking company, offering eye tracking and user sensing technologies for a range of devices and use cases. Today, Tobii’s eye tracking technology can be found in laptop computers, VR Headsets, AR glasses, innovative healthcare solutions, assistive devices, and many more applications and products. Recently, Tobii has been working on a set of technologies and capabilities related to foveation — what we call Tobii Spotlight Technology. This blog seeks to explore the current benefits made possible via foveation, with an emphasis on recent benchmarks Tobii conducted.

Foveation

Let’s start with a little background. Foveation is a computational process for image rendering that mimics the way human vision works. Dynamic foveated rendering (DFR) employs eye tracking to focus processing and bandwidth usage on areas of the image where high resolution is required — the foveal region where the subject is looking — and offers lower resolution toward the periphery. Optimizing the image in this way simulates human vision while reducing demands for bandwidth and processing— cutting latency and improving response times.

Tobii Spotlight Technology is an advanced eye tracking solution, specialized for foveation. It delivers accurate, low latency tracking of the user’s gaze in real-time, enabling superior dynamic foveation that simulates realistic, human vision. By easing the strain on the GPU and improving overall rendering speed, Tobii Spotlight Technology offers improvements to the overall experience of a VR application:

• Better image quality — GPU savings enable high-resolution images to be rendered in real-time in the foveal region. DFR gains are greater for higher-resolution headsets since DFR can reduce the shading load significantly.
• Higher and smoother frame rates — frame rate defines the performance and presence of a VR application. A drop-in frame rate (typically in cases of complex shader rendering) results in judder, which is noticeable to the user and can even result in motion sickness or nausea. DFR helps to maintain smoother frame rates, and its savings are the greatest in the heaviest parts of a scene with the most complex shading.
• Enhanced graphics — VR applications are highly demanding when it comes to rendering compared to a standard PC game. To maintain the same level of performance, developers have traditionally optimized the application itself, which can result in a reduction of the quality of the scene or disabling some real-time effects. DFR enables more complex and realistic shading without increasing GPU load, enabling developers to include higher quality settings in their applications without impacting performance.
• Power savings — GPU load reduction also can enable potential power savings for laptops, headsets and other battery powered devices.

DFR with NVIDIA variable rate shading (VRS)

Today, many developers associate foveated rendering with NVIDIA VRS — a rendering technique that allows finer control over shading density and true super-sampling. With VRS, developers can choose to improve visual quality, reduce GPU costs, or even balance a little of both.

VRS applies varying amounts of processing power to different areas of the image. The technology works by altering the number of pixels that are processed by a single-pixel shader operation. These operations can now be applied to blocks of pixels, allowing applications to effectively vary the shading quality in different areas of the screen.

For maximum effect, VRS can also be paired with eye tracking to match optimal rendering quality to the user’s gaze. NVIDIA VRS custom patterns allow developers to optimize the shading density based on the foveal region. The smaller the foveal region, the larger the gain from GPU savings. The foveal region size is determined by:

• Effective field of view of the display — Angular extent for foveation does not vary with field of view. In other words, the percentage of the display which must be high-quality decreases as the field of view increases.
• Image artifacts produced by the foveated rendering technique.
• Latency in the presentation of the tracked foveation.
• Accuracy and robustness of the eye tracking system.
• User sensitivity to artifacts.

DFR with VRS foveation gives the user the most optimized custom pattern when combined with a low latency eye tracking signal. This maximizes the benefit of enabling VRS in the application since the shading rate can be reduced considerably, which improves the overall performance of the application and can enable better image quality with super-sampling in the foveal region.

Adapting to the user

Not all eye tracking signals are created equal. Latency, frequency, accuracy, and noise are all obvious contributors to foveal region size. Perhaps less obvious is the impact of signal reliability, population coverage, angular accuracy falloff, and eye tracking signal artifacts.

Additionally, the ability to track gaze varies across the population. Some people are easy to track, while some people are not trackable at all. A user who normally is easily trackable can become less so through tiredness, dehydration, and illness. Effective foveation should account for this variability.

Tobii has put significant investment into foveation-specific signal research, including the development of specialized foveated rendering tracking signals which reduce or eliminate some of the damaging signal artifacts which can be present on non-specialized signals.

Benchmarks

Recently, Tobii ran a variety of benchmarks comparing the performance benefits of both fixed and dynamic foveated rendering. Fixed foveated rendering (FFR) is a technique that assumes a forward viewing direction and limits the rendering costs of display areas which will not be clearly visible in the headset, mainly in the lens distortion region. On the other hand, DFR moves the foveal region wherever the user is looking, which further reduces the foveal region size.

The results (when compared to no foveation) consistently illustrate:

• DFR results in an average reduction in the GPU shading load of about 57% while running tests with locked 6dof (ensuring you have a constant frame for each pre-set) on various portions of the scene. The reduction is higher for more pixel-intensive portions of the scene.
• DFR reduces GPU load so dramatically that it makes resolutions of 8K and beyond possible on future headsets.
• DFR enables developers to add complex shaders and effects for graphical improvements while maintaining high performance.

While running tests on Vive Pro Eye, we optimized the foveation parameters for variable rate shading to achieve a shading rate of 16% for DFR. For fixed, we have configured the shading density to 40%, which works best for the Vive Pro Eye headset parameters and is not noticeable to the user in the periphery when enabled. The screen is divided into a few regions going from foveal (where the user is looking), mid (which is a transition from foveal to periphery), and periphery (region is optimized for maximum gain). In the figure below, the colored overlay shows the regions for both FFR and DFR with different size and shape parameters. The color coding is a gradient based on the log of density where blue = 1 sample, purple = 1/4 samples, purple red = 1/8 samples, red = 1/16 samples, and black = culled.

Showdown VR is a cinematic experience where the scene has variable complexity in different parts of the scene. This allows us to sample the GPU shading load for highest and lowest and compare them across Full, Fixed and Dynamic rendering modes. In the figure below, we observe that for some parts of the scene the shading load increases a lot for full rendering (check the second explosion). DFR results for shading load is relatively consistent with fewer spikes for even the heaviest part of scene making it a much smoother experience. For the test below, we consider a slight modification in the scene — the scene has been super sampled 3x times to increase the resolution for better image quality. Here, we observe further reduction in GPU shading load with DFR of around 74.59% even though the overall shading load for the scene has increased.

Next-gen headsets are aiming for higher resolution and larger field of view that demands more and more pixels to be rendered on screen. In the graph below, for a normal VR application we observe the pixels rendered increase exponentially. Comparing that with DFR there is a significant drop for higher resolution headsets going towards 8K and beyond. This directly affects the GPU shading load and hence the savings with DFR increase with increase in headset resolutions. This is also applicable for the applications supersampling for the existing-gen headsets.

Developers can also choose to use these performance enhancing benefits to greatly increase the visual effects in the scene, while retaining smooth frame rates. This enables the developers and designers to push boundaries on visuals and complex shaders. In the figure below, we have tested with some modifications on Showdown VR to improve shader and lighting effects with no additional cost on load.

Conclusion

For a VR application, having a consistently high performance is very important. DFR enables applications to maintain this high performance and additionally supports higher resolutions and better visual effects. When Tobii Spotlight Technology is combined with technologies such as NVIDIA’s variable rate shading, we get maximum benefit from foveation with reduction in GPU shading load. Going beyond just rendering, Tobii Spotlight Technology has several applications for dynamic foveation like foveated transport and streaming.

More on Tobii Spotlight Technology

Learn more on Tobii Spotlight Technology and other dynamic foveation applications.

Also check additional information in the Siggraph panel discussion. 

Want to learn more about foveated rendering and Tobii Spotlight Technology?

Download our eBook Eye tracking and dynamic foveated rendering — technologies that solve the VR challenge.