The processing capabilities of a headset, rendering requirements of an application, 5G network capacity, and edge-processing all have an impact on user experience.
Foveation technologies are a software solution for lowering processing and bandwidth needs, enabling the XR ecosystem meet the needs of graphically demanding applications and raise performance of existing devices and infrastructure.
By mimicking the natural degradation of perception, foveation makes it possible to limit high-definition rendering to the relatively small portion of an image where the user’s attention is focused. Fixed foveation, which assumes the user’s focus is on the center of the screen, provides some degree of optimization. The real gains, however, come with dynamic foveated rendering, leveraging the user’s gaze to optimize rendering and transport in real time — which requires eye tracking.
There are several reasons why foveation technologies are crucial for the current XR system, including their capability to:
Free up device resources — which can be used to, for example, enable complex shading, while maintaining high and steady frame/refresh rates.
Support the use of graphics-hungry applications on standalone devices — promoting visual quality, immersion, and graphics.
Lower bandwidth requirements, reducing the overall cost of delivery.
Tobii’s solution for dynamic foveated rendering provides outstanding computational efficiency for device performance. Delivering accurate, low-latency tracking of a person’s focus in real-time, the solution supports immersive user experiences at a fraction of the resource cost. On a standalone headset, our solution enables framerate increases of up to 78% and a drop of 10% in power consumption, with tests showing an average savings of 66% for GPU shading load on a PC-tethered device.
NVIDIA’s Variable Rate Supersampling (VRSS) solution is designed to raise image quality based using foveation technologies. The VRSS 2 release uses eye tracking information delivered by Tobii Spotlight for Foveated Rendering and delivers it, with minimal overhead, to the NVIDIA driver. This architecture ensures that VRSS-2-compatible applications will reap the benefits of dynamic foveated rendering without additional integration.
Our solution for foveated transport in the XR ecosystem is a lightweight foveation codec that leverages the gaze stream generated by our eye tracking integration platform in real-time. It reduces transmission over 5G, Wi-Fi, or cable in split rendering architectures. The optimization foveated transport delivers ensures that next-gen headsets will support ultra-realistic content while maintaining comfort with reduced video encoding and decoding latency, bandwidth usage and transfer latency, as well as improved visual quality.
Smooth frame rates
Consistently reduced GPU load provides headroom to maintain frame rates.
High frame rates
Reduced GPU load enables high frame rates.
High resolution
Supporting high resolution with existing GPU.
Enhanced graphics
Complex shading is possible without increasing GPU load.
"It saves processing power and works surprisingly well, and it's exactly what the next wave of VR headsets will lean on to create higher-res graphics on small headsets with limited battery life.”
"Foveated rendering relies on eye tracking. In fact, that eye tracking needs to be essentially perfect. Otherwise, there would be distracting delays in detail when looking around. Not all foveated rendering solutions are created equal. The better the eye tracking, the more gains can be found in rendering efficiencies."
"Foveated rendering on PS VR2 can bring a massive reduction in GPU usage while producing the same perceptual quality – and, combined with eye tracking technology, the performance gains are even better...We’ve also seen gains up to 3.6x faster when foveated rendering is combined with eye tracking."
XR headset manufacturers, users, and developers are all looking to eye tracking to overcome the challenges of delivering the best possible user experience with available system resources.
Eye tracking is highly adopted in XR, delivering immersion, performance, and feedback — value for everyone in the XR ecosystem.
Eye tracking — a foundational technology for XR — addressing challenges such as constrained system resources, user comfort, and immersion.
In this post, Doug explains foveated rendering in its two forms: dynamic and static, he talks about what you can do with the technology and how it's crucial for XR.
Tests carried out on headsets equipped with Tobii eye tracking and Tobii Spotlight technologies show that GPU shading load drops significantly when dynamic foveated rendering is applied. In this white paper, MI&S validates the tests and results.
VR users demand amazing content, low-priced devices, and fantastic experiences. This e-book describes how dynamic foveated rendering helps to meet these demands and deliver the best possible experience with available system resources.
Disclaimer
Benchmark tests carried out on Pico Neo 2 Eye
Benchmark tests carried out on HP Reverb G2 Omnicept Edition
