Eyes on HP: better HMDs, social interaction for work and life, and BYOD

In this blog post, Peter Peterson, Head of XR Software and Solutions, R&D at HP, gives his views on current XR market trends and possible challenges impacting the XR market, as well as key usage areas. This is the fourth post in Tobii’s blog series “The future of XR”, where Tobii’s business partners share their views on the development of the XR industry.

What are the most important trends that you see will impact the XR market in the coming 3–5 years?

Peter Peterson: Working at the forefront of XR technology, it’s critical to recognize there are a lot of important trends that are going to affect the XR market in the next 3 to 5 years. The VR market today is predominantly growing due to use by early technology adopters, enthusiasts, gamers and commercial users that take advantage of the increasing quality of visuals and comfort that leading providers, such as HP offer. This provides more realistic and immersive environments for the user and has allowed game developers and commercial content creators to really improve the efficacy of their applications while using VR. While looking at the entire industry, we’ve also seen VR devices come down in price and provide more initial out-of-the-box capability. We can’t ignore the emergence of “all in one” VR which removes the need of a laptop or desktop PC to provide a high-quality immersive experience. While these experiences are not as high fidelity as some of the PC VR equivalents, it allows people to experience their first foray into VR without needing to purchase a PC with a high-end GPU plus the VR HMD.

One of the most important trends HP is focused on is where the VR device itself needs to become more comfortable, smaller and provide higher visual fidelity while allowing users to wear it for longer periods of time.

VR is quickly going to get used for more productivity-based work and really requires a device which a user can wear comfortably throughout the day. That could be at the office, or using it casually at home.

The other important trends of VR are going to be its usage for social. VR doesn’t always need to be used for commercial use or productivity-based work. People are also not going to want to spend time in VR just playing games. How do we represent ourselves inside of VR while you’re interacting with other people? There are certain social applications today that try to do this, but we need to continue to evolve the technology and the types of sensors that allow us to infer more about the individual person back into the VR experience. Computer graphics have evolved greatly, and we are able to more realistically project people in 3D space, and very close to being able to cross the uncanny valley for avatars. I don’t think we always need hyper-realistic avatars to have a good social experience in VR, but those avatars should interact more in our likeness. They need to move their mouths, face, and limbs correctly while they are interacting. They need to blink, move their eyes, and look at you while you’re talking, in order to create the realistic experience that most social VR applications struggle with today.

Technology for bringing avatars to life in VR is quickly being adopted today through eye tracking, and mouth tracking solutions. HP is playing an important role to advance this technology through the HP Omnicept platform, and our first biometrically enhanced HMD: HP Reverb G2 Omnicept Edition.

What do you think the biggest challenges are to VR adoption today?

Peter Peterson: There are still a lot of challenges to VR adoption today. I’d say one of the largest bits of consumer feedback is price and making sure you have a PC that is powerful enough to drive a VR experience. The great news is that the next generation of PC’s will be able to drive the latest VR HMD’s without needing to have a dedicated high-performance PC for VR experiences. Of course, there will always be the “bleeding edge” of PC performance that will be able to drive even more realistic VR experiences.

While not everybody’s going to go out and purchase a VR HMD that’s PC connected, one of the more important things that’s happened to the VR market recently is the advent of the all-in-one HMD — primarily the offerings of Oculus Quest and what they have brought to the market. Although the experience on an Oculus might not be the most immersive you can receive today in VR, it shows people the promise of what VR can be, and so as they continue to want more, and demand more, from VR devices they’ll quickly realize the need for them to move up to a higher fidelity device like the PC tethered HP Reverb G2 VR HMD.

Another challenge today are the applications that are available for VR. I’m going to split those titles into consumer: (enthusiast, gamer) vs. commercial offerings. For the enthusiasts and gamers who want to use VR today they’re looking for a highly immersive experience and a good storyline for gameplay.

Over the last 10+ years there’s been a huge explosion in the amount of AAA games available for PC that have really brought highly immersive experiences to the gamer. VR specifically has been lagging in bringing great titles with long gameplay to VR, struggled to give the VR enthusiast and gamer an opportunity to justify a VR setup, just to play those games. For some time, VR unfortunately has been a bit of a novelty, but it has been very active in the simulation community. For people who use driving simulators or flight simulators at home, VR’s been very prominent. But again, that’s a small portion of the population that are doing this type of gaming at home. So, what we’ve seen recently is an influx of brand-new titles for the enthusiast and gamer in VR. Most notably Valve’s Half-Life: Alyx™ has provided an incredible gameplay in VR and taken advantage of the things that VR does well in order to provide a highly immersive experience. I’m extremely excited to see all the new games that are being designed specially for VR, and there will be no more excuses for lack of great games for the platform.

Now on the flip side, the commercial customer is looking for a different experience. First, they are interested in the developer experience, because most of the time they’re creating unique content that is either geared towards training, education, or collaboration.

Developers have realized there is a market to provide enterprises directly with these solutions so a new series of applications are hitting the market for commercial VR usage around collaboration. These are proving to be very interesting especially in today’s modern work environment. In 2020, COVID-19 brought new challenges to us in the ability to connect and work together, so we need ways for people to virtually come together and not just use typical video conferencing software.

While traditional work collaboration requires you to be present with your colleagues in the same space, what is not done well is representing individuals with a high degree of accuracy and realism. Currently we represent people by using avatars which are 3D representations of oneself, realistic likeness or a cartoonish representation based on a user customized character. The problem with these characters is that while we have the graphics and modeling capability to make them look real, they don’t behave like the user that’s behind them. They are missing the ability to express, emote, and generally look human while communicating. HP has been specifically working on new technologies to help solve this with our next generation VR HMD: The HP Reverb G2 Omnicept Edition. The HP Omnicept SDK is geared directly towards developers who are building these experiences and it allows them to integrate these features seamlessly. As a developer myself, I’m usually looking for toolkits that provide ease of integration and can solve and enhance my applications I’m developing without much fuss.

What will need to change over the next 3 years to shift or reduce these challenges?

Peter Peterson: The entire XR industry continues to need to innovate for all of us to overcome the next set of challenges. Not all the challenges we have are purely hardware technical innovation. Yes, a lot of the technologies are focused around hardware: high resolution displays, optics, power, and form factor — but the entire ecosystem of XR software and applications needs to evolve alongside it.

I’d like to see more XR technology, software companies, and independent developers working together to provide more platform agnostic solutions in order to homogenize the ecosystem.

Right now, we are seeing a buildup of software and software titles that are targeted for specific XR devices and runtimes.

We will also need to have a breakthrough in the amount of computing power required to drive high-resolution displays. Today we are pushing the limits of display technology in order to pack the greatest number of pixels into a high-density display and optical solution. In order to increase the number of pixels we still need to increase the output of graphics processing per watt. While graphics cards are getting more powerful, they also require increased power requirements, and this might be acceptable for desktop PC users who have their machines plugged into a wall and affectively have infinite reserve of power. This presents a very special challenge for mobile XR users where the power envelope is primarily low. If you look at the mobile phone market which is an interesting analogy to where we’re seeing with VR all in one, the battery technology needs to continue to advance further for us to be able to deliver more and more powerful VR experiences to mobile, battery powered devices. You could assume that there is a potential for a hybrid approach. We are on the forefront of next generation wireless technology, like 5G, and the amount of bandwidth that will be available, will certainly be able to deliver content directly to the mobile devices.

On the enterprise side, which industries will benefit the most from XR and why?

Peter Peterson: XR technology has never been more exciting for the enterprise user. There are so many important use cases for VR in the enterprise. One of the use cases that we primarily focus on is training.

Training is very interesting to us not only because an enterprise can save a considerable amount of money developing a training experience that’s virtual, they do not need to move their trainers and trainees into physical spaces anymore.

They can also simulate extremely dangerous environments inside of VR without putting the trainers and trainees at risk. Couple that with what HP has done with the HP Omnicept framework, we can go above and beyond the training experience, and can measure the efficacy of the training that’s being given. By incorporating the HP Reverb G2 Omnicept edition HMD, we can read the real-time “cognitive load” value from the user. This allows the developers building the training applications, using the HP Omnicept SDK, to incorporate that feedback real time inside of their application in order to tailor the training experience to the user. Long gone are the linear storylines for training, and now we can create a per user adaptation of training based on each user’s performance. At the end of that training material the trainer is going to have a high confidence level if you can perform what you’ve been trained on.

Education is as equally important to us, and VR will provide a high degree of efficacy for education environments. Today’s challenges with remote learning really call into the need for better more immersive virtual learning techniques. While students traditionally have learned through live instruction coupled with textbook theory there is a lot of fields that require practical application. VR provides that ability to visualize theory and allow the students to use practical application. Advancing VR particularly in education and providing greater access to it should provide better experiences for students and teachers.

Which XR application areas do you think will have the brightest future?

Peter Peterson: Just like the mobile phone has changed the way we do work today; it has become a piece of technology that not only helps us in our personal lives but also makes us more productive in our professional lives. There was a time where the personal phone and business phone were two separate devices. Unfortunately, the commercial side of the devices didn’t evolve to the consumer demands. Those demands wanted a more interactive entertainment experience, but they also did not want to carry multiple devices that could perform the same basic function.

I personally believe that we are going to get to a point where the same BYOD revolution that happened to mobile devices also will impact XR devices. I certainly feel like XR devices are going to become more personal.

They will become more customized for the individual user, and we’re going to see different device manufacturers cater to specific markets based on how you want to wear it, face size, shape, and features. There won’t be a one-size-fits-all device.

Generally, enterprise users and device managers will need a way to incorporate these devices into their corporate IT managed infrastructure. While current devices act more like a PC-accessory, the newer AIO (all-in-one) HMDs will need to be managed separately. We will need to offer new tools and services to seamlessly bring the devices into the corporate IT managed infrastructure without causing huge disruption and potentially creating new attack vectors. This was a large concern when new mobile phone devices were brought onto corporate networks for the first time.

This is also going to open a large opportunity for application developers to build new and unique content and services to target these types of devices. These devices won’t just be basic content viewers; they will have advanced features and be able to network with a series of sensors around it either to understand the environment or the human condition. That could be achieved through a network of available physiological sensors and be processed by the future of what HP is doing today with the HP Omnicept framework.

Application developers are going to want the development environments to consolidate into a few toolkits in order to allow them to provide the best experience across as many devices as possible. Similar to the formative days of mobile phone development, where there were multiple manufacturers, and operating systems, developers had to balance this and it became a challenge, as you were trying to provide your services to a the widest amount of users for your service.

Today we have two major platforms to developing for. Most of the time mobile phone developers develop for using services that will allow you to target both, unless you need specific services on one device or the other. This absolutely needs to happen with the XR developer ecosystem.

I think we’re already seeing it today with development environments Unity and Unreal Engine. They are catering towards write once and deploy to as many targets as it can support.

There are also consortiums specifically driving these standards. OpenXR (part of the Khronos group) is working to drive collaboration between multiple parties in the XR industry: hardware, software, and device accessory makers. The goal is to create are a common way of connecting these devices to be part of a homogenized XR ecosystem. HP is a member in the OpenXR group and is an active participant in helping on this mission.

The future is extremely bright for XR devices and technology, and I’m extremely excited to be a part of shaping the future at HP alongside partners like Tobii and the amazing technology they are bringing!