Puzzle Pieces for True Immersive Experiences

Jon Gabay for Mouser Electronics

(Source: Shutterstock.com/HQuality)

Immersive reality technology has been running a long race to gain widespread acceptance. For example, advancements in small, lower power, high-resolution displays have allowed accurate and seamless stereoscopic contiguous visuals over the range of eye movements. Faster, more precise three-axis accelerometers allow higher-speed processors to render images in pseudo-real-time; fast enough to exceed the flicker fusion rate of the eye and make a realistic panoramic-view scene seem genuine as an immersed user moves their head.

Several challenges have been overcome, and several more challenges remain before this technology is good enough and inexpensive enough to truly invade the gamers market, not to mention work its way into industry, business, military, law enforcement, and more.

Whether you are talking about extended reality (XR), spatial computing, holograms, or volumetric data, people tend to have expectations as to what the experience should be. When people put on a VR headset for the first time, many are enthralled, but many are disappointed. While graphics, sound, and head tracking are better, humans have a range of senses, and only a few can be used in an immersive environment. Let’s face it: Even today, we are not quite at “Avatar”-level immersion.

Still, people have not given up on immersive technology's promise, and they are counting on engineers to deliver the goods. Almost every industry you can think of is optimistic about the benefits immersive technology will bring, making the market for this technology as expansive as an engineer's imagination. Yes, delivering on their expectations is a challenge, but then, that's the life of an engineer: Solving challenges. Let's explore the types of challenges engineers are faced with when developing future immersive experiences.

To create a truly immersive experience, you need sensors, machine vision, 3D scanning, power management, and hundreds of different components in the user device itself. In some cases, like volumetric video, hundreds of sensors and cameras surround the action. So, is the technology itself the challenge?      

Yes, technology challenges are still an issue, and many clever engineers are addressing these. The key is creating an authentic experience. Real video, for example, can be used with digital overlays of rendered objects, either static or in motion. The video capture technology has risen to the challenge, and the video display technology has improved to meet the challenge.

A virtual reality game or an engineering walkthrough of a digital twin may not need the highest resolution. In those cases, thanks to The Law of Closure—the gestalt law that says if there is a small-enough break in an object, we will perceive the thing as continuing in a smooth pattern—lower resolution may be just fine. But if we're being wheeled into an operating room to undergo intricate brain surgery, it's probably better if our doctors practiced on the most realistic virtual cranium possible. Yet, even if high resolution isn't necessary, it could be the thing that helps gain market share.

In many cases, resolution is not as important as latency. Motion-to-photon latency is one of the barriers engineers are trying to break. Ideally, you want the user's spatial experience to be indistinguishable from reality. To do that, experts say you need a delay of under 15ms. Anything above that will be frustrating for a user, but it can also cause nausea or potential danger depending on the use case. If you cannot resolve problematic latency issues, your hardware will likely find its way to the landfill.

While off-the-shelf sensor systems are not quite at that 15ms rate yet, we continue to see incremental improvements. Oculus decided not to wait for component manufacturers to figure it out and built a sensor that supports sampling rates up to 1000hz and with just a 2ms delay. They've proven it can be done, so hopefully, theirs is the tide that raises all boats.

Compute power coupled with communications speeds are critical. With a self-contained unit that includes the main processing engines, real estate is crucial. There are a lot of high-end components that must fit into smaller and smaller spaces. Tethering a VR headset to a computer with cables, for example, can defeat the purpose of freedom of movement for gaming as a case in point.

Not every application will suffer from this, though. A training or educational use of virtual, augmented, and extended reality can accommodate cabling to a more powerful computer inside a desk, for example. Here, students are not mobile, meaning the cables don't get in the way. 

USB 3.0 with a good 5GB/s bandwidth is an excellent multipurpose interface and plug-and-play. As happened with the initial USB 2.0, connector sizes can shrink in future iterations, making connectivity less intrusive. Since USB is such a common standard embedded into so many processors, using it can reduce costs and design time in this case. 

The MIPI CSI-2 (Mobile Industry Processor Interface) standard is one of the most common interfaces used in head-mounted VR devices. Also, MIPI's 6GB/s high bandwidth is faster than USB 3.0. Finally, CSI-2's multi-core processors mean it uses few resources from the CPU.

Another advantage to a cabled interface is the ability to distribute power regulation. Increasingly higher screen resolutions and refresh rates lead to high display power consumption. Oculus tells us on their developer site that a governor process on their device "monitors an internal temperature sensor and tries to take corrective action when the temperature rises above certain levels to prevent malfunctioning or scalding surface temperatures.” This corrective action consists of lowering clock rates. For example, eliminating heat-generating voltage regulation in the headset can reduce overall temperatures.

While much attention is devoted to optical and visual clarity and continuity, the audio aspect of immersive technology is also very important to the experience. Like visuals, the audio must rely on the head tracking and position detection to alter the surround sound aspect of the immersive experience.

While high levels of audio power are unnecessary since the speakers and sonic emitters are relatively close to the ear, small size, low distortion, low voltage, and efficiency are required. An example of a well-suited amplifier for immersive use is the Maxim MAX98360A/B/C/D Digital Class-D Amplifiers. These very small 3.69 mm squared WLP packaged devices crank out up to 3.2 Watts into a 4 Ohm load at 5 V.

The low 2.2 ma quiescent current assures long battery life when not in use, and these parts feature 92% efficiency with a low 0.009% THD at 1KHz. Based on class D switching technology, the edge rate control technology eliminates the need for external filters and these parts also feature click and pop reduction circuitry.

The sample rates from 8KHz to 96 KHz are supported, and data words can be 16-, 24-, and 32-bit resolution to increase dynamic range. While able to interface to 1.2- and 1.8-Volt logic, these parts can tolerate 5.5V I/O and eliminate level shifting for logical control.

A simple I squared S interface keeps pin counts low (9 pins) and simplifies interfacing to digital control circuitry (Figure 3). Audio development tools like the MAX98360AEVSYS#FCQFN provide ready to go test platforms for quick and easy evaluation.

Figure 3 – Highly efficient and small sized amplifiers are easy to interface and ideal for immersive reality applications. (Image Source: Maxim Integrated)

Engineers can make immersive technology more inclusive by refining a universal design approach into their system and merging that with quality audio. Immersive technology is at that critical point many technologies encounter before becoming widely adopted game-changers. It is approaching maturity, but it is not yet widely adopted. Users still have hope that their experience will match their expectations. That makes now the perfect time to integrate accessibility into the technology.