Immersive Tech Takes Us From Spectator to Participant
This Technology Changes How We Interact with the World
(Source: pickingpok/Shutterstock.com)
No Longer Distanced by Reality
Even before the spoken word, humans used and understood gestures to convey information. Charcoal, and eventually pens and pencils, provided more clarity and specificity, and for the first time, we could encapsulate knowledge, news, and experiences in a written and illustrated form. But our saga of creativity and expression certainly did not end there. Think about what exists at our disposal to share our experiences and knowledge—from audio and video capabilities, to more interactive types of communication.
The idea of immersing oneself in a technologically-rendered personal environment is not new. We could argue that the use of stereo headphones is a form of immersive technology. Closing your eyes, you can almost feel yourself in a nature setting, for example, with forest sounds and creatures all around. Although video and movies provide a partially-immersive environment, the experiencer is still distanced by reality. But advancements in system engineering, display technology, audio technology, and mixed-signal digital technology let us move from spectators of information to participants in its creation.
Overcoming Challenges
The video headset was the first attempt at visual immersion. Early products such as the Stunt Master were nothing more than low-resolution video screens worn around the eyes (Figure 1, left). With only ¼ VGA resolution (320 x 240), the device was just a personal video display worn on the user’s head. Head tracking, interactivity, or built-in audio was not a part of the experience, but it did help pioneer the ergonomic and optical aspects of wearable technology. Early user Interfaces were clever but not very effective. An example is infrared-sensing gloves that would allow a computer with a receiver to detect and attempt to understand hand movements (Figure 1 right).
Figure 1: Early attempts at immersive technology were minimal but clever. The Stunt Master headset (left) featured ¼ VGA resolution and low-fidelity speakers. The IR glove controller (right) used strain technology to measure finger position and IR LEDs to transmit hand motion and button presses. (Source: Mouser Electronics)
Another stumbling block was low-speed and -density memory chips and modules—by today’s standards—without the depth to store 3D images. Head tracking was not practical, and energy management was not what it is today.
This technology is finally advanced enough to be on the cusp of becoming a reality. The real-world and everyday use for humans immersed in a technologically-created environment can benefit individuals and society if used wisely.
Real-World Uses for the Next Plateau
As is with many technologies we take for granted, military applications often pioneer the underlying technologies that find their way into the public domain. The cathode ray tube, television, radio, and microwave are just a few examples of this. With immersive technology, military training such as flight simulators spawned the immersive environments we are building upon.
Although simple by today's expectations, this work pioneered 3D mapping and representation, perspective processing, memory management and indexing techniques, shading and light source processing, and image stitching and scrolling in real-time. In addition, it used multiport or redundant memory blocks so that one block of memory was playing back the real-time video while the other block was updated for the following sequence.
Although many flight simulator enthusiasts enjoyed this technology for fun, the real-world capabilities of what immersive technology can do for society are just now becoming evident. Although gaming is pushing immersive technology into the affordable and popular realms of society, important uses will actually save lives as the technology merges with other modern mainstream capabilities.
For example, consider an aircraft engine in need of repair. Looking through a service manual has always been the mainstay, but this can be enhanced using immersive virtual technology. Instead of seeing a flat picture, or an isometric or parametric drawing, a virtual floating engine can come to life in front of a technician’s eyes. The technician can rotate it, virtually disassemble it, look right into it, and see where release clamps, bolts, and assemblies must be accessed (Figure 2). Of course, automotive repair, appliance repair, and virtually any machine repair procedure can be enhanced and made more understandable when the technician has a resolute virtual projection with which they can interact.
It gets even more interesting when sensor data is combined with the virtual image so the repair technician can see into the engine and find the faulty bearing, then determine how to reach that buried bolt that can’t be seen.
Figure 2: Virtual machines allow technicians and mechanics to study the intricate specifics of a complex assembly before starting a job. This is useful for training and actual repairs. (Source: Gorodenkoff/Shutterstock.com)
One of the most anticipated and likely impactful areas of use is within the medical field. For example, with high-resolution scan data from magnetic resonance imaging (MRI), a doctor can journey through a patient's body and examine a tumor from every angle. Then, the doctor can scale it to make it so large that its insides become visible.
When combined with interactive manipulators (such as a higher resolution of our control glove from Figure 1, right), a doctor can perform delicate surgery using robotic tools more precisely than a shaky human hand. Coupled with sensors—in this case, haptic feedback—the skilled surgeon can grab, pull, move, and stitch flesh without causing damage (Figure 3).
Figure 3: Virtual and augmented reality can let doctors perform delicate surgeries with more precision and control than humanly possible without the technology—even remotely. (Source: Gorodenkoff/Shutterstock.com)
This technology even allows for remote surgery. For example, on a mission to Mars, a surgeon on Earth can perform a specialized procedure on an astronaut a million miles away. Even simple procedures such as cleaning out an ear canal or clogged artery can be performed without adjusting oneself to zero gravity. Signal lag time is an issue, especially when millions of miles away. This is where artificial intelligence will help temper movements and actions for the delay times or if a signal is lost.
As an ordinary person grounded here on Earth, immersive technology can also save my life. Think about a time GPS routed you through a blizzard. It is impossible to know where the road and the lake are when everything is covered with snow.
Coupled with sensors and accurate GPS, an immersive virtual reality display can superimpose the actual road over the real-time video, so we know where and where not to drive. Two things to note from this example: The addition of sensor data beyond human’s senses added to the immersive experience to enhance abilities beyond what they were capable of without the technology. This is where virtual reality coupled with augmented reality makes a difference.
Another benefit virtual and augmented technology brings to society is the life-saving jobs performed remotely or virtually. For example, when defusing a bomb, a virtual presence device such as an articulated robot can journey to a dangerous location and perform delicate operations without endangering a human. The 3D depth perception will allow the bomb-squad operator to have much better depth perception than any 2D system could.
Other dangerous jobs such as manually shutting off a valve in a nuclear reactor can also be accomplished more safely and with higher skill than any 2D system alone could perform. Key in any of these precision remote and virtual operations is haptic feedback. The ability of an operator to feel the objects he is touching with force feedback technology is a must for remote surgery and any other critical operation. A simple infrared (IR) glove has given way to real multi-sensor and multi-actuator gloves such as the HaptX industrial-grade models (Figure 4).
Figure 4: Highly accurate sensors precisely detect finger and hand motions and gestures, and haptic force feedback technology dramatically increases tactile dexterity and force control. (Source: HaptX).
An interesting recent development is the video-based gesture detection, recognition, and control technology that features haptic feedback without a glove. Strategically placed video cameras are used to monitor hand movements, and a clever ultrasonic waveform interference technology actually lets the operator feel tactile feedback—kind of like a sonic force field (Figure 5).
Figure 5: Although high-def video cameras capture hand movements, ultrasonic emitters using precise phase control let the operator feel a movement. (Source: Mouser Electronics)
Balancing Augmented Reality with Virtual Reality
Like Arnold Schwarzenegger as The Terminator cyborg, witnessing reality with information superimposed can set us apart. Both augmented reality and virtual reality have their place, though.
Typically, with augmented reality, video screens are clear or transparent. In modern terms, organic light-emitting diode (OLED)technology is ideally suited to fill this need. Each pixel is a separate point of emissivity, so it doesn’t need a backlight or scanning laser to render an image.
Combined with stereoscopic video, a virtual headset can function well as a virtual reality and augmented reality experience. In this case, reality can be tuned out in adverse conditions, such as a balance adjustment on a stereo. High glare or white-out blizzard conditions do not have to be distractions when a clear and concise virtual world is available.
Figure 6: Augmented reality allows information to be superimposed over the reality of our senses. This can even extend the range of our senses by using coloration techniques to make heat visible, for example. (Source: HQuality/Shutterstock.com)
The ability to detect and warn of potential danger before the human senses do is a definite safety benefit and is not limited to just the human range of perception. Using thermal technology, heat can be seen and recolored to become visible. Such technology will let a driver know if a person or animal is in the road ahead well before his limited visual spectrum can pick it up. Ultraviolet can bring out details the human eye can't.
Extending the human vision range from radio frequency (RF) to X-rays could be a substantial diagnostic and troubleshooting technology. Whether in space, a nuclear reactor, or a submarine, all these environments depend on the ability to detect and fix a potential problem before it becomes a real problem.
Not Just Imaging
Here is where it gets fun. Vision isn't our only important survival sense. Many argue that the sense of hearing is just as important in some cases. We exist as a species because our sense of stereoscopic hearing allows us to detect dangers and locate an approximate position, all without seeing.
Hearing plays a significant role in immersive technology and the immersive experience. Although stereo has been satisfying for most listeners for a long time, it is not an accurate 3-D rendering of the soundscape. Even though several flavors of surround-sound technology have been designed and implemented, they all depend on additional speakers and subwoofers. It's easy to implement this in a room but much more difficult on a headset.
Digital signal processing algorithms take stereo sound information and extract specific channels for surround-sound standards. These synthesized tracks can then be amplified and applied to the surround-sound speakers. Translators also go back and forth between stereo and 5.1, 7.1, 7.2, 9.2 surround-sound standards.
Here is where the technology needs to diverge. For entertainment, reproduction is most important. For example, if something is recorded in surround-sound 5.1, it will sound best when played purely on a 5.1 system. For augmented reality, the ability to sense and reproduce in 3-D is important. A soldier in a stealth mission might be able to use thermal technology to identify an enemy, and filtered, directional surround sound can enhance accuracy. If an enemy is behind a wall, the 3-D sound could still detect breathing or a heartbeat.
Modern systems such as Dolby Atmos are designed for true surround-sound immersive experiences such as in movie theaters. These systems can have up to 64 speakers, which would be too many for a wearable headset. It also introduces processing resource needs because audio is no longer audio. It is a digitally modeled sound object whose characteristics, distance, motion, and direction cause an audio track to be synthesized and distributed to multiple speakers. Although this is cool in a theater, it is likely not feasible for wearable immersive systems.
We Possess the Puzzle Pieces
Designing and building immersive technology headsets are within reach, though. The pieces of the puzzle are ready to design into your device. This is true for augmented, virtual, and combined-reality systems.
Thin Film Technology (TFT) displays are used everywhere and are inexpensive, multiple-sourced, and available off the shelf. These require LED backlights and are opaque, meaning they can be used for virtual reality primarily and augmented reality when combined with a stereoscopic camera.
OLED technology might be the best choice here. It is flexible, can be bent and deformed, and is clear so you can see right through it. But best of all, embedded in the clear film are microscopic LEDs that create a lit display without a backlight. OLED displays exist but are not as readily available as TFTs. Scanning laser heads-up technology is another option.
Multiple manufacturers also make ultra-small and high-resolution video cameras available thanks to the smartphone revolution and selfie movements. The small footprints make them ideal for mounting on glasses frames as well as totally immersive opaque headsets.
Audio processors and small but full-range speakers can also be purchased from reputable suppliers with proven track records. Again, the mobile device industry has pushed this technology forward quickly because of the need to listen to music.
Perhaps the most important advancements in technology making these designs feasible are the advanced multi-axis accelerometers we have at our disposal. These are fast enough to monitor head movements and, when coupled with a quick and responsive video processing and rendering system, provide clear nausea-free use. (Note: When displays lag head tracking, that separation of perceptions can cause nausea.)
Other advancements such as ultra-dense and fast memory, energy management chips, and high-density battery technology are all falling into place to make this technology cost-effective and efficient enough to become the preferred user interface going forward. In addition, the high-bandwidth, high-frequency wireless communications technologies also allow these immersive devices to be completely free of any tethering.
Design Options and Considerations
Factors to consider when deciding how to move forward with your augmented and/or virtual-reality applications are cost, time to market, reliability, and special functions. Cost and time to market are very high on the list for commercial and consumer applications. Ruggedness, reliability, and special functions are a high priority for military and aerospace designs. Everything else falls in between.
In either case, feasibility studies and prototyping can take place by OEMing an existing headset. Several players have offerings with varying prices and levels of performance. These are available in cabled as well as cable-free configurations. The most talked-about is the Occulus Quest 2 for around $300 (USD). It features a self-contained compute and graphics engine, wireless communications, BLUETOOTH® interface for peripherals, and good head tracking. Sony PlayStation® has a competitive model as well that is even in the ballpark price-wise. HTC and Valve Index VR kits are also worthy of mention.
Development tools are also available to allow scenescape and multi-character interaction. Headsets with graphics engines and CPUs are now in place, and communities are developing to help support this brave new virtual world.
Forging Forward into This New World
Virtual and augmented reality let users immerse themselves into artificial realms. With virtual reality, the immersive experience can be a different world than the one outside the headset, while augmented reality enhances information and extends the range of human senses. Luckily, we possess the technology to bring it together and create these worlds capable of enhancing education, machinery repair, surgery, navigation, and extending the range of senses.