Using HMI to Reveal the Hidden World of Extended Reality

Countdown

The timer on the wall methodically counts down towards zero.

We are trapped.

My family, along with my team member Kevin, are working feverishly to find hidden clues, crack codes, solve challenging puzzles, and escape!

Exposed to a deadly virus, if we do not find the security override code and antidote within moments, the facility we are in will self-destruct, and there will be no escape.

Time is of the essence if we want to succeed.

At last, we sprint over to punch the code into the keypad and make it out with only seconds to spare.

We have just participated and succeeded in a high-adrenaline escape game. Escape games are where players are locked inside a themed room and must complete their mission before time runs out. It is a real-life adventure created through an immersive experience of carefully crafted rooms, equipment, and props.

In order to be successful in an escape game, you must be able to connect all the clues in a manner similar to connect-the-dots.

Connect the Dots

Connecting the dots requires you to view things differently. Extended Reality is unlocking a whole new way of seeing—and interacting with—the world around you.

Tomorrow’s technology includes the emergence of Extended Reality. Extended Reality (XR) refers to all real and virtual combined environments and human-machine interactions generated by computer technology and wearables. XR includes the following concepts as well as their areas of convergence and overlap (Figure 1):

• Augmented Reality (AR): A technology that produces a composite view by superimposing high-resolution images on a real-world view.
• Augmented Virtuality (AV): A technology, sometimes called Hybrid Reality (HR) or Mixed Reality (MR), that merges real and virtual worlds.
• Virtual Reality (VR): A technology that provides a computer-generated, three-dimensional image representation of an object or objects, which a user can interact with in a manner to real-world objects.

Figure 1: Innovative technology, including AR and VR, is a subset of XR. (source: Mouser Electronics)

Emerging technological advances—including the Internet of Things (IoT), artificial intelligence (AI), blockchains, smart homes, and automation—will allow XR to augment human and machine interactions. Not only will XR redefine how we see things, but it will forever transform the human-machine interface (HMI) and actions employing human cognition as an input.

In the future, successful companies will need to adjust their processes in order to implement and execute XR to its fullest potential. By operating at the edge instead of at the core, these innovative technologies will enable human creativity and consciousness to open up new applications and markets. Get ready to connect one of the many dots in IoT. Learn how to adapt and change for what tomorrow brings by understanding the technologies, how they may be used, and what we might all need to do as we prepare for XR and the future of HMI.

Extended Reality and the Role it Will Play in the Future

XR is approaching steadily. IoT—a system of smart, connected devices—is connecting the dots. IoT will receive a big boost from 5G, which will enable massive connectivity by providing capacity-enhanced mobile broadband with ultra-high reliability and low latency. Applying XR to IoT designs provides an opportunity for the physical world to connect and interact with the digital world. Electronic components and electronic systems are empowering XR’s highly immersive and powerful sensory and cognitive experience.

As water naturally finds its proper level, so will XR find its appropriate role in assisting engineers based upon its capability to understand the world around us. XR may play prominent roles in designs that are extremely dangerous, expensive, time-consuming, or perhaps difficult to reproduce or simulate in real situations. In these contexts, XR will enable engineers to operate with human and machine insights while taking advantage of XR’s human sensory model and experience to glean additional insights. These insights will help make applications safer, less expensive, and less risky, while also producing shorter leadtimes.

XR Platforms Will Aid in the Design Process

In the world of electronic engineering, XR platforms for the design process have not yet achieved widespread impact and adoption. Some cutting edge institutions and firms are employing these platforms, but XR’s prohibitive costs and complexities have prevented its widespread adoption. As its costs and complexities decline, electronic engineers should one day be able to specify electronic components and design boards with the assistance of XR technology.

XR Will Provide Explosive Growth in Human Collaboration

As XR becomes more prevalent, it will provide explosive growth in human collaboration. XR will enable large networks of people to work cooperatively together and learn from others. Just as the success of my team’s escape from the escape room adventure required us all to work together in various capacities—to provide unique insights, crack and decipher clues as a group, share resources, experiment with various combinations, and come to general agreements as to how to proceed—so XR will enable and require design engineers to work in mutual cooperation with both human and machine (including computers, robots, etc.) teams to accomplish the most successful designs.

XR Will Drive Human Creativity

XR will drive human creativity augmentation rather than human-physical automation. This means that XR will help unlock human creative ingenuity. Humanity will not be replaced. Rather, humanity will be needed to continue to steer and create value for others. The creative mind will be unleashed, able to accomplish more things. Human-machine interactions—augmented by XR—will drive the development of new applications that meet human needs. XR is a tool created from the vitality of the human mind. Because tools can be used unwisely, humans must work together to ensure that XR is employed responsibly.

The human mind and digital machines have inherently different abilities and capabilities. A human mind is very rich in its capabilities. Digital machines, while lacking in the human mind’s richness, can far exceed human capabilities in handling, manipulating, filtering, processing, and sorting specific forms of digital information. When engineers coordinate the power of the human mind and digital machines, the combination creates augmented tools of expression and understanding. An example of this is when a human pilot operates a transcontinental aircraft with the sophisticated electronics and systems found in today’s jet aircraft. The synergistic collaboration of the human pilot, aircraft, electronics, and systems allows for an extremely efficient and safe mode of transportation. XR’s synergistic collaboration capabilities will enable tomorrow’s design creations to leverage both the human and the machine. As a tool, XR will enhance people’s skills and their ability to create. This new tool will not replace humans but rather allow them to explore new opportunities and possibilities.

Extended Reality Applications

In the world of XR, consumer applications are presently dominated by gaming applications. However, in the future, consumer applications will pave the way for commercial business adaptations. These new applications ultimately will drive market growth and adoption. Some of the large markets expected to use XR include:

• Warehousing and Inventory Management
• Healthcare and Medical
• Industrial and Manufacturing
• Education
• Emergency Response (Fire, Police, Government, etc.)
• Engineering
• Entertainment and Leisure
• Marketing and Advertising
• Retail
• Security

Let's take a look at a few of these applications.

Warehousing and Inventory Management

XR technology will provide new opportunities for efficiency and effectiveness in warehousing and inventory management. Mouser Electronics, where I am employed, manages a massive inventory. As a worldwide, leading authorized distributor of semiconductors and electronic components for over 750 industry-leading manufacturers, Mouser specializes in the rapid introduction of new products and technologies for design engineers and buyers. Selection, speed, and accuracy are essential. By implementing XR capabilities into warehouse operations and inventory management systems, distributors can optimize standard tasks. For example, using XR can help warehouse workers locate and pick inventory with greater speed, efficiency, and accuracy by sending instructions to the worker through a tablet or perhaps wearable smart glasses (Figure 2). Additionally, warehouse managers can use XR to train workers, view warehouse activities in real-time, and design warehouse layouts.

Figure 2: Warehousing and inventory management can operate more efficiently with XR technology. (Source: Mouser Electronics)

Healthcare and Medical

XR offers the promise of doctors being able to examine your body more three-dimensionally, allowing them to more accurately grasp potential issues and relationships by manipulating physical objects in the digital realm (Figure 3).

Figure 3: Surgeons perform brain surgery using XR technology to obtain a better visual and three-dimensional understanding of what they are operating on. (Source: Mouser Electronics)

Industrial and Manufacturing

Industry and manufacturing are undergoing a revolution known as Industry 4.0. It is a fundamental movement towards cyber-physical systems that are securely connected in a manufacturing enterprise. Industry 4.0 is manifesting through the convergence of many emerging technologies—including IoT, 5G, and XR. This transformation of technology promises to connect and enable communication between machines, systems, and interface with humans. This movement will enhance efficiency by proactively adjusting processes towards the optimization of resources. Industry 4.0’s real-time networking will provide flexibility to change “on-the-fly”—thus speeding production rates, minimizing delays and downtimes, and optimizing production schedules. This new way of managing operations will increase safety and security by providing relevant information before any realization of a failure or out-of-boundary condition. Engineers may take advantage of this transformation by deploying XR technologies to control automation processes, make adjustments, and provide direction and guidance to robots and cobots (Figure 4).

Figure 4: Industrial engineer using a virtual interface screen to check and control a robot arm in a factory. (Source: Mouser Electronics)

Looking Ahead into Our Heads

We have looked at some applications where XR combined with IoT transforms the situation. At present, the cognitive science and medical communities are exploring interactions between cognitive AI services and human minds. One area of research is the augmentation of HMI by removing intermediary displays. Rather than looking through a headset or managing a virtual tablet, the goal is to create a more direct experience upon the perception happening in the brain itself (Figure 5).

Figure 5: What is the future of the HMI? It may likely involve a more direct and personal cognitive interface experience. (Source: Mouser Electronics)

Conclusion

When you look at things differently, the things you behold appear differently. Tomorrow’s technology includes XR. Emerging advances will allow XR to augment human and machine interactions, forever transforming the way we use our human minds to manipulate, improve, and use the things we make. As a design engineer, will you welcome and connect one of the many dots in IoT?

The clock is ticking, tick-tock, tick-tock. The countdown has begun. Here comes XR. Are you ready?