Download code from GitHub
Welcome to the world of Digital Reality. The purpose of Digital Reality is to bring immersive experiences, such as taking or transporting you to different world or places, make you interact within those immersive, mix digital experiences with reality, and ultimately open new horizons to make you more productive. Applications of Digital Reality are advancing day by day; some of them are in the field of gaming, education, defense, tourism, aerospace, corporate productivity, enterprise applications, and so on.
The spectrum and scenarios of Digital Reality are huge. In order to understand them better, they are broken down into three different categories:
- Virtual Reality (VR): It is where you are disconnected from the real world and experience the virtual world. Devices available on the market for VR are Oculus Rift, Google VR, and so on. VR is the common abbreviation of Virtual Reality.
- Augmented Reality (AR): It is where digital data is overlaid over the real world. Pokemon GO, one of the very famous games, is an example of this globally. A device available on the market, which falls under this category, is Google Glass.Augmented Reality is abbreviated to AR.
- Mixed Reality (MR): It spreads across the boundary of the real environment and VR. Using MR, you can have a seamless and immersive integration of the virtual and the real world. Mixed Reality is abbreviated to MR.
This book is mainly focused on developing MR applications using Microsoft HoloLens devices.
Although these technologies look similar in the way they are used, and sometimes the difference is confusing to understand, there is a very clear boundary that distinguishes these technologies from each other. As you can see in the following diagram, there is a very clear distinction between AR and VR. However, MR has a spectrum, which overlaps across all three boundaries of real world, AR, and MR.
Digital Reality Spectrum
The following table describes the differences between the three. Later in the chapter, we will go through the details of each one of them.
Virtual Reality | Augmented Reality | Mixed Reality |
|
|
|
Let's explore VR, AR, and MR in more detail.
The dictionary definition of VR reads, "
a realistic and immersive simulation of a three-dimensional environment, created using interactive software and hardware, and experienced or controlled by movement of the body
". However, we will say that there is one and only one primary purpose of the VR, that is, to make you believe that you are somewhere else, for example, in a gaming environment, a war zone, or touring some city.
Virtual Reality illustration
A person wearing a VR device/headset will be able to view the virtual environment, move around wearing it, and interact with items within that virtual environment. Most Virtual Realities are created using a head-mounted helmet or a set of goggles, which are given the generic name: head-mounted display (HMD). HMD is usually connected to a computer or a smartphone, which does all the 3D rendering work, and HMD is just used to display that rendered 3D content. HMD completely covers user's view, by covering both eyes. As a result, the user is completely cut off from the outside world and completely focused on the virtual or digital world.
HMDs basically consists of stereoscopic displays and motion tracking hardware. The way HMDs implement stereoscopic displays is by generating different image for each eye, which results in generating the illusion of depth. Within HMDs, motion tracking hardware mostly consists of a gyroscope and accelerometer to measure motion/position changes. This helps in simulating real-world experiences.
There are various implementations of VR in diverse fields currently, and numerous new ones are coming up every day. It will be a very interesting domain for the next few years, where we will globally see lots of new fields picking up VR and merging it with other upcoming technologies. One such technology, which we can think of, is Artificial Intelligence (AI); Are we talking about the movie "Matrix" here?
While it's too early to predict possibilities in the future, let's explore some of the current fields where VR is being implemented. They are as follows:
- Gaming: Gaming was one of the first fields to pick up, start using, and commercialize VR. Apart from single-user VR games, multi-user online VR games are also available today. With the availability of low-cost devices, such as Google VR and smartphones, VR games/applications are within the reach of everyone.
- Tourism: Tourism is another field that has picked up VR. VR is used for generating stereoscopic 360-degree panoramic views, where the user interact with the virtual place and explores it. So, no more traveling or air-travel; just wear your VR device and explore new places every day.
- Education: Distant learning, visualization, and interactive teaching a few global examples of VR implementations within the education field. Earlier, teachers used to describe the scene and students used to imagine the possibilities. Now, they just visualize new possibilities and interact with them, and spend more time in thinking what's next.
- Architecture: Architecture is a field where VR has made life easier for the architects by making it easy to create prototypes. Now, they do not have to create cardboard prototypes or sketches to convey their thoughts and ideas. VR has made it possible for them to share ideas early in the cycle of development and receive early feedback. This saves lot of time in rework.
- Enterprise productivity: Especially, for enterprise training scenarios, VR is used very frequently. Scenarios in consideration can be simulating war situations for the armed forces, flight simulation for pilots, medical diagnostics simulations for doctors, and many others.
- Web content: With the support of WebGL technology by most of the latest browsers available in the market, rendering Virtual 3D content has become very easy. A lot of commercial websites for automobile companies, for example, have started publishing their automobile features in the form of 3D content, which has made it very interactive for end users to visualize the content.
AR is all about bringing digital information and overlaying it over the real environment. The only difference from VR is that it creates a totally artificial environment around you, whereas AR uses the environment around you and overlays the digital information over it. For VR, you will require a VR device, but for AR it can be achieved by simply using a smartphone, tablet, or dedicated VR devices.
AR can be categorized into three different types based on the display types:
- Screen-based
- HMD-based
- Projection-based
A common example of screen-based AR is overlaying digital information over smartphones or tablet camera displays. For example, you switch on and point your smartphone/tablet camera over an object, and the application recognizes that object and overlays that object information, such as the price or description, as digital information over the object image.
Another example of screen-based AR is the video game Pokemon GO, in which, based on the user's location and direction, digital characters are overlaid over video images.
User is viewing augmented object using smartphone
In the preceding figure, a user is viewing an augmented object, that is, the elephant's digital object is overlaid over the video frame. This is an example of screen-based AR.
Using AR HMD devices, digital information is overlaid directly over the user's view of the real world. So, there is no need to hold any screen to view the digital information. Digital information is directly rendered over the view area of the user's eyes.
HMD with information overlaid over the real view
In the preceding image, the user is using a head mounted AR device to view the physical element in front of him. Within the view of the user, the AR device embeds information about the physical object and provides them with a more immersive experience.
User projection-based AR, a projection is rendered on the target surface itself. This target surface could be anything, such as building, person, room, and so on. To render this kind of projection, the system needs to know the exact dimensions of the target surface, and then using single/multiple projectors, it renders the projection on the target:
Phone dialer pad projection over the hand
In the preceding image, the user is viewing a projection of the phone dialer on the palm of their hand.
Applying AR is quite different from applying VR. AR applications are more focused on integration scenarios with the real world. Some are as follows:
- Gaming: With the release of the Pokemon GO game by Niantic, the demand for AR games has picked up drastically in the market. A lot of companies are coming up with AR games and launching them.
- Architecture/construction/archaeology: AR can be used to visualize completed buildings for in-progress or upcoming building construction. Digital architecture/building images can be overlaid over the real view of the property or ground.
- E-Commerce: Businesses can't reach each customer with demo-able physical product, and opening showrooms is every city is a costly business, especially for start-ups and newcomers in the market. So, reaching out to customers through AR, online furniture retail companies, for example, allows users to consume AR through smartphones, let them visualize their furniture products, and enable them to design their house interiors.
- Education: Traditional education systems require hands-on instructions and real/prototype equipment to explain it better to students. With AR, teachers and students can visualize the same equipment in virtual mode, with very similar training instructions as with an actual device. Another implementation of AR in the education system is distance learning, where students and a teacher very far away, can use VR devices for interactions and virtual classrooms.
- Medical: In the medical field, there are different imaging techniques for various requirements, such as X-ray, ultrasound, and magnetic resonance imaging (MRI), but there is no consolidated view for medical practitioners. AR could be used for these scenarios, where the output from different imaging techniques could be overlaid over the patient and give a consolidated view to medical practitioners.
- Industrial design: AR is being used for designing, sharing ideas, and brainstorming design views among different designers and architects and supplies quick feedback and brainstorming cycles. Earlier, the same process used to take a long time, as designers used to create physical prototypes and then discuss them.
- Travel/navigation/tourism: AR is also used for developing navigational applications; for example, travel-related digital information is overlaid on the vehicle windscreen, which helps the driver in real-time navigation without looking at any other device, such as a GPS or smartphone. AR is also used to develop travel-related applications, which help the user with location-specific information of the place where the user is currently placed, such as historical information about tourist places, or information about nearby restaurants and cuisines.
We talked about VR and AR. Now, think of MR as a hybrid of VR and AR. Using MR, a user can view the real world just like in AR, but can also view virtual objects as in VR. The difference with MR is that those virtual objects are referenced or anchored with the real space. That means that the virtual object is interacting within the real world.
For example, suppose you have a physically real table in front of you, and now using MR you are viewing a virtual ball; if you drop a virtual ball object on a table, in MR, that virtual ball will interact with the physical table. This means that it will bounce on the physical table, and roll off from the table to the ground.
The immersive experience of the virtual world with the real world is so strong in MR that it is very hard to break that illusion.
A user viewing a MR object in real space
At the time of writing this book, there are very few MR devices available on the market; Microsoft HoloLens is one of them. This book is focused on developing the MR experience, using HoloLens.
MR is still evolving, and lot of different industries are still exploring possibilities with it. The following are some of them, which represent early progress:
- Education and training: MR has already started transforming the way education and industrial training are delivered. Companies have started using MR as a medium for delivering readiness/training content to participants.
- Exploring new markets: MR is giving an opportunity to enterprises to rethink their marketing strategy and target new customers. Few popular automobile companies have started using MR device in their specialized car showrooms. With this, they can target consumers early in the sales cycle, even before the first model of the car is available.
- Healthcare: The possibilities for MR in the healthcare industry are enormous. Physicians can use MR devices to visualize consolidated reports, and these devices can also be used as a guidance device during operations or diagnosis.
- Gaming: The gaming industry may become one of the biggest consumers of MR devices. There are so many possibilities for MR games that can interact with the real world.
We learned about VR, AR, and MR; let's see what devices are currently available on the market to explore all these different experiences:
Digital Reality device family
- Oculus: Owned by Facebook, the first in the industry to launch a VR device
- HTC Vive: A VR device launched by the Taiwanese phone manufacturer HTC
- Sony PlayStation VR: A VR device that works with Sony PlayStation, but also works with any personal computer
- Google VR: This is a Google VR device, and is used along with Android smartphones
- Gear VR: This is a Samsung VR device, and is used along with Galaxy smartphones
- OSVR: An open source VR device, started with the support of companies such as Intel and Razer
- Google Glass: An AR device launched by Google
- HoloLens: Microsoft MR wireless and wearable device
- Met: Mixed or Augmented Reality device, which projects virtual images in the wearer's field of vision
- Magic Leap: MR device, still under development
We have seen in this chapter the different types of reality solutions available on the market and their possible usage. We have learned about how these digital realities are similar to each other and how they differ. We have also explored several field applications of these technologies. We have seen the different sets of devices available in the current market place. The knowledge gained from this chapter will help you grasp the subject discussed in subsequent chapters.
Further on within this book, we will focus on MR application development using the HoloLens device. In the next chapter, we will focus on HoloLens as a device. Later, our focus will be on application development, taking you through the step-by-step process of developing a new holographic application and deploying it on the HoloLens device.