Enterprise Augmented Reality Projects

By Jorge R. López Benito , Enara Artetxe González
    Advance your knowledge in tech with a Packt subscription

  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Introduction to AR and How It Fits the Enterprise

About this book

Augmented reality (AR) is expanding its scope from just being used in mobile and game applications to enterprise. Different industries are using AR to enhance assembly line visualization, guide operators performing difficult tasks, attract more customers, and even improve training techniques. In this book, you'll gain comprehensive insights into different aspects of developing AR-based apps for six different enterprise sectors, focusing on market needs and choosing the most suitable tool in each case.

You'll delve into the basics of Unity and get familiar with Unity assets, materials, and resources, which will help you build a strong foundation for working on the different AR projects covered in the book. You'll build real-world projects for various industries such as marketing, retail, and automation in a step-by-step manner. This will give you hands-on experience in developing your own industrial AR apps. While building the projects, you'll explore various AR frameworks used in the enterprise environment such as Vuforia, EasyAR, ARCore, and ARKit, and understand how they can be used by themselves or integrated into the Unity 3D engine to create AR markers, 3D models, and components of an AR app.

By the end of this book, you'll be well versed in using different commercial AR frameworks as well as Unity for building robust AR projects.

Publication date:
December 2019
Publisher
Packt
Pages
388
ISBN
9781789807400

 

Introduction to AR and How It Fits the Enterprise

This book was written for enthusiasts of emerging and interactive technologies who are interested in building AR applications for any market field from the very beginning, and learning about the latest tools to develop them. No prior AR experience is required.

The idea of this book is to introduce you to different AR frameworks by completing a set of projects in different enterprise environments. Although specific programming skills are not mandatory, minimal knowledge is desirable so that you understand the code in each example. Some object-oriented programming skills would be helpful, but they are not required. This book has been written from scratch, so you'll be able to follow any coding necessities step by step.

In this chapter, we will learn what Augmented Reality (AR) is and we will present a brief tour of its history, types, and how it works. We will also introduce the Software Development Kits (SDKs) that we will work with in this book. Finally, we will look at examples of the use of AR and, in particular, its application inside the enterprise. The main objective of this chapter is to understand how AR is being used today, including markets, technologies, devices, and development tools with a variety of AR recognition modes, such as target images, cloud markers, GPS-based, object or face tracking, and spatial mapping. You will also learn about the main AR tools we are going to use in this book and a brief introduction to their main features.

By the end of this chapter, you will have a basic understanding of how AR works and what its different uses are so that you can follow the technical chapters ahead more easily.

In this chapter, we will cover the following topics:

  • Understanding AR
  • Working with AR
  • Using AR in enterprises
 

Understanding AR

AR is the term that's used to describe the technology that allows users to visualize part of the real world through the camera of a technological device (smartphone, tablet, or AR glasses) with virtual graphical information that's been added by this device. The device adds this virtual information to existing physical information. By doing this, tangible physical elements combine with virtual elements, thus creating augmented reality in real-time. The following image shows how AR works:

A user seeing a 3D apple in AR with a tablet

Now, we are going to look at the beginnings of AR and learn how AR can be divided according to its functionality.

Short history – the beginnings of a new reality

AR is not a new technology. The beginnings of AR begin with the machine that was invented by Morton Heilig, a philosopher, visionary, and filmmaker, when, in 1957, he began to build a prototype with an appearance similar to the arcade video game machines that were very popular in the 90s. The following image shows a schema of how the prototype worked:

A schema on how the invention worked (This image is created by Morton Heilig)

Morton called his invention Sensorama, an experience that projected 3D images, added a surround sound, made the seat vibrate, and created wind that was thrown as air at the viewer. The closest similar experience we can feel today is seeing a movie in a 4D cinema, but these experiences were created more than 60 years ago.

In 1968, Harvard Electrical Engineering professor Ivan Sutherland created a device that would be the key to the future of the AR technology known as the Human-Mounted Display (HMD). Far from the AR glasses that we know of today, this HMD, called the Sword of Damocles, was a huge machine that hung from the ceiling of a laboratory and worked when the user was placed in the right place. In the following image, you can see what this invention looked like:

The Sword of Damocles (this image was created by OyundariZorigtbaatar) 

In 1992, Boeing researcher Tom Caudell invented the term AR, and at the same time, AR technology was boosted from two other works. The first AR system, from L.B. Rosenberg, who works for the United States Air Force, is a device that gives advice to the user on how to perform certain tasks as they are presented, something like a virtual guide. This can be seen in the following image:

Virtual Fixtures AR system on the left and its view on the right (this image was created by AR Trends)

The other research in this area was led at Columbia University, where a team of scientists invented an HMD that interacted with a printer. The device, baptized as Karma (AR based on knowledge for maintenance assistance), projected a 3D image to tell the user how to recharge the printer, instead of going to the user manual.

The following diagram is a representation of the continuum of advanced computer interfaces, based on Milgram and Kishino (1994), where we can see the different subdivisions of the MIXED REALITY (MR) that go from the REAL ENVIRONMENT to the VIRTUAL REALITY. AR that's located nearer to the REAL ENVIRONMENT is divided between spatial AR and see-through AR. However, the appearance of mobile devices in the 21st century has allowed a different version of AR, where we can display it using the device screen and camera:

MIXED REALITY and its subdivisions

Now that we have introduced the beginnings of AR, let's learn how this technology can be classified depending on the trigger that's used to show virtual elements in the real world.

The magic behind AR

AR can be created in many ways; the main challenge is how to make the combination of the real and virtual worlds as seamless as possible. Based on what is used to trigger the virtual elements to appear in the real world, AR can be classified as follows:

  • GPS coordinates: We use GPS coordinates, compasses, and accelerometers to locate the exact position of the user, including the cardinal point they are looking at. Depending on where the user is pointing to, they will see some virtual objects or others from the same position.
  • Black and white markers: We use very simple images, similar to black and white QR codes, to project virtual objects on them. This was one of the first AR examples, although nowadays they are used less often as there are more realistic ways to create the AR experience.
  • Image markers: We use the camera of the mobile device to locate predefined images (also called targets or markers) and then project virtual objects over them. This type of AR has substituted black and white markers.
  • Real-time markers: The user creates and defines their own images with the mobile camera to project any virtual object in them.
  • Facial recognition: Through the camera, we capture the movements of the face to execute certain actions in a request, for example, to give facial expressions to a virtual avatar.
  • SLAM: Short for Simultaneous Localization And Mapping, this technology understands the physical world through feature points, thereby making it possible for AR applications to recognize 3D objects and scenes, as well as to instantly track the world, and overlay digital interactive augmentations.
  • Beacons: eBeacons, RFID, and NFC are identification systems that use radio frequency or bluetooth, similar to GPS coordinates, to trigger the AR elements.

Now, you have a better grasp of what AR is and where it comes from. We have covered the basics of AR by looking at the first prototypes, and classified different types of AR according to the element that triggers the virtual images so that they appear on the screen. The next step is to see what is required to work with it.

 

Working with AR 

AR projects need some basic elements to fulfill its function:

  • Digital support, mainly tablet, mobile phone, or glasses. These will be the devices we will use to recognize what we are seeing and incorporate the information into AR.
  • Sensors to perceive reality as a camera, GPS, 3D scanner, and so on. They are usually integrated into mobile devices.
  • An algorithm or application to understand the reality that is being observed, interpreted, and displayed with the associated information.
  • Digital content to enrich reality. This is all the information that you incorporate into AR.

The result is an interface in which reality is observed with additional information located in the space it needs to be in. The following subsections present different AR libraries and SDKs, as well as how they function.

ARToolKit – the first AR library

ARToolKit was the world's first open source software library and allows the creation of AR applications, in which virtual images overlap in the real world. It uses video tracking capabilities to calculate the position of the camera and the orientation relative to the position of the physical markers in real-time. Once the position of the real camera is established, the virtual camera is placed at the same point and a virtual layer of information, such as a 3D object, is superimposed on the real marker. ARToolKit was originally developed by Hirokazu Kato and the initial release was published in 1999 by the HIT Laboratory of the University of Washington. It is currently maintained as an open source project hosted at SourceForge with commercial licenses available at ARToolworks: http://www.artoolkitx.org/.

Introduction to the AR SDKs 

Now, we will introduce the AR SDKs that we will use in this book. From the first appearance of ARToolKit, AR SDKs have very much evolved. Let's introduce and describe the different SDKs we are going to use in this book to create our own AR experiences:

  • Vuforia
  • EasyAR
  • ARCore
  • ARKit

Let's talk a little bit about each of them.

Vuforia

Initially developed by Qualcomm, and currently run by PTC, Vuforia is one of the oldest and most well-known AR SDKs, and it's one of the most stable and best-performing pieces of software in the market. Vuforia offers a wide variety of features, including 2D and 3D tracking, markerless AR, and fiducial markers called Vumarks. It provides multiple examples and extra features, such as virtual buttons, runtime image target creation, and background video texture manipulation.

EasyAR

An AR engine that allows us to create AR solutions in an easy way and offers multiple AR features, including technology that recognizes and tracks the position, rotation, and scale of a previously selected image in the real world, such as a book cover, a photograph, or a business card. It allows us to map a real place and keep track of the objects within it and contains technology to localize and track the position and orientation of real 3D objects instead of flat images and features, which allow us to take videos of the AR scene while we are playing it.

ARCore

The first version was launched in February 2018 and is Google's platform for creating AR applications for Android and iOS. It makes use of different capabilities, such as motion tracking, to estimate the position and orientation of the mobile device regarding the real world; the environmental location, to find and track horizontal surfaces such as the ground, tablets, or walls; and light estimation, to place 3D elements realistically into the real world. Targeted at Android 7.0 and above, not all devices in the market support this technology, although this number has increased greatly since the first SDK version.

ARKit

Apple launched the first version of ARKit in 2017, along with Xcode 9 and iOS 11, to bring AR to iOS devices. The framework, which is included in Xcode, lets users place virtual content in the real world. Months after its official release, it added new features such as 2D image detection and face detection. In 2018, with the iOS 12 release, ARKit 2 was launched with new features such as 3D object tracking, adding realistic textures to the scenes, and creating multiuser AR experiences. At the time of writing this book, the latest version is iOS 13, with ARKit 3 promising a huge improvement on the current state as it will add a new way of interacting with virtual elements, such as hiding virtual objects when a person is detected in front of them. It also allows users to interact with 3D objects via gestures and capture not only facial expressions but the body motions of a person.

Now that we are acquainted with the main AR tools we will use throughout this book, let's look at how we will apply this software to different enterprise fields.

 

Using AR in enterprise

AR can be a valuable asset and technology in multiple fields. In this section, we will learn how it is being used in sectors such as industry, marketing, retail, and tourism. In the first case, we will learn how industry 4.0 has put AR in the spotlight, since it's the natural interface for the human-machine interaction in areas such as production, maintenance, training, and more.

This section's main objective is to give you a better insight into how to apply AR in your own projects before you start to code. This way, you will be able to adapt the examples in this book to your own needs. 

Let's start introducing the uses of AR in the industry field.

Using AR in industry 4.0

The use of this technology in the industrial sector is growing. Its application focuses, among other aspects, on being able to prepare for the breakdown of a machine and thus know what part has been damaged and how to proceed to repair it. This use of AR represents a significant cost saving as it favors rapid action before the breakdown in question. You can also use the AR to visualize a plant or product before carrying it out. This means that you can perfect what to build before turning it into something physical. The use of AR in industry favors the application of one of the enabling technologies of industry 4.0 that help us increase the competitiveness and productivity of a company.

Some examples are as follows:

  • Smart assistance
  • Virtual prototyping
  • Logistics and warehouse management
  • Digital Twins

Let's look at these examples in more detail.

Smart assistance

The use of AR with maintenance process management platforms provides direct support to operators during the operation of industrial machines, thus shortening the time and increasing accuracy in the performance of the task or repair. Some tasks you can have under this case are as follows:

  • Follow guided instructions and workflows
  • Assign activities in real-time
  • Access checklists and list of assigned tasks
  • Create a digital report of the interventions
  • Automatically detect 3D objects
  • View 3D manuals and work orders in real-time
  • Have an expert remote collaborator through video calls
  • Extract and display real-time data from machines

In this case, the use of AR technology improves business productivity by reducing costs and the possibility of errors.

Virtual prototyping

AR can be applied to visualize and configure 3D industrial plants and machinery in real-time. 3D models can be shown in the industrial environment where the validation is carried out, but also in exhibition spaces such as fairs and events. Through AR, you can visualize the hologram of the industrial machinery in the place where it will be installed to evaluate its dimensions, escape spaces, and safety margins. By doing this, you optimize the production processes. In addition, the packaging, shipping, or transport of heavy industrial machinery from one part of the world to another for presentation at fairs has a very high cost. With AR, you can see your product as if it were real in the center of your stand.

Logistics and warehouse management

Through 3D viewers or glasses connected to a warehouse management system, a warehouse operator receives information for the storage of merchandise directly on the device, which allows them to improve the times and quality of their work and, ultimately, the production yield.

Digital Twins

Thanks to AR technologies, you can create a virtual and dynamic replica of a real system, product, or service that, through the information that's obtained from sensors or automatisms, models its behavior. In this way, it allows intelligent monitoring and analysis, as well as the realization of simulations and experiments in the replica, without risks or costs, with the aim of improving the performance and effectiveness of the real system.

In this book, we are going to learn how to use AR for manufacturing by creating a prototype automation with an AR guide and training.

Using AR in other enterprise areas

The industry is not the only field where AR can be of value. The versatility of this technology makes it the perfect communication item between the users and the final product or process and adds a visual and interactive layer to engage them. Whether it's used to make a product more attractive or help users understand a process with added information, AR can be a great asset in many fields. Let's look at some other enterprise areas that we will later cover in this book where AR adds value. 

AR for manufacturing

AR has many uses in this area, such as prototyping, production, and training. In these areas, AR can help speed up and reduce costs in prototyping by visualizing the designs over the real world and allowing designers, workers, and potential customers to walk around and through the model and manipulate its three dimensions.

AR for training

AR has become a valuable asset in education and training. It allows us to visualize concepts in three dimensions over an image or directly in the room so that we can access the information in a quicker and more dynamic way (just pointing with the camera instead of looking through a book or searching the internet for the information) or to create deeper personal projects (giving life to a painting, adding instructions or extra information over a handcrafted project, creating an animated presentation, and so on).

AR for marketing

Using AR, the user is in charge of their experience and can interact with it. Now, brands go beyond the wow effect and create functional experiences to promote and sell their products. Examples of this include augmented catalogs, which show the products in 3D over their flat images; virtual mirrors, where you can buy the glasses or complements you are trying on AR; packages that come to life to explain the elements inside the fabrication process; and so on.

AR for retail

Retail is one of the fields where AR offers a wider range of possibilities, from satisfying and engaging the consumer in order to reduce returned products, to linking products with social media or personalizing the shopping experience. Some examples of this include trying products before buying them so that users can visualize clothes, shoes, glasses, or even makeup over themselves before actually buying the products. By doing this, they can see how a product such as a piece of furniture, art, or even wall paint looks on their homes with AR.

Other examples include seeing extra information about a product, such as comments and reviews, before buying a product, receiving geopositioned information and discounts from the stores on it, and in supermarkets or big stores, orienting customers through sections to the product they want.

AR for automation

AR is the natural interface and connection to the Internet of Things (IoT) and Big Data. It allows workers to visualize and interact with the data coming from the machines and sensors of a factory in an easy and attractive way, either using mobile devices or AR headsets.

AR use in automation can go from the facial recognition of an employee to getting access to a concrete machine, to real-time on-site surveillance of the production process or remote access and control to the system through AR glasses.

AR for tourism

The thing that's attractive about AR is the visual content it displays over the real world. That makes this technology perfect for enhancing traveling experiences, from showing skyline information to making animals in a museum come to life or even translating signs and guides in real-time. Some of the most common uses of AR in tourism include serving as a live guide in the streets of a new city, where a user can go around the city while the AR app is showing them where the most interesting points to see are; showing attractions and points of interest (POIs) over a map when a user points at a map with the camera; giving all kinds of extra information on paintings, sculptures, or monuments so that when a user points at a painting it can show a video about the author, the place and time where it was painted, or even make it come to life in a 3D simulation. Apart from all these experiences, when AR is combined with other immersive technologies, such as virtual worlds or 360º videos, the experience goes one step ahead, allowing users to visit several places at the same time (for example, in a museum network, while visiting one of them, to be able to virtually visit the others).

In this section, we have looked at several examples of how we can use AR for very different fields, from visualizing a prototype before building it, to attracting customers or adding value to art pieces. Now, you have a better understanding of the possibilities AR offers in the world of enterprises and how to apply it according to the needs of each sector.

 

Summary

In this chapter, we have taken our first look at AR by learning what it is, where it comes from, and what type of software and elements we need in order to create it. We have learned about the names and main features of the SDKs we are going to use in this book and are now ready to start creating our own AR projects.

In the next chapter, we will introduce Untiy 3D, a 3D tool that can be used as the base for nearly any AR project since most AR SDKs are compatible with it. You will learn about how it works and create a simple example that will be the basis for the AR projects that follow.

About the Authors

  • Jorge R. López Benito

    Jorge R. López Benito is a tech entrepreneur passionate about emerging and exponential technologies with more than 10 years of experience in researching and building Augmented Reality (AR), Virtual Reality (VR), and Artificial Intelligence (AI) products. In 2010, he became the co-founder and CEO of CreativiTIC, which focused on the research of disruptive technologies, new business models, and market innovation trends. He has been involved in several R&D projects throughout his career, including FP7 and H2020 European projects, and because of that, he was named as a Key Innovator in AR & Deep Learning Technologies by the European Commission; whom he also supports as an independent expert on interactive technologies to explore emerging trends and identify opportunities for impact across the industries, and also as a Jury Member at the European Innovation Council (EIC) Accelerator.

    Browse publications by this author
  • Enara Artetxe González

    Enara Artetxe González is the co-founder and CTO of CreativiTIC, a company dedicated to Mixed Reality (MR) technologies. With a bachelor's degree in telecommunications engineering, she has been working with Augmented Reality (AR) and Virtual Reality (VR) technologies for more than 10 years, developing innovative solutions for desktop, mobile, web, and head-mounted display (HMD) platforms in fields such as industry, healthcare, and education. She has participated in several R&D projects including FP7 and H2020 European research projects, and because of that, she was named as a Key Innovator in AR & Deep Learning Technologies by the European Commission. She actively supports vocational actions to involve girls in science and technology through STEAM methodologies.

    Browse publications by this author
Book Title
Unlock this book and the full library for FREE
Start free trial