Unity 2018 Augmented Reality Projects: Build four immersive and fun AR applications using ARKit, ARCore, and Vuforia

AR is augmented reality. Augmented reality is taking some form of reality and changing it in a specific way to enhance the experience. What augmented reality typically refers to is as follows:

Sound perception

The text in this image is not important. We are just showcasing the video perception overlay

The text in this image is not important. We are just showcasing the Graphic perception overlay

Haptic perception

GPS perception

This means that we could boost visual graphics of an object and view it in a different perspective than what we are accustomed to, or we could add something that isn't there. The video is a little bit different, as it requires the software to interface with specialized hardware, such as glasses, cell phones, HUDs, and other devices.

We could boost the auditory aspect of the world around us. We could take words we see in one language and have it say those words in another, or we could pick up those faint sounds that we always hear and tune out, then amplify them and really bring them to the forefront.

Haptic perception is a little more difficult, but it could be achieved with sensors that emulate touch. We could make something vibrate softly or harshly to simulate various effects, or we could make the game or application entirely touch- or motion-sensor-based. There are many other things we could use for haptic perception in applications or games. It is a field that is constantly being researched and expanded upon.

For GPS data, we can use the user's location to know where the user is in the app or game world. Another use for GPS data would be to know if something of interest to the user should be shown to them.

Because Unity3D loves to take care of most of the finer details of implementation for us, we don't have to worry too much about incorporating DLLs (dynamic-link libraries) or writing the wrapper classes to work with the majority of the popular AR and VR devices. There are exceptions to this rule, depending on the platform and whether the engine has been updated to specifically work with those devices.

Android and iOS are the most popular devices for applications and games to have AR incorporated into them, however, the various tech giants have been working hard to add more and more devices into the mix, with varying amounts of success. Some of these will obviously not have implementation with Unity3D, although you can write wrapper classes as previously mentioned, however, that is outside the scope of this book.

An incomplete list of AR devices

Let's take a quick look at some AR-capable devices. This should give us a slightly better idea of the different types of devices we can use and deploy to:

• The Meta 2 is a head-mounted display headset that uses a sensory array for hand interactions and positional tracking, it has a visual field view of 90 degrees, and a resolution of 2560 x 1440:

Meta 2

• AR displays can be rendered on devices resembling eyeglasses, such as the Google Glass:

Google Glass

• Another such device is the HoloLens:

HoloLens

• There is something called a head-up display, which is commonly referred to as a HUD. It is an alternate implementation of AR displays:

HUD

• There are many new devices being researched on and created all the time. Augmented reality is still in its infancy.

All throughout this book, we will create applications and games that are inspired by the definition of AR. Since there are four main aspects of AR, we will use those four aspects and create a specific application or game for that chapter. Since we have a finite number of sensors that are available for Android and iOS devices, some sensors will be utilized in multiple projects.

ARCore is a platform for building augmented reality apps for Android devices. ARCore uses three key technologies to integrate virtual content with the world through the camera. It uses motion tracking, environmental understanding, and light estimation. ARCore works by tracking the position of the device as it moves and builds its own understanding of the real world. It is able to identify interesting points and readings from the phone's sensors and has the ability to determine both the position and the orientation of the phone as it moves. ARCore only has a few supported devices at the present time, which are as follows:

If you don't own one of these devices, you are stuck using the Android Emulator for testing purposes. This is a very obvious downside, since not everyone owns one of these specific phones; additionally, the Android Emulator is an experimental piece of software and is subject to change often. The upside is that ARCore works with Unity3D and Unreal Engine as well as native to Android devices using the Java programming language.

ARKit, which was introduced in iOS 11, is a framework for easily creating augmented reality projects for the iPhone and iPad. ARKit's features include:

 The downsides to ARKit are that it is an experimental software and subject to change often, and that it requires the use of the Apple iPhone X to take full advantage of the TrueDepth Camera. You cannot compile this on Windows for Mac, so it is a requirement to have a macOS to even test the code properly. The pros, however, are that ARKit works with Unity3D and Unreal Engine and can utilize the A9, A10, and A11 Apple processors. In other words, it works with the iPhone 6S and beyond.

Vuforia is one of the most popular platforms to help you work with augmented reality development. It supports the following:

Vuforia is able to do many different things, such as recognition of different types of visual objects (such as boxes, cylinders, and planes), text and environment recognition, and VuMark, which is a combination of picture and QR code. Also, using the Vuforia Object Scanner, you can scan and create object targets. The recognition process can be implemented using the database (local or cloud storage). The Unity plugin is simple to integrate and very powerful. All plugins and functionalities of the platform are free to use but include the Vuforia watermarks.

The limitations just relate to the number of VuMarks and the amount of Cloud recognition:

The obvious downside is that this is not 100% free software, although they do have a developer tier with 1,000 cloud recognitions and 1,000 targets for free per month.

ARToolKit is an open source tracking library for AR projects. It is supported on Android, iOS, Linux, and macOS. ARToolKit has the functionality to utilize the following:

It is fast enough for real-time AR applications. It is also free and open source software, with plugins for Unity and OpenSceneGraph. The downside to this software is that it has a huge variety of functions, so it is difficult to integrate the library, and it takes more time to explore all the available options and settings.

It is now time to set up a Unity3D project for each of the different SDKs that will serve as the basis for later chapters when we use each of them to build an application or a game. Let's start with Vuforia, since it is the simplest one to set up:

Registering on Vuforia

Adding a Vuforia key

Vuforia key info

Now, we are ready to set up Vuforia and get the demo project working appropriately.

As stated previously, Vuforia is fully integrated into Unity3D as of 2017.2, and is a dream to work with once you learn the basics of the SDK. Vuforia is made to strictly deal with the graphics portion of AR. It can recognize images and objects, and it has the ability to interact with the real world, because it uses computer vision. Since Vuforia is built into Unity3D, we will do the installation of Unity with Vuforia all in one go.

If you don't have Unity3D installed on your computer now, let's go ahead and do that:

Vuforia Unity installation

Now that Unity3D has been installed, let's create a brand new Unity project:

Before we go any further, we should know exactly what these options do and how they look when added to the project prior to the license being applied.

AR Camera replaces the standard camera, as it has the base camera component and Audio Listener components. We also see that it has two scripts attached, Vuforia Behavior and Default Initialization ErrorHandler:

The next set of items to talk about is in the Component toolbar. The special components lie within the AR, Scripts, and XR portions of the Component window, as marked in the following screenshot for reference. In order to use them, you have to have a GameObject in the scene and add the component from the toolbar to it. We have World Anchor, Tracked Pose Driver, Spatial Mapping Collider, and Spatial Mapping Renderer available. We should do a deep dive in so that we can know exactly what these items do:

It should be noted that there are items related to Vuforia in the Scripts section, however, we will not cover those here. But, just for the sake of making sure that the items are listed, they are as follows:

In the Inspector pane, we have Vuforia Configuration. The following is a screenshot of it; next, we will define what it does:

Vuforia Configuration allows you to enter your License Key. Clicking on Add License will load the Vuforia Developer landing page. It also allows you to specify what you want Vuforia to be configured to work for, such as a HUD, smartglasses, a webcam, or a smartphone.

Since we are here, let's go ahead and add our app key to Vuforia (see Vuforia Add License for its location):

The text in this screenshot is not important. It shows a sphere that appears on the camera feed when the image is recognized.

ARToolKit is a bit more difficult to set up and get started with.

There are two different ways you can get ARToolKit in a project and ready to develop with. The first option is the easiest, and that is through the Asset Store: https://assetstore.unity.com/packages/tools/artoolkit-6-unitypackage-94863. This is the latest version of ARToolKit in the Asset Store and it will be imported directly into Unity for you. The other option is to go to the main website of ARToolKit: https://github.com/artoolkit/artoolkit5. This allows you to get additional tools and the actual SDK, along with the Unity Package.

For installation into Unity3D, we will go with the second option, since it is less intuitive than the first option:

Inside the ARToolKit5-Unity folder, there are the Example Scenes, Examples, Materials, Resources, Scripts, Changelog, and Readme files and folders:

The basics are out of the way, so we can begin building our "Hello World" with ARToolKit:

Since I am building for Windows, I will go to the first option of Video config and input the following:

 <?xml version="1.0" encoding="utf-8"?>
 <dsvl_input>
 <camera show_format_dialog="false" frame_width="1280" frame_height="720" frame_rate="29.97">
 <pixel_format>
 <RGB32 flip_v="true"/>
 </pixel_format>
 </camera>
 </dsvl_input>

Now, since my computer does not presently have a webcam attached to it, I get an in-game error message, but the code compiles and runs as expected. If you have a webcam attached and it is recognized appropriately, you should have a direct feed from your webcam.

This concludes the Hello World for ARToolKit; we will be revisiting this one later on for more in-depth and fun usage of this SDK.

ARCore and ARKit are fundamentally very similar, but you can't compile ARKit on a Windows environment, which is what I'm presently using. Not to worry; when we get to ARKit, I will be using a macOS to give you a proper view and feel of how things are when using it.

That being said, it is now time to take a deeper look at ARCore.

ARCore was made by Google and is currently in the early preview stages; it hasn't even reached release 1.0 yet, so there are bound to be many changes, some of which could be extremely damaging to existing apps or games.

There are two ways to get the SDK preview for Unity. The first is through a Unity Package file (https://developers.google.com/ar/develop/unity/quickstart-android) and the other is through GitHub (https://github.com/google-ar/arcore-unity-sdk). Now, since I have been having an issue with downloading from Amazon Web Services lately, I will use the second link:

It is key to remember that, if you don't have a Samsung Galaxy 8 or a Google Pixel phone, you will not be able to run proper tests on your device. However, if you also install Android Studio, you do get access to the Android Emulator:

                                  5. XR Settings: We want ARCore Supported to be checked:

                               6. Next up, we want to unzip the SDK or import the package into Unity3D:

Right away, we should see a DLLNotFoundException for tango_client_api2. This is normal and is well-known by the community. It should not cause any errors at runtime, though; it should be fixed in a later release.

ARKit requires the usage of macOS High Sierra, because of the XCode 9 and above requirement for compiling and making changes to the API itself. So, I would highly suggest having a macOS from late 2011 or newer. I am utilizing a Mac Mini 2011 model with 8 GB of RAM, although the standard 4 GB should be plenty. Unity3D does utilize OpenCL/OpenGL extensively, which requires a GFX card capable of utilizing Metal. 2011 and earlier macOSs do not have this ability natively; this could be circumvented by having an external GPU (Radeon RX 480 is currently the only GPU supported officially for this).

With that out of the way, we can begin with installing and configuring ARKit for Unity3D on our macOS.

There are a couple of ways you can install ARKit:

Next up, we have quite a bit to unpack regarding what is actually in this package:

Before we build our own "Hello World" example, we should build the UnityARKitScene.unity to iOS to get a taste of what ARKit is capable of, as it demonstrates all of the basic functionality of ARKit in that scene.

UnityARKitScene is included in the plugin as well as a couple of other example projects. We will compile the UnityARKitScene as our Hello World application.

However, before we do that, we need to talk about the file structure, because those who are not well-versed with compiling to iOS will have some serious issues compiling without further clarification. You may have noticed quite a few items that we have not mentioned that are in the plugin, so let's go ahead and discuss what all of them do before moving on.

\UnityARKitPlugin main directory files:

Plugins\iOS\UnityARKit\NativeInterface cs files:

\Plugins\iOS\UnityARKit\Helpers cs files:

\Plugins\iOS\UnityARKit\Shaders shader files:

\UnityARKitPlugin\Resources file:

UnityARKitPlugin\ARKitRemote files:

UnityARKitPlugin\Examples files:

Now that we have a fundamental understanding of everything inside this package, let's build our Hello World with ARKit.

Building Hello World in ARKit

The first thing we need to do after we open the UnityARKitScene is to set up the build settings:

1. Click on Build Settings and select Player Settings.
2. We want to scroll down to Identification. The Bundle Identifier should be set to com.unity.ARKitHelloTutorial, Version to 0.1, and Build to 10.1. The Automatically Sign checkbox should be checked. Leave the Automatic Signing Team ID settings blank:

3. Build the UnityARKitScene for iOS only. Run Xcode in Debug mode.
4. Only the Development Build checkbox should be checked; everything else should be left as default.
5. Click Build. I will save the file as iOSTest in my data drive within a folder called iOS:

6. Building should not take very long at all, perhaps around two minutes for a first build.
7. The next thing we want to do is open the folder we saved the project in and open the .xcodeproj file in Xcode:

8. Let's take a look at the Base project you will see in Xcode:

9. The first thing we want to check is the Identity tab to make sure that these settings are the same as in Unity3D:

10. Now, we need to look at the Signing subsection:

11. We need to make sure to add our Personal Team name to it, which you can get by signing in to your Apple Developer account and clicking on the arrow for the team you want to use:

12. Deployment Info is next. The Deployment Target needs to be changed to 11.2. Devices should be set to iPhone only. The Main Interface is LaunchScreen-iPhone.xib:

13. Click on Build Settings at the top, as there are a few settings we need to change here as well.
14. In Architectures, ARCHS for iPhone should be set to Standard. SDKROOT should be Latest iOS (iOS 11.2). SUPPORTED_PLATFORMS should be iOS:

15. Next up, scroll down to look at signing, and the values should be already set to the correct ones:

16. Now, click on Product and Build:

17. The build should have completed successfully with roughly 47 warnings, which is normal:

18. Now, we can build and test in the simulator. What we want to do is change from the iPhone to one of the simulators in the list, so click on the device list next to your iPhone or whatever device you have:

19. You will see a big list of devices you want to use. This will range from simulations of devices to the iOS device you have connected to your macOS:

20.  Click on the simulation you want to use, and then build and run the application.

 Congratulations! We have completed this Hello World application.