Download code from GitHub
We live in very exciting times. Technology is changing at a pace so rapid that it is becoming near impossible to keep up with these new frontiers as they arrive; they seem to arrive on a daily basis now. Moore's Law continues to stand, meaning that technology is getting smaller and more powerful at a constant rate. As I said, very exciting.
One of these new emerging technologies that finally is reaching a place more material than science fiction stories is Augmented or Mixed Reality. Imagine a world where our communication and entertainment devices are worn, and the digital tools we use, as well as the games we play, are holographic projections in the world around us. These holograms know how to interact with our world and change themselves to fit our needs. Microsoft has led the charge by releasing such a device: the HoloLens.
The Microsoft HoloLens changes the paradigm of what we know as personal computing--we can now have our Word window up on the wall (this is how I am typing right now), we can have research material floating around it, and we can have our communication tools, such as Gmail and Skype, in the area as well. We are finally no longer trapped by a virtual desktop, on a screen, sitting at a physical desktop; we aren't even trapped in the confines of a room anymore.
The HoloLens is a first of its kind head-worn, standalone computer with a sensor array, which includes microphones and multiple types of camera, a spatial sound speaker array, a light projector, and an optical waveguide.
The HoloLens is not only a wearable computer, it is also a complete replacement for the standard 2D display. It has the capability of using holographic projection to create multiple screens throughout an environment and fully 3D-rendered objects. With the HoloLens sensor array, these holograms can fully interact with the environment you are in.
The sensor array allows the HoloLens to see the world around it, to see the input from the user's hands, and for it to hear voice commands. Although Microsoft has been very quiet about what the entire sensor array includes, we have a good general idea about the components used in the sensor array; let's take a look at them:
- One IMU: The Inertia Measurement Unit (IMU) is a sensor array that includes an accelerometer, a gyroscope, and a magnetometer. This unit handles head orientation tracking and compensates for the drift that comes from the Gyroscope's eventual lack of precision.
- Four environment understanding sensors: These together form the spatial mapping that the HoloLens uses to create a mesh of the world around the user.
- One depth camera: This is also known as a structured light 3D scanner. This device is used for measuring the 3D shape of an object using projected light patterns and a camera system. Microsoft first used this type of camera inside the Kinect for the Xbox 360 and Xbox One.
- One ambient light sensor:Ambient light sensors or photosensors are used for ambient light sensing and proximity detection.
- 2 MP photo/HD video camera: This is used for taking pictures and video.
- Four-microphone array: These do a great job of listening to the user and not the sounds around them. Voice is one of the primary input types with HoloLens.
Putting all of these elements together forms a Holographic computer that allows the user to see, hear, and interact with the world around them in new and unique ways:
The HoloLens development environment breaks down into two primary tools, Unity and Visual Studio. Unity is the 3D environment that we will do most of our work in; this includes adding holograms, creating user interface elements, adding sound and particle systems, and other things that bring a 3D program to life.
Visual Studio on the other hand is the glue that makes everything work. Here, we write scripts or machine code to make our 3D creations come to life and add a level of control and immersion that Unity cannot produce on its own.
Unity is a software framework designed to speed up the creation of games and 3D-based software. Generally speaking, Unity is known as a game engine but the more apparent the holographic world becomes, the more we will use such a development environment for different kinds of applications.
Unity is an application that allows us to take 3D models, 2D graphics, particle systems, and sound to make them interact with each other and our user. Many elements are dragged and dropped and plugged and played; what you see is what you get. This can simplify the iteration and testing process. As developers, we most likely do not want to build and compile little changes we make in the development process forever. This allows us to see the changes in context to ensure that they work; then once we hit a group of changes, we can test them on the HoloLens ourselves. This does not work for every aspect of HoloLens-Unity development, but it does work for a good 80%-90%.
Microsoft Visual Studio Community is a great, free Integrated Development Environment (IDE). Here, we use programming languages, such as C# or JavaScript, to code changes in the behavior of objects, and generally make things happen inside of our programs.
The HoloToolkit-Unity is a repository of samples, scripts, and components to help speed up the process of development. This covers a large selection of areas in HoloLens development, such as the following:
- Input:Gaze, gesture, and voice are the primary ways in which we interact with the HoloLens.
- Sharing:The sharing repository helps allow users to share holographic spaces and connect to each other via the network.
- Spatial Mapping:This is how the HoloLens sees our world. A large 3D mesh of our space is generated and give our holograms something to interact with or bounce off of.
- Spatial Sound:The speaker array inside the HoloLens does an amazing job of giving the illusion of space. Objects behind us seem like they are behind us.
The HoloLens emulator is an extension to Visual Studio that will simulate how a program will run on the HoloLens. This is great for those who want to get started with HoloLens development but do not have an actual HoloLens, yet. This software does require the use of Microsoft Hyper-V, a feature only available inside the Windows 10 Pro operating system. Hyper-V is a virtualization environment, which allows the creation of a virtual machine. This virtual machine emulates the specific hardware, so one can test without the actual hardware.
This collection of tools adds IntelliSense and debugging features to Visual Studio; if you use Visual Studio and Unity, these are a must have:
- IntelliSense:An intelligent code completion tool for Microsoft Visual Studio. This is designed to speed up many processes when writing code. The version that comes with Visual Studios tools for Unity has Unity-specific updates.
- Debugging: Before this extension existed, debugging Unity apps proved to be a little tedious. With this tool, we can now debug Unity applications inside Visual Studio, thus speeding the bug squashing process considerably.
The following are some the useful tools that are required:
- Image editor: Photoshop and Gimp both are good examples of programs that allow us to create 2D UI elements and textures for objects in our apps.
- 3D modeling software: 3D Studio Max, Maya, and Blender are all programs that allow us to make 3D objects that can be imported in Unity.
- Sound editing software: There are a few resources for free sounds on the Web. With that in mind, Sound Forge is a great tool for editing those sounds, layering sounds together to create new sounds.
Now, we will get our development environment installed and set up so that we can start building the next killer app for HoloLens. We will start with our main tools, Visual Studio and Unity HoloLens technical preview, and then add the extensions that will help speed things along.
Microsoft has made the installation process for Visual Studio very simple. Just be warned that it can take a while to download and install it:
- Click on
Download Community 2015; this is a free version of Visual Studio that works well with Unity and the HoloLens environment. By default, you should be installing at least Visual Studios 2015 Update 3.
- Run
vs_community. exeand complete the installation process. - Create a Microsoft Developers account (this is a requirement for the HoloLens development).
- The first time you run VS2015, you will be prompted to input your account credentials.
Now, we will install Unity HoloLens technical preview. At the time of writing this, version 5.4. 0f 3 is the most recent and stable version. This is a custom version of Unity made to work specifically with HoloLens. As a result, it does automate a few processes for HoloLens developers:
To install Unity HoloLens technical preview, we will need to perform the following steps:
- Go to https://unity3d.com/partners/windows/hololens.
- Scroll down to the bottom of the page.
- Click on the
Unity Download Assistant. - Follow the instructions to complete the installation process.
- You will need to create an account with Unity.
In your web browser, go to https://visualstudiogallery.msdn.microsoft.com/8d26236e-4a64-4d64-8486-7df95156aba9. Later, click on the Download button and run vstu2015.exe:
To install the HoloLens emulator, perform the following steps:
- Go to http://go.microsoft.com/fwlink/?LinkID=823018.
- Run
EmulatorSetup.exeonce it finishes downloading. - Install the emulator.
- This installation process can take a while.
To install HoloToolkit-Unity, perform the following steps:
- Enter https://github.com/Microsoft/HoloToolkit-Unity in your browser.
- Click on
Clone or download. - Click on
Download ZIP. - Once the ZIP file is downloaded, unzip the file to your desktop; we will copy this into our project later.
In the Unity3d quick overview, we will have a quick, very general overview of Unity. In the next chapter, we will begin the first stages of our project. With the following overview, what you will learn in the chapter will stick a little bit better:
- Click on the
Unity HoloLensicon to run Unity, as follows:
- If you have a Unity account, enter your account name and password, otherwise you will need to create a new account before you can move forward.
- Type the name of the project--
HoloLens Beginners Guide--into theProject nameinput field. - Ensure that you have 3D selected as the project type and click on
Create project.
When the project is created and the program finishes loading, you should see the following default view:
Here, we can see our main work areas:
- Scene: This is where we view our scene and can make certain types of changes.
- Hierarchy: This is a list of all objects currently in the scene.
- Project Window: Here, we have all the assets of the project. This will list our 3D models, scripts, textures, materials, sounds, prefabs, and all other elements of the programs.
- Inspector: When an object is selected, we will be given many changeable details about that object.
You may notice a few in at the upper part of certain windows. These windows offer other functions, as shown in the following screenshot:
- Console: Here, we can have the debug information presented to us. This allows us to test certain parts of a program to ensure that it is working.
- Game View: When we test our project on our development computer, this is the window that our program will appear in; this is the main camera view.
