Before the birth of Microsoft Kinect, few people were familiar with the technology of motion sensing. Similar devices have been invented and developed originally for monitoring aerial and undersea aggressors in wars. Then in the non-military cases, motion sensors are widely used in alarm systems, lighting systems and so on, which could detect if someone or something disrupts the waves throughout a room and trigger predefined events. Although radar sensors and modern infrared motion sensors are used more popularly in our life, we seldom notice their existence, and can hardly make use of these devices in our own applications.
But Kinect changed everything from the time it was launched in North America at the end of 2010. Different from most other user input controllers, Kinect enables users to interact with programs without really touching a mouse or a pad, but only through gestures. In a top-level view, a Kinect sensor is made up of an RGB camera, a depth sensor, an IR emitter, and a microphone array, which consists of several microphones for sound and voice recognition. A standard Kinect (for Windows) equipment is shown as follows:
The Kinect device
The Kinect drivers and software, which are either from Microsoft or from third-party companies, can even track and analyze advanced gestures and skeletons of multiple players. All these features make it possible to design brilliant and exciting applications with handsfree user inputs. And until now, Kinect had already brought a lot of games and software to an entirely new level. It is believed to be the bridge between the physical world we exist in and the virtual reality we create, and a completely new way of interacting with arts and a profitable business opportunity for individuals and companies.
In this book, we will try to make an interesting game with the popular Kinect technology for user inputs, with the major components explained gradually in each chapter. As Kinect captures the camera and depth images as video streams, we can also merge this view of our real-world environment with virtual elements, which is called Augmented Reality (AR). This enables users to feel as if they appear and live in a nonexistent world, or something unbelievable exists in the physical earth.
In this chapter, we will first introduce the installation of Kinect hardware and software on personal computers, and then consider a good enough idea compounded of Kinect and augmented reality elements, which will be explained in more detail and implemented in the following chapters.
Note
Before installing the Kinect device on your PCs, obviously you should buy Kinect equipment first. In this book, we will depend on Kinect for Windows or Kinect for Xbox 360, which can be learned about and bought at:
http://www.microsoft.com/en-us/kinectforwindows/
http://www.xbox.com/en-US/kinect
Please note that you don't need to buy an Xbox 360 at all. Kinect will be connected to PCs so that we can make custom programs for it. An alternative choice is Kinect for Windows, which is located at:
http://www.microsoft.com/en-us/kinectforwindows/purchase/
The uses and developments of both will be of no difference for our cases.
It is strongly suggested that you have a Windows 7 operating system or higher. It can be either 32-bit or 64-bit and with dual-core or faster processors.
Note
Linux developers can also benefit from third-party drivers and SDKs to manipulate Kinect components, which will be introduced in the Appendix, Where to Go from Here, of this book.
Before we start to discuss the software installation, you can download both the Microsoft Kinect SDK and the Developer Toolkit from:
http://www.microsoft.com/en-us/kinectforwindows/develop/developer-downloads.aspx
In this book, we prefer to develop Kinect-based applications using Kinect SDK Version 1.5 (or higher versions) and the C++ language. Later versions should be backward compatible so that the source code provided in this book doesn't need to be changed.
After we have downloaded the SDK and the Developer Toolkit, it's time for us to install them on the PC and ensure that they can work with the Kinect hardware. Let's perform the following steps:
Run the setup executable with administrator permissions. Select I agree to the license terms and conditions after reading the License Agreement.
The Kinect SDK setup dialog
Follow the steps until the SDK installation has finished. And then, install the toolkit following similar instructions.
The hardware installation is easy: plug the ends of the cable into the USB port and a power point, and plug the USB into your PC. Wait for the drivers to be found automatically.
Now, start the Developer Toolkit Browser, choose Samples: C++ from the tabs, and find and run the sample with the name Skeletal Viewer.
You should be able to see a new window demonstrating the depth/skeleton/color images of the current physical scene, which is similar to the following image:
The depth (left), skeleton (middle), and color (right) images read from Kinect
We chose to set up the SDK software at first so that it will install the motor and camera drivers, the APIs, and the documentations, as well as the toolkit including resources and samples onto the PC. If the operation steps are inversed, that is, the hardware is connected before installing the SDK, your Windows OS may not be able to recognize the device. Just start the SDK setup at this time and the device should be identified again during the installation process.
But before actually using Kinect, you still have to ensure there is nothing between the device and you (the player). And it's best to keep the play space at least 1.8 m wide and about 1.8 m to 3.6 m long from the sensor. If you have more than one Kinect device, don't keep them face-to-face as there may be infrared interference between them.
Note
If you have multiple Kinects to install on the same PC, please note that one USB root hub can have one and only one Kinect connected. The problem happens because Kinect takes over 50 percent of the USB bandwidth, and it needs an individual USB controller to run. So plugging more than one device on the same USB hub means only one of them will work.
The depth image at the left in the preceding image shows a human (in fact, the author) standing in front of the camera. Some parts may be totally black if they are too near (often less than 80 cm), or too far (often more than 4 m).
Note
If you are using Kinect for Windows, you can turn on Near Mode to show objects that are near the camera; however, Kinect for Xbox 360 doesn't have such features.
You can read more about the software and hardware setup at:
http://www.microsoft.com/en-us/kinectforwindows/purchase/sensor_setup.aspx
Now it's time for us to define the goal we are going to achieve in this book. As a quick but practical guide for Kinect and augmented reality, we should be able to make use of the depth detection, video streaming, and motion tracking functionalities in our project. 3D graphics APIs are also important here because virtual elements should also be included and interacted with irregular user inputs (not common mouse or keyboard inputs).
A fine example is the Fruit Ninja game, which is already a very popular game all over the world. Especially on mobile devices like smartphones and pads, you can see people destroy different kinds of fruits by touching and swiping their fingers on the screen.
With the help of Kinect, our arms can act as blades to cut off flying fruits, and our images can also be shown along with the virtual environment so that we can determine the posture of our bodies and position of our arms through the screen display.
Unfortunately, this idea is not fresh enough for now. Already, there are commercial products with similar purposes available in the market; for example:
http://marketplace.xbox.com/en-US/Product/Fruit-Ninja-Kinect/66acd000-77fe-1000-9115-d80258410b79
But please note that we are not going to design a completely different product here, or even bring it to the market after finishing this book. We will only learn how to develop Kinect-based applications, work in our own way from the very beginning, and benefit from the experience in our professional work or as an amateur. So it is okay to reinvent the wheel this time, and have fun in the process and the results.
Kinect, which is a portmanteau of the words "kinetic" and "connect", is a motion sensor developed and released by Microsoft. It provides a natural user interface (NUI) for tracking and manipulating handsfree user inputs such as gestures and skeleton motions. It can be considered as one of the most successful consumer electronics device in recent years, and we will be using this novel device to build the Fruit Ninja game in this book.
We will focus on developing Kinect and AR-based applications on Windows 7 or higher using the Microsoft Kinect SDK 1.5 (or higher) and the C++ programming language. Mainly, we have introduced how to install Kinect for Windows SDK in this chapter. Linux and Mac OS X users can first read the Appendix, Where to Go from Here, which provides an alternative method to call Kinect functionalities on other systems. Developers of .NET, processing, or other language tools may also find useful resources in the last chapter of this book.
In the next chapter, we will learn how to start and shut down the Kinect device in our applications, and prepare a basic framework for further development.
