Working sample for controlling the mouse by hand

Exclusive offer: get 50% off this eBook here
OpenNI Cookbook

OpenNI Cookbook — Save 50%

Learn how to write NIUI-based applications and motion-controlled games with this book and ebook

$26.99    $13.50
by Soroush Falahati | August 2013 | Cookbooks Open Source

This article by Soroush Falahati, the author of OpenNI Cookbook explains a working sample for controlling the mouse by hand. Here, we will write a working example for using nite::HandTracker.

This article will show you how to control the position of the mouse cursor using the NiTE hand tracker feature and how to simulate a click event.

We are going to use the Hand Raise gesture for the tracking of hands and the Click (push) gesture for the mouse click. Also, because of the low resolution of the depth output compared to the resolution of the monitor, which results in low accuracy of the hand tracker, we decided to move the mouse depending on its distance with the starting point. This is similar to how you control your mouse with a joystick.

This sample is Windows only because it uses the Microsoft Windows API.

(For more resources related to this topic, see here.)

Getting ready

Create a project in Visual Studio and prepare it for working with OpenNI and NiTE.

How to do it...

  1. Copy ReadLastCharOfLine() and HandleStatus() functions to the top of your source code (just below the #include lines).
  2. Then add following lines of code:

    class MouseController : public nite::HandTracker::NewFrameListener { private: float startPosX, startPosY; int curX, curY; nite::HandId handId; RECT desktopRect; public: MouseController(){ startPosX = startPosY = -1; POINT curPos; if (GetCursorPos(&curPos)) { curX = curPos.x; curY = curPos.y; }else{ curX = curY = 0; } handId = -1; const HWND hDesktop = GetDesktopWindow(); GetWindowRect(hDesktop, &desktopRect); } void onNewFrame(nite::HandTracker& hTracker){ nite::Status status = nite::STATUS_OK; nite::HandTrackerFrameRef newFrame; status = hTracker.readFrame(&newFrame); if (!HandleStatus(status) || !newFrame.isValid()) return; const nite::Array<nite::GestureData>& gestures = newFrame.getGestures(); for (int i = 0; i < gestures.getSize(); ++i){ if (gestures[i].isComplete()){ if (gestures[i].getType() == nite::GESTURE_CLICK){ INPUT Input = {0}; Input.type = INPUT_MOUSE; Input.mi.dwFlags = MOUSEEVENTF_LEFTDOWN | MOUSEEVENTF_LEFTUP; SendInput(1, &Input, sizeof(INPUT)); }else{ nite::HandId handId; status = hTracker.startHandTracking( gestures[i].getCurrentPosition(), &handId); } } } const nite::Array<nite::HandData>& hands = newFrame.getHands(); for (int i = hands.getSize() -1 ; i >= 0 ; --i){ if (hands[i].isTracking()){ if (hands[i].isNew() || handId != hands[i].getId()){ status = hTracker.convertHandCoordinatesToDepth( hands[i].getPosition().x, hands[i].getPosition().y, hands[i].getPosition().z, &startPosX, &startPosY); handId = hands[i].getId(); if (status != nite::STATUS_OK){ startPosX = startPosY = -1; } }else if (startPosX >= 0 && startPosY >= 0){ float posX, posY; status = hTracker.convertHandCoordinatesToDepth( hands[i].getPosition().x, hands[i].getPosition().y, hands[i].getPosition().z, &posX, &posY); if (status == nite::STATUS_OK){ if (abs(int(posX - startPosX)) > 10) curX += ((posX - startPosX) - 10) / 3; if (abs(int(posY - startPosY)) > 10) curY += ((posY - startPosY) - 10) / 3; curX = min(curX, desktopRect.right); curX = max(curX, desktopRect.left); curY = min(curY, desktopRect.bottom); curY = max(curY, desktopRect.top); SetCursorPos(curX, curY); } } break; } } } };

  3. Then locate the following line:

    int _tmain(int argc, _TCHAR* argv[]) {

  4. Add the following inside this function:

    nite::Status status = nite::STATUS_OK; status = nite::NiTE::initialize(); if (!HandleStatus(status)) return 1; printf("Creating hand tracker ...\r\n"); nite::HandTracker hTracker; status = hTracker.create(); if (!HandleStatus(status)) return 1; MouseController* listener = new MouseController(); hTracker.addNewFrameListener(listener); hTracker.startGestureDetection(nite::GESTURE_HAND_RAISE); hTracker.startGestureDetection(nite::GESTURE_CLICK); printf("Reading data from hand tracker ...\r\n"); ReadLastCharOfLine(); nite::NiTE::shutdown(); openni::OpenNI::shutdown(); return 0;

How it works...

Both the ReadLastCharOfLine() and HandleStatus() functions are present here too. These functions are well known to you and don't need any explanation.

Then in the second part, we declared a class/struct that we are going to use for capturing the new data available event from the nite::HandTracker object.

But the definition of this class is a little different here. Other than the onNewFrame() method, we defined a number of variables and a constructor method for this class too. We also changed its name to MouseController to be able to make more sense of it.

class MouseController : public nite::HandTracker::NewFrameListener { private: float startPosX, startPosY; int curX, curY; nite::HandId handId; RECT desktopRect;

As you can see, our class is still a child class of nite::HandTracker::NewFrameListener because we are going use it to listen to the nite::HandTracker events. Also, we defined six variables in our class. startPosX and startPosY are going to hold the initial position of the active hand whereas curY and curX are going to hold the position of the mouse when in motion. The handId variable is responsible for holding the ID of the active hand and desktopRecthold for holding the size of the desktop so that we can move our mouse only in this area. These variables are all private variables; this means they will not be accessible from the outside of the class. Then we have the class's constructor method that initializes some of the preceding variables. Refer to the following code:

public: MouseController(){ startPosX = startPosY = -1; POINT curPos; if (GetCursorPos(&curPos)) { curX = curPos.x; curY = curPos.y; }else{ curX = curY = 0; } handId = -1; const HWND hDesktop = GetDesktopWindow(); GetWindowRect(hDesktop, &desktopRect); }

In the constructor, we set both startPosX and startPosY to -1 and then store the current position of the mouse in the curX and curY variables. Then we set the handId variable to -1 to know-mark that there is no active hand currently, and retrieve the value of desktopRect using two Windows API methods, GetDesktopWindow() and GetWindowRect().

The most important tasks are happening in the onNewFrame() method. This method is the one that will be called when new data becomes available in nite::HandTracker; after that, this method will be responsible for processing this data.

As the running of this method means that new data is available, the first thing to do in its body is to read this data. So we used the nite::HandTracker::readFrame() method to read the data from this object:

void onNewFrame(nite::HandTracker& hTracker){ nite::Status status = nite::STATUS_OK; nite::HandTrackerFrameRef newFrame; status = hTracker.readFrame(&newFrame);

When working with nite::HandTracker, the first thing to do after reading the data is to handle gestures if you expect any. We expect to have Hand Raise to detect new hands and click gesture to perform the mouse click:

const nite::Array<nite::GestureData>& gestures = newFrame.getGestures(); for (int i = 0; i < gestures.getSize(); ++i){ if (gestures[i].isComplete()){ if (gestures[i].getType() == nite::GESTURE_CLICK){ INPUT Input = {0}; Input.type = INPUT_MOUSE; Input.mi.dwFlags = MOUSEEVENTF_LEFTDOWN | MOUSEEVENTF_LEFTUP; SendInput(1, &Input, sizeof(INPUT)); }else{ nite::HandId handId; status = hTracker.startHandTracking( gestures[i].getCurrentPosition(), &handId); } } }

As you can see, we retrieved the list of all the gestures using nite::HandTrackerFrameRef::getGestures() and then looped through them, searching for the ones that are in the completed state. Then if they are the nite::GESTURE_CLICK gesture, we need to perform a mouse click. We used the SendInput() function from the Windows API to do it here. But if the recognized gesture wasn't of the type nite::GESTURE_CLICK, it must be a nite::GESTURE_HAND_RAISE gesture; so, we need to request for the tracking of this newly recognized hand using the nite::HandTracker::startHandTracking() method.

The next thing is to take care of the hands being tracked. To do so, we first need to retrieve a list of them using the nite::HandTrackerFrameRef::getHands() method and then loop through them. This can be done using a simple for loop as we used for the gestures. But as we want to read this list in a reverse order, we need to use a reverse for loop. The reason we need to read this list in the reverse order is that we always want the last recognized hand to control the mouse:

const nite::Array<nite::HandData>& hands = newFrame.getHands(); for (int i = hands.getSize() - 1 ; i >= 0 ; --i){

Then we need to make sure that the current hand is under-tracking because we don't want an invisible hand to control the mouse. The first hand being tracked is the one we want, so we will break the looping there, of course, after the processing part, which we will remove from the following code to make it clearer.

if (hands[i].isTracking()){ . . . break; }

Speaking of processing, in the preceding three lines of code (with periods) we have another condition. This condition is responsible for finding out if this hand is the same one that had control of the mouse in the last frame. If it is a new hand (either it is a newly recognized hand or it is a newly active hand), we need to save its current position in the startPosX and startPosY variables.

if (hands[i].isNew() || handId != hands[i].getId()){ status = hTracker.convertHandCoordinatesToDepth( hands[i].getPosition().x, hands[i].getPosition().y, hands[i].getPosition().z, &startPosX, &startPosY); handId = hands[i].getId(); if (status != nite::STATUS_OK){ startPosX = startPosY = -1; }

If it was the same hand, we have another condition.

Do we have the startPosX and startPosY variables already or do we not have them yet? If we have them, we can calculate the mouse's movement. But first we need to calculate the position of the hand relative to the depth frame.

}else if (startPosX >= 0 && startPosY >= 0){ float posX, posY; status = hTracker.convertHandCoordinatesToDepth( hands[i].getPosition().x, hands[i].getPosition().y, hands[i].getPosition().z, &posX, &posY);

Once the process of conversation ends, we need to calculate the new position of the mouse depending on how the hand's position changes. But we want to define a safe area for it to be static when small changes happen. So we calculate the new position of the mouse only if it has moved by more than 10 pixels in our depth frame:

if (status == nite::STATUS_OK){ if (abs(int(posX - startPosX)) > 10) curX += ((posX - startPosX) - 10) / 3; if (abs(int(posY - startPosY)) > 10) curY += ((posY - startPosY) - 10) / 3;

As you can see in the preceding code, we also divided the changes by 3 because we didn't want it to move too fast.

But before setting the position of the mouse, we need to first make sure that the new positions are in the screen view port using the desktopRect variable:

curX = min(curX, desktopRect.right); curX = max(curX, desktopRect.left); curY = min(curY, desktopRect.bottom); curY = max(curY, desktopRect.top);

After calculating everything, we can set the new position of the mouse using SetCursorPos() from the Windows API:

SetCursorPos(curX, curY);

Step three and four are not markedly differently. In this step, we have the initialization process; this includes the initialization of NiTE and the creation of the nite::HandTracker variable.

status = nite::NiTE::initialize(); . . . nite::HandTracker hTracker; status = hTracker.create();

Then we should add our newly defined structure/class as a listener to nite::HandTracker so that nite::HandTracker can call it later when a new frame becomes available:

MouseController* listener = new MouseController(); hTracker.addNewFrameListener(listener);

Also, we need to have an active search for a hand gesture because we need to locate the position of the hands and we also have to search for another gesture for the mouse click. So we need to call the nite::HandTracker::startGestureDetection() method twice for both the Click (also known as push) and Hand Raise gestures here:

hTracker.startGestureDetection(nite::GESTURE_HAND_RAISE); hTracker.startGestureDetection(nite::GESTURE_CLICK);

At the end, we will wait until the user presses the Enter key to end the app. We do nothing more in our main thread except just waiting. Everything happens in another thread.

ReadLastCharOfLine(); nite::NiTE::shutdown(); openni::OpenNI::shutdown(); return 0;

Summary

In this article, we learnt how to write a working example for using nite::HandTracker, controlled the position of the mouse cursor using the NiTE hand tracker feature, and simulated a click event

Resources for Article:


Further resources on this subject:


OpenNI Cookbook Learn how to write NIUI-based applications and motion-controlled games with this book and ebook
Published: July 2013
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:

About the Author :


Soroush Falahati

Soroush Falahati is a Microsoft MCPD certificated C# developer of Web and Windows applications, now preparing for a new MCSD certification from Microsoft. He started programming at the age of 13 with VB5 and then continued to VB.Net, C#, C++, C for microcontrollers, as well as scripting languages such as PHP and JavaScript.

He is currently the owner of an e-commerce company that uses web applications and smart phone apps as primary advantages over other competitors.

As a hobby, Soroush supports robotic teams by voluntarily training them on how to program microcontrollers.

Books From Packt


OpenCV Computer Vision with Python
OpenCV Computer Vision with Python

OpenCV 2 Computer Vision Application Programming Cookbook
OpenCV 2 Computer Vision Application Programming Cookbook

Kinect for Windows SDK Programming Guide
Kinect for Windows SDK Programming Guide

Augmented Reality with Kinect
Augmented Reality with Kinect

Kinect in Motion – Audio and Visual Tracking by Example
Kinect in Motion – Audio and Visual Tracking by Example

OpenSceneGraph 3 Cookbook
OpenSceneGraph 3 Cookbook

OpenSceneGraph 3.0: Beginner's Guide
OpenSceneGraph 3.0: Beginner's Guide

OpenGL 4.0 Shading Language Cookbook
OpenGL 4.0 Shading Language Cookbook


Your rating: None Average: 1.3 (3 votes)

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
5
e
K
9
d
U
Enter the code without spaces and pay attention to upper/lower case.
Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software