Chapter 2. Real Talk – Real Time

When we create an application that acts on gestural input, a major concern is responsiveness. Not only is using a laggy application no fun, but there are many situations in which we desire to process and respond to the hand motion with as little latency as possible. In this chapter we'll discuss the following:

• Blocking

• Describing raw input as simple gestures

• Shared-memory concurrency with pthreads

• Building a MIDI controller for real-time audio applications

With this in mind, we will walk through creating a controller suitable for live musical performances. Because our code will be interacting with music software and hardware, we must use our operating system interfaces for transmitting MIDI (the standard musical control protocol that all devices support), which is not guaranteed to happen without blocking. While using routines that can potentially involve sending messages to hardware, it's generally good to assume that the execution of our program may be paused...

A simple gesture recognizer

One of the handy methods we can call on a Leap::Hand is sphereRadius(), which returns the size of an imaginary ball filling up a hand. The more outstretched your fingers are the larger the radius, and a more clenched fist will produce smaller values.

What does our ball gesture recognizer class look like? Feast your eyes on this:

typedef std::shared_ptr<BallGesture> BallGesturePtr;
   
void BallGesture::recognizedControls(const Leap::Controller &controller, std::vector<ControlPtr> &controls) {
    Leap::Frame frame = controller.frame();
    if (frame.hands().empty())
        return;

If no hands are found in the current frame, there's not much else for us to do here. However, if the user is waving some appendages around, we can do our thing.

    for (int i = 0; i < frame.hands().count(); i++) {
        if (i > 1) break;

If one or two hands are detected in the frame, we might be able to do something useful. We can get extra fancy if we return...

Using the MIDI output

A MIDI client is an entity that can communicate with other MIDI clients, and may have one or more inputs and outputs. Our client will be sort of useless without any I/O, so we create a virtual source into which we can inject our freshly minted MIDI messages. The descriptive string passed in will appear in the MIDI configuration dialogs of any application that supports MIDI, such as my DAW of choice, Ableton Live.

Let's get to the meat of sending a MIDI control message now; we will deal with the packet initialization in a minute. Remember our onControlUpdated() routine? We can use the mapped control value and the controller index number to build a message to transmit.

    midi_control_index ctrlIdx = control->controlIndex();
    midi_control_value ctrlVal = control->mappedValue();

ctrlIdx and ctrlVal correspond to the number of the MIDI control and the output value for that control respectively. For our ball radius gesture, the control indices will be BALL_RADIUS_HAND_1...

Blocking and latency

So, can we play the Leap now? We absolutely can! There is a wealth of free and trial music software available for generating all sorts of noises a quick Google search away. You should be able to select the Leap MIDI control device in your music program's settings, and use its MIDI learn feature to map your ball radius gestures to anything you desire (filter cut-offs, effect parameters, and synthesizer knobs are especially fun).

This is all well and good but there is one very serious, unaddressed issue with our MIDI controller: blocking. The sad truth is that our code is not ready to be used by the general public to give performances because the gesture response time is utterly nondeterministic. In the onControlUpdated() routine described earlier, our call to MIDIReceived() is part of a poorly documented API that may or may not involve communication with blocking subsystems or hardware, and that frankly we know very little about. Depending upon the user's setup, hardware...

Multiprocessing with threads

The solution to our conundrum is to run more than one thread of execution at the same time. CPUs and modern operating systems are very good at dealing with multiple processes and multiple threads of execution within processes, and this suits our purposes nicely. What we really want is one task that is receiving callbacks from the controller and handling the input, with a separate task taking our control messages and outputting them. This would give us a robust setup in which we can easily do anything we want in the output thread, without introducing issues associated with blocking the input processing thread. We could be happily logging output to a file, sending it over a network, outputting MIDI to hardware, controlling GPIO pins, doing expensive calculations or 3D graphics, and not affect the response time of our Leap callbacks.

As with many things in software development, there is more than one way to achieve our desired result of multitasking: POSIX threading...

Refactoring for multithreading

Before continuing further on, perhaps it would be a good time to refactor our virtual MIDI device output functionality into its own class, as the MIDI output routines with the attendant threading code should be encapsulated for maintaining some semblance of clean program structure.

namespace leapmidi {
class Device {
public:
    Device();
    ~Device();
    void init();
    void queueControlMessage(midi_control_index controlIndex, midi_control_value controlValue);

We'll go ahead and create a leapmidi::Device class. Users of our class construct a new Device class and call init() to set up the MIDI virtual device, and then send MIDI control messages using queueControlMessage().

protected:
    void initPacketList();
    void createDevice();

Our earlier MIDI virtual device setup code can go in here.

    pthread_t messageQueueThread;
    void *messageSendingThreadEntry();

This function will be called when our MIDI output thread (kept track of by messageQueueThread) begins...

The producer-consumer race condition

What we have created is a classic producer-consumer pattern, where one thread is producing work and the other is processing the work on the queue. We are trying to make our application take care of the following:

• Respond to events in as close to real-time as possible

• Not lose any events

So, we need to be absolutely certain that we handle new items added to the queue as rapidly as possible. A naive implementation might look something like this:

            - release message queue lock
            - (possible race condition here!)
            - wait for condition variable broadcast
            - check queue again

But that would be utterly wrong. Suppose that queueControlMessage() is called after unlocking our message queue mutex, but before our thread starts waiting for the main Leap processing thread condition variable signal. The condition variable signals are ephemeral in the extreme. If no thread is waiting at the exact moment the signal is sent, we will...

Summary

Multithreaded C++ applications are inherently nontrivial and full of debugging pitfalls. Your best strategy is to abstract the communication between your threads to a simple and well-tested mechanism for safe transportation of messages between your threads. Note that the callbacks used by the Leap library are invoked in separate threads, so take care while sharing data between your callbacks and any other threads your program may be running. This makes it possible for applications to easily receive updates concurrently with the rest of the program's execution, but you are still responsible for appropriately handling any other blocking that may take place in your program.