Processing 2: Creative Coding HOTSHOT

Processing can be extended by contributed libraries. Most of these additional libraries can be installed by navigating to Sketch | Import Library… | Add Library..., as shown in the following screenshot:

In the Library Manager dialog, enter ttslib in the search field to filter the list of libraries.

Click on the ttslib entry and then on the Install button, as shown in the following screenshot, to download and install the library:

To use the new library, we need to import it to our sketch. We do this by clicking on the Sketch menu and choosing Import Library... and then ttslib.

We will now add the setup() and draw() methods to our sketch. We will leave the draw() method empty for now and instantiate a TTS object in the setup() method. Your sketch should look like the following code snippet:

import guru.ttslib.*;

TTS tts;
void setup() {
  tts = new TTS();
}

void draw() {
}

Now we will add a mousePressed() method to our sketch, which will get called if someone clicks on our sketch window. In this method, we are calling the speak() method of the TTS object we created in the setup() method.

void mousePressed() {
  tts.speak("Hello, I am a Computer");
}

Click on the Run button to start the Processing sketch. A little gray window should appear.

Turn on your speakers or put on your headphones, and click on the gray window. If nothing went wrong, a friendly male computer voice named kevin16 should greet you now.

Let's start a new sketch by navigating to File | New.

Add a setup() and a draw() method.

Now add the prepared script to the Processing sketch by navigating to Sketch | Add File and selecting the file you just downloaded.

Add the following line to your setup() method:

void setup() {
  String[] rawLines = loadStrings( "romeo_and_juliet.txt" );
}

If you renamed your text file, change the filename accordingly.

Create a new tab by clicking on the little arrow icon on the right and choosing New Tab.

Name the class Line. This class will hold our text lines and the speaker.

Add the following code to the tab we just created:

public class Line {
  String speaker;
  String text;

  public Line( String speaker, String text ) {
    this.speaker = speaker;
    this.text = text;
  }
}

Switch back to our main tab and add the following highlighted lines of code to the setup() method:

void setup() {
  String[] rawLines = loadStrings( "romeo_and_juliet.txt" );

  ArrayList lines = new ArrayList();
  for ( int i=0; i<rawLines.length; i++) {
    if (!"".equals(rawLines[i])) {
      String[] tmp = rawLines[i].split("#");
      lines.add( new Line( tmp[0], tmp[1].trim() ));
    }
  }

}

We have read our text lines and parsed them into the lines array list, but we still need a class that does something with our text lines. So create another tab by clicking on the arrow icon and choosing New Tab from the menu; name it Drama.

Our Drama class will be a thread that runs in the background and tells each of the speaker objects to read one line of text. Add the following lines of code to your Drama class:

public class Drama extends Thread {
  int current;
  ArrayList lines;
  boolean running;

  public Drama( ArrayList lines ) {
    this.lines = lines;
    current = 0;
    running = false;
  }

  public int getCurrent() {
    return current;
  }

  public Line getLine( int num ) {
    if ( num >=0 && num < lines.size()) {
      return (Line)lines.get( num );
    } else {
      return null;
    }
  }

  public boolean isRunning() {
    return running;
  }
}

Now we add a run() method that gets executed in the background if we start our thread. Since we have no speaker objects yet, we will print the lines on the console and include a little pause after each line.

  public void run() {
    running = true;

    for ( int i =0; i < lines.size(); i++) {
      current = i;
      Line l = (Line)lines.get(i);
      System.out.println( l.text );
      delay( 1 ); 
    }
    running = false;
  }

Switch back to the main sketch tab and add the highlighted code to the setup() method to create a drama thread object, and then feed it the parsed text-lines.

Drama drama;

void setup() {

  String[] rawLines = loadStrings( "romeo_and_juliet.txt" );
  ArrayList lines = new ArrayList();
  for ( int i=0; i<rawLines.length; i++) {
    if (!"".equals(rawLines[i])) {
      String[] tmp = rawLines[i].split("#");
      lines.add( new Line( tmp[0], tmp[1].trim() ));
    }
  }

  drama = new Drama( lines );
}

So far our sketch parses the text lines and creates a Drama thread object. What we need next is a method to start it. So add a mousePressed() method to start the drama thread.

void mousePressed() {
  if ( !drama.isRunning()) {
    drama.start();
  }
}

Now add a little bit of text to the draw() method to tell the user what to do. Add the following code to the draw() method:

void draw() {
  background(255);
  textAlign(CENTER);
  fill(0);

  text( "Click here for Drama", width/2, height/2 );
}

Currently, our sketch window is way too small to contain the text, and we also want to use a bigger font. To change the window size, we simply add the following line to the setup() method:

void setup() {

  size( 800, 400 );
  
  String[] rawLines = loadStrings( "romeo_and_juliet.txt" );
  ArrayList lines = new ArrayList();
  for ( int i=0; i<rawLines.length; i++) {
    if (!"".equals(rawLines[i])) {
      String[] tmp = rawLines[i].split("#");
      lines.add( new Line( tmp[0], tmp[1].trim() ));
    }
  }

  drama = new Drama( lines );
}

To change the used font, we need to tell Processing which font to use. The easiest way to find out the names of the fonts that are currently installed on the computer is to create a new sketch, type the following line, and run the sketch:

    println(PFont.list());

Copy one of the font names you like and add the following line to the Romeo and Juliet sketch:

  void setup() {

    size( 800, 400 );
    textFont( createFont( "Georgia", 24 ));

...

Replace the font name in the code lines with one of the fonts on your computer.

First, we open the sketch from the previous task and start by creating two TTS objects, one for each of our robot actors. Both use the default voice named kevin16, but we change the pitch for our Juliet-Robot.

void setup() {
  size( 800, 400 );
  textFont( createFont( "Georgia", 24 ));

  String[] rawLines = loadStrings( "romeo_and_juliet.txt" );
  ArrayList lines = new ArrayList();
  for ( int i=0; i<rawLines.length; i++) {
    if (!"".equals(rawLines[i])) {
      String[] tmp = rawLines[i].split("#");
      lines.add( new Line( tmp[0], tmp[1].trim() ));
    }
  }

  TTS romeo = new TTS();
  TTS juliet = new TTS();
  juliet.setPitchShift( 2.4 );
  drama = new Drama( lines, romeo, juliet );
}

Switch to the Drama thread and add two variables to our actors.

public class Drama extends Thread {
  TTS romeo;
  TTS juliet;

  int current;
  ArrayList lines;
  boolean running;
  …

We also need to extend the constructor of our Drama class to enable us to add the actors.

  public Drama( ArrayList lines, TTS romeo, TTS juliet ) {
    this.lines = lines;
    this.romeo = romeo;
    this.juliet = juliet;
    current = 0;
    running = false;
  }

In the run() method, we take the current line and choose the actor object depending on the actor variable we added to each line. We also don't need the delay() and the println() methods anymore.

  public void run() {
    running = true;

    for ( int i =0; i < lines.size(); i++) {
      current = i;
      Line l = (Line)lines.get(i);
      if ( "J".equals( l.speaker )) {
        juliet.speak( l.text );
      }
      if ( "R".equals( l.speaker )) {
        romeo.speak( l.text );
      }
    }
    running = false;
  }

Since each of our robots gets his own speaker, we don't want to hear the text on both speakers. We want one robot to use the right one and the other robot, the left one. Fortunately, ttslib provides a speakLeft() and a speakRight() method, which do exactly what we need. So change the two speak() methods to look like the following:

  ...
  if ( "J".equals( l.speaker )) {
      juliet.speakLeft( l.text );
  }
  if ( "R".equals( l.speaker )) {
    romeo.speakRight( l.text );
  }
  ...

Currently, our draw() method is somewhat boring and also somewhat misleading if the drama thread is already running. So we will change it to display five lines of the currently read script. Add an if statement to the draw() method that checks the state of the running variable we added to our Drama thread earlier.

void draw() {
  background(255);
  textAlign(CENTER);
  fill(0);

  if ( !drama.isRunning() ) {
    text( "Click here for Drama", width/2, height/2 );
  } else {
  }
}

Now we add a for loop that displays the previous two lines, the line that's currently being read, and the next two lines of text in our sketch window. We will now change the text alignment so that it matches the speaker of our robot actors. The text will be aligned to the right if our robot uses speakRight(), and it will be aligned to the left if our robot uses speakLeft().

void draw() {
  background(255);
  textAlign(CENTER);
  fill(0);

  if ( !drama.isRunning() ) {
    text( "Click here for Drama", width/2, height/2 );
  }
  else {
    int current = drama.getCurrent();
    for ( int i = -2; i < 3; i ++) {
      Line l = drama.getLine(i + current);
      if ( l != null) {
        if ( "J".equals( l.speaker )) {
          textAlign( LEFT );
          text( l.text, 10, height/2 + i * 30 );
        }
        else {
          textAlign( RIGHT );
          text( l.text, width - 10, height/2 + i * 30 );
        }
      }
    }
  }
}

To show the current line more prominently, we will change the color of the text. We set the color of the current line to black and all other lines to a lighter gray by adding a fill() statement to the for loop.

    for ( int i = -2; i < 3; i ++) {
      fill( abs(i) * 100 );
      Line l = drama.getLine(i + current);
      if ( l != null) {
        if ( "J".equals( l.speaker )) {
          textAlign( LEFT );
          text( l.text, 10, height/2 + i * 30 );
        }
        else {
          textAlign( RIGHT );
          text( l.text, width - 10, height/2 + i * 30 );
        }
      }
    }

Now run your code and click on the sketch window to start the drama thread.

Open http://tcts.fpms.ac.be/synthesis/mbrola.html in your browser and click on Download. Download the MBROLA binary for your platform.

Download the female_us1 voice from the MBROLA site.

Create a folder for the MBROLA binary and unzip the two packages you just downloaded. Make sure that the path to the MBROLA binary contains no blanks, since FreeTTS can't deal with it.

Rename the MBROLA binary to mbrola.

Now go back to your Romeo and Juliet sketch and add the following highlighted line to your setup() method:

void setup() {
  System.setProperty("mbrola.base", "/path/to/mbrola/");

  size( 800, 400 );
  textFont( createFont( "Georgia", 24 ));

  String[] rawLines = loadStrings( "romeo_and_juliet.txt" );
  ArrayList lines = new ArrayList();
  for ( int i=0; i<rawLines.length; i++) {
    if (!"".equals(rawLines[i])) {
      String[] tmp = rawLines[i].split("#");
      lines.add( new Line( tmp[0], tmp[1].trim() ));
    }
  }

  drama = new Drama( lines );
  TTS romeo = new TTS();
  TTS juliet = new TTS();
  juliet.setPitchShift( 2.4 );
  drama = new Drama( lines, romeo, juliet );
}

Make the path of the system property point to the folder where your mbrola binary and the us1 voice are located.

Now you can change the Juliet TTS object to the following:

    TTS juliet = new TTS( "mbrola_us1" );

You will also need to change the pitch of the voice as mbrola_us1 is already a female voice and we don't need to simulate it anymore.

Disassemble your speakers and try to get rid of the body. We only need the speakers, but make sure you don't remove or destroy the cables.

Cut a hole into your cardboard boxes where the speakers should go. Make the holes a bit smaller than the speakers, as shown in the following picture, because we need to glue them to the box later:

Paint your cardboard boxes. I made one white and the other green.

While the boxes are drying, paint some hair on the heads like in the following picture. I made Romeo-Robot's hair black and Juliet-Robot's hair brown.

After the paint has dried, fix the speakers to the cardboard box using some hot glue, as shown in the following picture. Make sure you don't get any glue on the membrane of your speakers.

Now use some markers to draw a face for the robots. You can see the faces of my robots in the following picture:

Use a needle to attach the heads to the bodies of the robots.

Now connect your robot-actors to your sound card and place Juliet on a balcony (for example an empty shoe-box) and make them act.