Romeo and Juliet

Exclusive offer: get 50% off this eBook here
Processing 2: Creative Coding Hotshot

Processing 2: Creative Coding Hotshot — Save 50%

Learn Processing with exciting and engaging projects to make your computer talk, see, hear, express emotions, and even design physical objects with this book and ebook

$26.99    $13.50
by Nikolaus Gradwohl | August 2013 | Java Open Source

Robots and the performing arts share a long history. In fact, the word "Robot" was first coined in 1920 for a play by the Czech author Karel Capek named "Rossum's Universal Robots". The play featured six robots, but since nobody was able to build a talking Robot at that time, humans had to play them. Times have changed a lot and we don't need humans to disguise themselves as robots anymore. In this article by Nikolaus Gradwohl, author of Processing 2: Creative Coding Hotshot, we will do it the other way round and make some robots that play humans. Unfortunately, "Rossum's Universal Robots" would require nine humans and six robots, so we've chosen a scene that's simpler to perform. We are going to build a pair of robots who play the humans in the famous balcony scene from Romeo and Juliet.

(For more resources related to this topic, see here.)

Mission Briefing

To create the Processing sketches for this project, we will need to install the Processing library ttslib. This library is a wrapper around the FreeTTS Java library that helps us to write a sketch that reads out text. We will learn how to change the voice parameters of the kevin16 voice of the FreeTTS package to make our robot's voices distinguishable. We will also create a parser that is able to read the Shakespeare script and which generates text-line objects that allow our script to know which line is read by which robot.

A Drama thread will be used to control the text-to-speech objects, and the draw() method of our sketch will print the script on the screen while our robots perform it, just in case one of them forgets a line. Finally, we will use some cardboard boxes and a pair of cheap speakers to create the robots and their stage. The following figure shows how the robots work:

Why Is It Awesome?

Since the 18th century, inventors have tried to build talking machines (with varying success). Talking toys swamped the market in the 1980s and 90s. In every decent Sci-Fi novel, computers and robots are capable of speaking. So how could building talking robots not be awesome? And what could be more appropriate to put these speaking capabilities to test than performing a Shakespeare play? So as you see, building actor robots is officially awesome, just in case your non-geek family members should ask.

Your Hotshot Objectives

We will split this project into four tasks that will guide you through the general on of the robots from beginning to end. Here is a short overview of what we are going to do:

  • Making Processing talk
  • Reading Shakespeare
  • Adding more actors
  • Building robots

Making Processing talk

Since Processing has no speaking capabilities out of the box, our first task is adding an external library using the new Processing Library Manager. We will use the ttslib package, which is a wrapper library around the FreeTTS library.

We will also create a short, speaking Processing sketch to check the installation.

Engage Thrusters

  1. Processing can be extended by contributed libraries. Most of these additional libraries can be installed by navigating to Sketch | Import Library… | Add Library..., as shown in the following screenshot:

  2. In the Library Manager dialog, enter ttslib in the search field to filter the list of libraries.
  3. Click on the ttslib entry and then on the Install button, as shown in the following screenshot, to download and install the library:

  4. To use the new library, we need to import it to our sketch. We do this by clicking on the Sketch menu and choosing Import Library... and then ttslib.
  5. We will now add the setup() and draw() methods to our sketch. We will leave the draw() method empty for now and instantiate a TTS object in the setup() method. Your sketch should look like the following code snippet:

    import guru.ttslib.*;
    TTS tts;
    void setup() {
    tts = new TTS();
    }
    void draw() {
    }

  6. Now we will add a mousePressed() method to our sketch, which will get called if someone clicks on our sketch window. In this method, we are calling the speak() method of the TTS object we created in the setup() method.

    void mousePressed() {
    tts.speak("Hello, I am a Computer");
    }

  7. Click on the Run button to start the Processing sketch. A little gray window should appear.
  8. Turn on your speakers or put on your headphones, and click on the gray window. If nothing went wrong, a friendly male computer voice named kevin16 should greet you now.

Objective Complete - Mini Debriefing

In steps 1 to 3, we installed an additional library to Processing. The ttslib is a wrapper library around the FreeTTS text-to-speech engine.

Then we created a simple Processing sketch that imports the installed library and creates an instance of the TTS class. The TTS objects match the speakers we need in our sketches. In this case, we created only one speaker and added a mousePressed() method that calls the speak() method of our tts object.

Reading Shakespeare

In this part of the project, we are going to create a Drama thread and teach Processing how to read a Shakespeare script. This thread runs in the background and is controlling the performance. We focus on reading and executing the play in this task, and add the speakers in the next one.

Prepare for Lift Off

Our sketch needs to know which line of the script is read by which robot. So we need to convert the Shakespeare script into a more machine-readable format. For every line of text, we need to know which speaker should read the line. So we take the script and add the letter J and a separation character that is used nowhere else in the script, in front of every line our Juliet-Robot should speak, and we add R and the separation letter for every line our Romeo-Robot should speak. After all these steps, our text file looks something like the following:

R# Lady, by yonder blessed moon I vow,
R# That tips with silver all these fruit-tree tops --
J# O, swear not by the moon, the inconstant moon,
J# That monthly changes in her circled orb,
J# Lest that thy love prove likewise variable.
R# What shall I swear by?
J# Do not swear at all.
J# Or if thou wilt, swear by thy gracious self,
J# Which is the god of my idolatry,
J# And I'll believe thee.

Engage Thrusters

Let's write our parser:

  1. Let's start a new sketch by navigating to File | New.
  2. Add a setup() and a draw() method.
  3. Now add the prepared script to the Processing sketch by navigating to Sketch | Add File and selecting the file you just downloaded.
  4. Add the following line to your setup() method:

    void setup() {
    String[] rawLines = loadStrings
    ( "romeo_and_juliet.txt" );
    }

  5. If you renamed your text file, change the filename accordingly.
  6. Create a new tab by clicking on the little arrow icon on the right and choosing New Tab.

  7. Name the class Line. This class will hold our text lines and the speaker.
  8. Add the following code to the tab we just created:

    public class Line {
    String speaker;
    String text;
    public Line( String speaker, String text ) {
    this.speaker = speaker;
    this.text = text;
    }
    }

  9. Switch back to our main tab and add the following highlighted lines of code to the setup() method:

    void setup() {
    String[] rawLines = loadStrings
    ( "romeo_and_juliet.txt" );
    ArrayList lines = new ArrayList();
    for ( int i=0; i<rawLines.length; i++) {
    if (!"".equals(rawLines[i])) {
    String[] tmp = rawLines[i].split("#");
    lines.add( new Line( tmp[0], tmp[1].trim() ));
    }
    }

    }

  10. We have read our text lines and parsed them into the lines array list, but we still need a class that does something with our text lines. So create another tab by clicking on the arrow icon and choosing New Tab from the menu; name it Drama.
  11. Our Drama class will be a thread that runs in the background and tells each of the speaker objects to read one line of text. Add the following lines of code to your Drama class:

    public class Drama extends Thread {
    int current;
    ArrayList lines;
    boolean running;
    public Drama( ArrayList lines ) {
    this.lines = lines;
    current = 0;
    running = false;
    }
    public int getCurrent() {
    return current;
    }
    public Line getLine( int num ) {
    if ( num >=0 && num < lines.size()) {
    return (Line)lines.get( num );
    } else {
    return null;
    }
    }
    public boolean isRunning() {
    return running;
    }
    }

  12. Now we add a run() method that gets executed in the background if we start our thread. Since we have no speaker objects yet, we will print the lines on the console and include a little pause after each line.

    public void run() {
    running = true;
    for ( int i =0; i < lines.size(); i++) {
    current = i;
    Line l = (Line)lines.get(i);
    System.out.println( l.text );
    delay( 1 );
    }
    running = false;
    }

  13. Switch back to the main sketch tab and add the highlighted code to the setup() method to create a drama thread object, and then feed it the parsed text-lines.

    Drama drama;
    void setup() {
    String[] rawLines = loadStrings
    ( "romeo_and_juliet.txt" );
    ArrayList lines = new ArrayList();
    for ( int i=0; i<rawLines.length; i++) {
    if (!"".equals(rawLines[i])) {
    String[] tmp = rawLines[i].split("#");
    lines.add( new Line( tmp[0], tmp[1].trim() ));
    }
    }
    drama = new Drama( lines );
    }

  14. So far our sketch parses the text lines and creates a Drama thread object. What we need next is a method to start it. So add a mousePressed() method to start the drama thread.

    void mousePressed() {
    if ( !drama.isRunning()) {
    drama.start();
    }
    }

  15. Now add a little bit of text to the draw() method to tell the user what to do. Add the following code to the draw() method:

    void draw() {
    background(255);
    textAlign(CENTER);
    fill(0);
    text( "Click here for Drama", width/2, height/2 );
    }

  16. Currently, our sketch window is way too small to contain the text, and we also want to use a bigger font. To change the window size, we simply add the following line to the setup() method:

    void setup() {
    size( 800, 400 );

    String[] rawLines = loadStrings
    ( "romeo_and_juliet.txt" );
    ArrayList lines = new ArrayList();
    for ( int i=0; i<rawLines.length; i++) {
    if (!"".equals(rawLines[i])) {
    String[] tmp = rawLines[i].split("#");
    lines.add( new Line( tmp[0], tmp[1].trim() ));
    }
    }
    drama = new Drama( lines );
    }

  17. To change the used font, we need to tell Processing which font to use. The easiest way to find out the names of the fonts that are currently installed on the computer is to create a new sketch, type the following line, and run the sketch:

    println(PFont.list());

  18. Copy one of the font names you like and add the following line to the Romeo and Juliet sketch:

    void setup() {
    size( 800, 400 );
    textFont( createFont( "Georgia", 24 ));
    ...

  19. Replace the font name in the code lines with one of the fonts on your computer.

Objective Complete - Mini Debriefing

In this section, we wrote the code that parses a text file and generates a list of Line objects. These objects are then used by a Drama thread that runs in the background as soon as anyone clicks on the sketch window. Currently, the Drama thread prints out the text line on the console.

In steps 6 to 8, we created the Line class. This class is a very simple, so-called Plain Old Java Object (POJO) that holds our text lines, but it doesn't add any functionality.

The code that is controlling the performance of our play was created in steps 10 to 12. We created a thread that is able to run in the background, since in the next step we want to be able to use the draw() method and some TTS objects simultaneously.

The code block in step 12 defines a Boolean variable named running, which we used in the mousePressed() method to check if the sketch is already running or should be started.

Classified Intel

In step 17, we used the list() method of the PFont class to get a list of installed fonts. This is a very common pattern in Processing. You would use the same approach to get a list of installed midi-interfaces, web-cams, serial-ports, and so on.

Processing 2: Creative Coding Hotshot Learn Processing with exciting and engaging projects to make your computer talk, see, hear, express emotions, and even design physical objects with this book and ebook
Published: May 2013
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:

Adding more actors

In this task, we will combine the things we did in the previous two tasks and add some TTS objects to our Drama thread. We will need two robot actors for this scene speaking with different voices, and since we want to build robots containing a speaker each, we need one of our TTS objects to speak on the let speaker and the other one on the right.

Unfortunately, FreeTTS only comes with one male voice, so we will have to increase the pitch of the voice for our Juliet-Robot.

Engage Thrusters

  1. First, we open the sketch from the previous task and start by creating two TTS objects, one for each of our robot actors. Both use the default voice named kevin16, but we change the pitch for our Juliet-Robot.

    void setup() {
    size( 800, 400 );
    textFont( createFont( "Georgia", 24 ));
    String[] rawLines = loadStrings
    ( "romeo_and_juliet.txt" );
    ArrayList lines = new ArrayList();
    for ( int i=0; i<rawLines.length; i++) {
    if (!"".equals(rawLines[i])) {
    String[] tmp = rawLines[i].split("#");
    lines.add( new Line( tmp[0], tmp[1].trim() ));
    }
    }
    TTS romeo = new TTS();
    TTS juliet = new TTS();
    juliet.setPitchShift( 2.4 );
    drama = new Drama( lines, romeo, juliet );

    }

  2. Switch to the Drama thread and add two variables to our actors.

    public class Drama extends Thread {
    TTS romeo;
    TTS juliet;

    int current;
    ArrayList lines;
    boolean running;

  3. We also need to extend the constructor of our Drama class to enable us to add the actors.

    public Drama( ArrayList lines, TTS romeo, TTS juliet ) {
    this.lines = lines;
    this.romeo = romeo;
    this.juliet = juliet;

    current = 0;
    running = false;
    }

  4. In the run() method, we take the current line and choose the actor object depending on the actor variable we added to each line. We also don't need the delay() and the println() methods anymore.

    public void run() {
    running = true;
    for ( int i =0; i < lines.size(); i++) {
    current = i;
    Line l = (Line)lines.get(i);

    if ( "J".equals( l.speaker )) {
    juliet.speak( l.text );
    }
    if ( "R".equals( l.speaker )) {
    romeo.speak( l.text );

    }
    }
    running = false;
    }

  5. Since each of our robots gets his own speaker, we don't want to hear the text on both speakers. We want one robot to use the right one and the other robot, the let one. Fortunately, ttslib provides a speakLeft() and a speakRight() method, which do exactly what we need. So change the two speak() methods to look like the following:

    ...
    if ( "J".equals( l.speaker )) {
    juliet.speakLeft( l.text );
    }
    if ( "R".equals( l.speaker )) {
    romeo.speakRight( l.text );
    }
    ...

  6. Currently, our draw() method is somewhat boring and also somewhat misleading if the drama thread is already running. So we will change it to display five lines of the currently read script. Add an if statement to the draw() method that checks the state of the running variable we added to our Drama thread earlier.

    void draw() {
    background(255);
    textAlign(CENTER);
    fill(0);
    if ( !drama.isRunning() ) {
    text( "Click here for Drama", width/2, height/2 );
    } else {
    }

    }

  7. Now we add a for loop that displays the previous two lines, the line that's currently being read, and the next two lines of text in our sketch window. We will now change the text alignment so that it matches the speaker of our robot actors. The text will be aligned to the right if our robot uses speakRight(), and it will be aligned to the let if our robot uses speakLeft().

    void draw() {
    background(255);
    textAlign(CENTER);
    fill(0);
    if ( !drama.isRunning() ) {
    text( "Click here for Drama", width/2, height/2 );
    }
    else {
    int current = drama.getCurrent();
    for ( int i = -2; i < 3; i ++) {
    Line l = drama.getLine(i + current);
    if ( l != null) {
    if ( "J".equals( l.speaker )) {
    textAlign( LEFT );
    text( l.text, 10, height/2 + i * 30 );
    }
    else {
    textAlign( RIGHT );
    text( l.text, width - 10, height/2 + i * 30 );
    }
    }
    }

    }
    }

  8. To show the current line more prominently, we will change the color of the text. We set the color of the current line to black and all other lines to a lighter gray by adding a fill() statement to the for loop.

    for ( int i = -2; i < 3; i ++) {
    fill( abs(i) * 100 );
    Line l = drama.getLine(i + current);
    if ( l != null) {
    if ( "J".equals( l.speaker )) {
    textAlign( LEFT );
    text( l.text, 10, height/2 + i * 30 );
    }
    else {
    textAlign( RIGHT );
    text( l.text, width - 10, height/2 + i * 30 );
    }
    }
    }

  9. Now run your code and click on the sketch window to start the drama thread.

Objective Complete - Mini Debriefing

In this task of our current mission, we added two TTS objects and changed the voice parameters to make them sound different in step 1. Then we extended our Drama thread and added TTS objects for the voices of our robot actors. In steps 4 and 5, we modified the run method to use the voices we just created instead of just printing the text lines.

In steps 6 to 9, we made changes to the draw() method and made it display five lines of text. The line that's currently spoken is black, and the two lines before and after it fade to a light gray.

The fill() method is used to change not only the fill color of an object, but also the text color. Because the index of our for loop runs from -2 to 2, we can simply take the absolute value and multiply it with 100 to get the gray level. The following is a screenshot of the running sketch:

Classified Intel

FreeTTS and ttslib also allow you to use a binary TTS engine named MBROLA. Unfortunately, it's only distributed in binary form, and at the time of writing, it only works on Linux. So if you are using Linux and want to give it a try, you can make the following changes to our Romeo and Juliet sketch:

  1. Open http://tcts.fpms.ac.be/synthesis/mbrola.html in your browser and click on Download. Download the MBROLA binary for your platform.
  2. Download the female_us1 voice from the MBROLA site.
  3. Create a folder for the MBROLA binary and unzip the two packages you just downloaded. Make sure that the path to the MBROLA binary contains no blanks, since FreeTTS can't deal with it.
  4. Rename the MBROLA binary to mbrola.
  5. Now go back to your Romeo and Juliet sketch and add the following highlighted line to your setup() method:

    void setup() {
    System.setProperty("mbrola.base", "/path/to/mbrola/");
    size( 800, 400 );
    textFont( createFont( "Georgia", 24 ));
    String[] rawLines = loadStrings
    ( "romeo_and_juliet.txt" );
    ArrayList lines = new ArrayList();
    for ( int i=0; i<rawLines.length; i++) {
    if (!"".equals(rawLines[i])) {
    String[] tmp = rawLines[i].split("#");
    lines.add( new Line( tmp[0], tmp[1].trim() ));
    }
    }
    drama = new Drama( lines );
    TTS romeo = new TTS();
    TTS juliet = new TTS();
    juliet.setPitchShift( 2.4 );
    drama = new Drama( lines, romeo, juliet );
    }

  6. Make the path of the system property point to the folder where your mbrola binary and the us1 voice are located.
  7. Now you can change the Juliet TTS object to the following:

    TTS juliet = new TTS( "mbrola_us1" );

  8. You will also need to change the pitch of the voice as mbrola_us1 is already a female voice and we don't need to simulate it anymore.

MBROLA is a text-to-speech engine developed at the Faculte Polytechnique de Mons in Belgium. The author requires every publication that mentions their work to mention their book, An Introduction To Text-To-Speech Synthesis, Thierry Dutoit, Kluwer Academic Publishers, Dordrecht, Hardbound, ISBN 1-4020-0369-2, April 1997, 312 pp.

Building robots

Now we are ready for the final task of our Romeo and Juliet project. We will take some cardboard boxes, Styrofoam spheres, and a pair of cheap speakers or headphones and turn them into our robot-actors.

The robots described in this section are just my version. If you like, you can build them to be completely different and as complex or as simple as you want.

Prepare for Lift Off

To build our robots, we need some materials and tools, which should not be too hard to find. I used the following for my robot-actors:

  • Two cardboard boxes for the bodies
  • Two Styrofoam spheres for the heads
  • A pair of cheap speakers or headphones
  • Googly eyes
  • Acrylic paint
  • Marker pens
  • Needles
  • A hot glue gun

The following picture shows the main materials I used for my robots:

Engage Thrusters

Let's build some robots:

  1. Disassemble your speakers and try to get rid of the body. We only need the speakers, but make sure you don't remove or destroy the cables.
  2. Cut a hole into your cardboard boxes where the speakers should go. Make the holes a bit smaller than the speakers, as shown in the following picture, because we need to glue them to the box later:

  3. Paint your cardboard boxes. I made one white and the other green.
  4. While the boxes are drying, paint some hair on the heads like in the following picture. I made Romeo-Robot's hair black and Juliet-Robot's hair brown.

  5. After the paint has dried, fix the speakers to the cardboard box using some hot glue, as shown in the following picture. Make sure you don't get any glue on the membrane of your speakers.

  6. Now use some markers to draw a face for the robots. You can see the faces of my robots in the following picture:

  7. Use a needle to attach the heads to the bodies of the robots.
  8. Now connect your robot-actors to your sound card and place Juliet on a balcony (for example an empty shoe-box) and make them act.

Objective Complete - Mini Debriefing

In this task, we completed our robot-actors by building bodies for them. We used some cardboard boxes, painted them, and added a cheap pair of speakers by gluing them into the boxes. Each robot got a head made of a painted Styrofoam sphere.

As I already said in the introduction to this task, there is no right or wrong way to build your robots. Build them as small or as big as you like. Add some hair, make them a nose, sew a dress for Juliet, draw Romeo a mustache, and so on.

Mission Accomplished

In this mission, we added speaking capabilities to Processing by installing the ttslib library. We learned how to simulate multiple speakers by changing the pitch of voice or installing an additional TTS engine on Linux.

We also defined a speak text format for the Shakespeare script to make parsing easier and created a Drama thread that manages our text lines and controls our robots.

We completed our robot-actors, and they are eagerly waiting to perform in front of an audience. So gather your friends and family and make your robots a stage. They will love it, trust me. Well, at least your robots will love it, as you can see in the following picture:

You Ready to go Gung HO? A Hotshot Challenge

Now that you have completed building your robot-actors, why don't you try to take them to the next level?

  • Make them perform different plays. There are plenty of famous dialogs out there that could be performed by your robot-actors.
  • Currently your robot-actors are stationary. Try to add some servo motors and make them move their heads using an Arduino.
  • Connect some webcams to your computer and use Processing to record some videos of the performances from different angles.
  • Make your robot-actors perform a play with a real actor.
  • Make your robots read live input from the Web such as Twitter feeds or the news.
  • Your robots don't have to look like humans; make them look like zombies, aliens, animals, and so on.

Summary

This article helped you learn how to make some cardboard robots that perform the famous balcony scene from Shakespeare's Romeo and Juliet. It also helped you create some talking robots using a pair of cheap speakers and some cardboard boxes, and learn how to install and use a text-to-speech library in Processing.

Resources for Article:


Further resources on this subject:


Processing 2: Creative Coding Hotshot Learn Processing with exciting and engaging projects to make your computer talk, see, hear, express emotions, and even design physical objects with this book and ebook
Published: May 2013
eBook Price: $26.99
Book Price: $44.99
See more
Select your format and quantity:

About the Author :


Nikolaus Gradwohl

Nikolaus Gradwohl was born in 1976 in Vienna, Austria, and always wanted to become an inventor like Gyro Gearloose. When he got his first Atari, he decide that becoming a computer programmer was the closest he could get to that dream. He has since made a living writing programs for nearly anything that can be programmed, ranging from an 8-bit microcontroller to mainframes. In his free time, he likes gaining knowledge on programming languages and operating systems.

Nikolaus has been using Processing since 2008, and has written countless sketches and some Processing libraries.

You can see some of his work on his blog at http://www.local-guru.net/.

Books From Packt


 Kinect for Windows SDK Programming Guide
Kinect for Windows SDK Programming Guide

 Kinect in Motion – Audio and Visual Tracking by Example
Kinect in Motion – Audio and Visual Tracking by Example

Raspberry Pi Home Automation with Arduino
Raspberry Pi Home Automation with Arduino

C Programming for Arduino
C Programming for Arduino

Processing 2: Creative Programming Cookbook
Processing 2: Creative Programming Cookbook

 jMonkeyEngine 3.0 Beginner's Guide
jMonkeyEngine 3.0 Beginner's Guide

Raspberry Pi Media Center
Raspberry Pi Media Center

Android 3.0 Application Development Cookbook
Android 3.0 Application Development Cookbook


Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software