Interesting phrase detection

Imagine that a program can take a bunch of text data and automatically find the interesting parts, where "interesting" means that the word or phrase occurs more often than expected. It has a very nice property—no training data is needed, and it works for any language that we have tokens for. You have seen this most often in tag clouds such as the one in the following figure:

The preceding figure shows a tag cloud generated for the lingpipe.com home page. However, be aware that tag clouds are considered to be the "mullets of the Internet" as noted by Jeffery Zeldman in http://www.zeldman.com/daily/0405d.shtml, so you will be on shaky ground if you deploy such a feature on a website.

How to do it...

To get the interesting phrases from a small dataset with tweets about Disney, perform the following steps:

1. Fire up the command line and type:

   java -cp lingpipe-cookbook.1.0.jar:lib/lingpipe-4.1.0.jar:lib/opencsv-2.4.jar com.lingpipe.cookbook.chapter4.InterestingPhrases...