Now that we have a basic understanding of web crawlers, we are ready to create our own. In this simple web crawler, we will keep track of the pages visited using ArrayList instances. In addition, jsoup will be used to parse a web page and we will limit the number of pages we visit. Jsoup (https://jsoup.org/) is an open source HTML parser. This example demonstrates the basic structure of a web crawler and also highlights some of the issues involved in creating a web crawler.

We will use the SimpleWebCrawler class, as declared here:

public class SimpleWebCrawler { 
 
    private String topic; 
    private String startingURL; 
    private String urlLimiter; 
    private final int pageLimit = 20; 
    private ArrayList<String> visitedList = new ArrayList<>(); 
    private ArrayList<String> pageList = new ArrayList<>(); 
    ... 
    public static void main(String[] args) { 
        new SimpleWebCrawler(); 
    } 
 
} 

The instance variables are detailed here:

In the SimpleWebCrawler constructor, we initialize the instance variables to begin the search from the Wikipedia page for Bishop Rock, an island off the coast of Italy. This was chosen to minimize the number of pages that might be retrieved. As we will see, there are many more Wikipedia pages dealing with Bishop Rock than one might think.

The urlLimiter variable is set to Bishop_Rock, which will restrict the embedded links to follow to just those containing that string. Each page of interest must contain the value stored in the topic variable. The visitPage method performs the actual crawl:

    public SimpleWebCrawler() { 
        startingURL = https://en.wikipedia.org/wiki/Bishop_Rock, " 
            + "Isles_of_Scilly"; 
        urlLimiter = "Bishop_Rock"; 
        topic = "shipping route"; 
        visitPage(startingURL); 
    } 

In the visitPage method, the pageList ArrayList is checked to see whether the maximum number of accepted pages has been exceeded. If the limit has been exceeded, then the search terminates:

    public void visitPage(String url) { 
        if (pageList.size() >= pageLimit) { 
            return; 
        } 
       ... 
    } 

If the page has already been visited, then we ignore it. Otherwise, it is added to the visited list:

    if (visitedList.contains(url)) { 
        // URL already visited 
    } else { 
        visitedList.add(url); 
            ... 
    } 

Jsoup is used to parse the page and return a Document object. There are many different exceptions and problems that can occur such as a malformed URL, retrieval timeouts, or simply bad links. The catch block needs to handle these types of problems. We will provide a more in-depth explanation of jsoup in web scraping in Java:

    try { 
        Document doc = Jsoup.connect(url).get(); 
            ... 
        } 
    } catch (Exception ex) { 
        // Handle exceptions 
    } 

If the document contains the topic text, then the link is displayed and added to the pageList ArrayList. Each embedded link is obtained, and if the link contains the limiting text, then the visitPage method is called recursively:

    if (doc.text().contains(topic)) { 
        out.println((pageList.size() + 1) + ": [" + url + "]"); 
        pageList.add(url); 
 
        // Process page links 
        Elements questions = doc.select("a[href]"); 
        for (Element link : questions) { 
            if (link.attr("href").contains(urlLimiter)) { 
                visitPage(link.attr("abs:href")); 
            } 
        } 
    } 

This approach only examines links in those pages that contain the topic text. Moving the for loop outside of the if statement will test the links for all pages.

The output follows:

1: [https://en.wikipedia.org/wiki/Bishop_Rock,_Isles_of_Scilly]
2: [https://en.wikipedia.org/wiki/Bishop_Rock_Lighthouse]
3: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&oldid=717634231#Lighthouse]
4: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=717634231]
5: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&oldid=716622943]
6: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=716622943]
7: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&oldid=716608512]
8: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=716608512]
...
20: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=716603919]

In this example, we did not save the results of the crawl in an external source. Normally this is necessary and can be stored in a file or database.

Here we will illustrate the use of the crawler4j (https://github.com/yasserg/crawler4j) web crawler. We will use an adapted version of the basic crawler found at https://github.com/yasserg/crawler4j/tree/master/src/test/java/edu/uci/ics/crawler4j/examples/basic. We will create two classes: CrawlerController and SampleCrawler. The former class set ups the crawler while the latter contains the logic that controls what pages will be processed.

As with our previous crawler, we will crawl the Wikipedia article dealing with Bishop Rock. The results using this crawler will be smaller as many extraneous pages are ignored.

Let's look at the CrawlerController class first. There are several parameters that are used with the crawler as detailed here:

The basic class is shown here:

public class CrawlerController { 
 
  public static void main(String[] args) throws Exception { 
    int numberOfCrawlers = 2; 
    CrawlConfig config = new CrawlConfig(); 
    String crawlStorageFolder = "data"; 
     
    config.setCrawlStorageFolder(crawlStorageFolder); 
    config.setPolitenessDelay(500); 
    config.setMaxDepthOfCrawling(2); 
    config.setMaxPagesToFetch(20); 
    config.setIncludeBinaryContentInCrawling(false); 
    ... 
  } 
}

Next, the CrawlController class is created and configured. Notice the RobotstxtConfig and RobotstxtServer classes used to handle robot.txt files. These files contain instructions that are intended to be read by a web crawler. They provide direction to help a crawler to do a better job such as specifying which parts of a site should not be crawled. This is useful for auto generated pages:

    PageFetcher pageFetcher = new PageFetcher(config); 
    RobotstxtConfig robotstxtConfig = new RobotstxtConfig(); 
    RobotstxtServer robotstxtServer =  
        new RobotstxtServer(robotstxtConfig, pageFetcher); 
    CrawlController controller =  
        new CrawlController(config, pageFetcher, robotstxtServer); 

The crawler needs to start at one or more pages. The addSeed method adds the starting pages. While we used the method only once here, it can be used as many times as needed:

    controller.addSeed( 
      "https://en.wikipedia.org/wiki/Bishop_Rock,_Isles_of_Scilly"); 

The start method will begin the crawling process:

    controller.start(SampleCrawler.class, numberOfCrawlers); 

The SampleCrawler class contains two methods of interest. The first is the shouldVisit method that determines whether a page will be visited and the visit method that actually handles the page. We start with the class declaration and the declaration of a Java regular expression class Pattern object. It will be one way of determining whether a page will be visited. In this declaration, standard images are specified and will be ignored:

    public class SampleCrawler extends WebCrawler { 
        private static final Pattern IMAGE_EXTENSIONS =  
            Pattern.compile(".*\\.(bmp|gif|jpg|png)$"); 
 
        ... 
    } 

The shouldVisit method is passed a reference to the page where this URL was found along with the URL. If any of the images match, the method returns false and the page is ignored. In addition, the URL must start with https://en.wikipedia.org/wiki/. We added this to restrict our searches to the Wikipedia website:

    public boolean shouldVisit(Page referringPage, WebURL url) { 
        String href = url.getURL().toLowerCase(); 
        if (IMAGE_EXTENSIONS.matcher(href).matches()) { 
            return false; 
        } 
        return href.startsWith("https://en.wikipedia.org/wiki/"); 
    }

The visit method is passed a Page object representing the page being visited. In this implementation, only those pages containing the string shipping route will be processed. This further restricts the pages visited. When we find such a page, its URL, Text, and Text length are displayed:

    public void visit(Page page) { 
        String url = page.getWebURL().getURL(); 
 
        if (page.getParseData() instanceof HtmlParseData) { 
            HtmlParseData htmlParseData =  
                (HtmlParseData) page.getParseData(); 
            String text = htmlParseData.getText(); 
            if (text.contains("shipping route")) { 
                out.println("\nURL: " + url); 
                out.println("Text: " + text); 
                out.println("Text length: " + text.length()); 
            } 
        } 
    } 

The following is the truncated output of the program when executed:

URL: https://en.wikipedia.org/wiki/Bishop_Rock,_Isles_of_Scilly
Text: Bishop Rock, Isles of Scilly...From Wikipedia, the free encyclopedia ... Jump to: ... navigation, search For the Bishop Rock in the Pacific Ocean, see Cortes Bank. Bishop Rock Bishop Rock Lighthouse (2005)
...
Text length: 14677

Notice that only one page was returned. This web crawler was able to identify and ignore previous versions of the main web page.

We could perform further processing, but this example provides some insight into how the API works. Significant amounts of information can be obtained when visiting a page. In the example, we only used the URL and the length of the text. The following is a sample of other data that you may be interested in obtaining:

OAuth is an open standard used to authenticate users to many different websites. A resource owner effectively delegates access to a server resource without having to share their credentials. It works over HTTPS. OAuth 2.0 succeeded OAuth and is not backwards compatible. It provides client developers a simple way of providing authentication. Several companies use OAuth 2.0 including PayPal, Comcast, and Blizzard Entertainment.

A list of OAuth 2.0 providers is found at https://en.wikipedia.org/wiki/List_of_OAuth_providers. We will use several of these in our discussions.

The sheer volume of data and the popularity of the site, among celebrities and the general public alike, make Twitter a valuable resource for mining social media data. Twitter is a popular social media platform allowing users to read and post short messages called tweets. Twitter provides API support for posting and pulling tweets, including streaming data from all public users. While there are services available for pulling the entire set of public tweet data, we are going to examine other options that, while limiting in the amount of data retrieved at one time, are available at no cost.

We are going to focus on the Twitter API for retrieving streaming data. There are other options for retrieving tweets from a specific user as well as posting data to a specific account but we will not be addressing those in this chapter. The public stream API, at the default access level, allows the user to pull a sample of public tweets currently streaming on Twitter. It is possible to refine the data by specifying parameters to track keywords, specific users, and location.

We are going to use HBC, a Java HTTP client, for this example. You can download a sample HBC application at https://github.com/twitter/hbc. If you prefer to use a different HTTP client, ensure it will return incremental response data. The Apache HTTP client is one option. Before you can create the HTTP connection, you must first create a Twitter account and an application within that account. To get started with the app, visit apps.twitter.com. Once your app is created, you will be assigned a consumer key, consumer secret, access token, and access secret token. We will also use OAuth, as discussed previously in this chapter.

First, we will write a method to perform the authentication and request data from Twitter. The parameters for our method are the authentication information given to us by Twitter when we created our app. We will create a BlockingQueue object to hold our streaming data. For this example, we will set a default capacity of 10,000. We will also specify our endpoint and turn off stall warnings:

    public static void streamTwitter( 
        String consumerKey, String consumerSecret,  
        String accessToken, String accessSecret)  
            throws InterruptedException { 
 
        BlockingQueue<String> statusQueue =  
            new LinkedBlockingQueue<String>(10000); 
        StatusesSampleEndpoint ending =  
            new StatusesSampleEndpoint(); 
        ending.stallWarnings(false); 
        ... 
    } 

Next, we create an Authentication object using OAuth1, a variation of the OAuth class. We can then build our connection client and complete the HTTP connection:

    Authentication twitterAuth = new OAuth1(consumerKey,  
        consumerSecret, accessToken, accessSecret); 
    BasicClient twitterClient = new ClientBuilder() 
            .name("Twitter client") 
            .hosts(Constants.STREAM_HOST) 
            .endpoint(ending) 
            .authentication(twitterAuth) 
            .processor(new StringDelimitedProcessor(statusQueue)) 
            .build(); 
    twitterClient.connect(); 

For the purposes of this example, we will simply read the messages received from the stream and print them to the screen. The messages are returned in JSON format and the decision of how to process them in a real application will depend upon the purpose and limitations of that application:

    for (int msgRead = 0; msgRead < 1000; msgRead++) { 
      if (twitterClient.isDone()) { 
        out.println(twitterClient.getExitEvent().getMessage()); 
        break; 
      } 
 
      String msg = statusQueue.poll(10, TimeUnit.SECONDS); 
      if (msg == null) { 
        out.println("Waited 10 seconds - no message received"); 
      } else { 
        out.println(msg); 
      } 
    } 
    twitterClient.stop(); 

To execute our method, we simply pass our authentication information to the streamTwitter method. For security purposes, we have replaced our personal keys here. Authentication information should always be protected:

    public static void main(String[] args) { 
   
      try { 
        SampleStreamExample.streamTwitter( 
            myKey, mySecret, myToken, myAccess);  
      } catch (InterruptedException e) { 
        out.println(e); 
      } 
    } 

Here is truncated sample data retrieved using the methods listed above. Your data will vary based upon Twitter's live stream, but it should resemble this example:

{"created_at":"Fri May 20 15:47:21 +0000 2016","id":733685552789098496,"id_str":"733685552789098496","text":"bwisit si em bahala sya","source":"\u003ca href="http:\/\/twitter.com" rel="nofollow"\u003eTwitter Web 
...
ntions":[],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"tl","timestamp_ms":"1463759241660"}

Twitter also provides support for pulling all data for one specific user account, as well as posting data directly to an account. A REST API is also available and provides support for specific queries via the search API. These also use the OAuth standard and return data in JSON files.

Wikipedia (https://www.wikipedia.org/) is a useful source of text and image type information. It is an Internet encyclopedia that hosts 38 million articles written in over 250 languages (https://en.wikipedia.org/wiki/Wikipedia). As such, it is useful to know how to programmatically access its contents.

MediaWiki is an open source wiki application that supports wiki type sites. It is used to support Wikipedia and many other sites. The MediaWiki API (http://www.mediawiki.org/wiki/API) provides access to a wiki's data and metadata over HTTP. An application, using this API, can log in, read data, and post changes to a site.

There are several Java APIs that support programmatic access to a wiki site as listed at https://www.mediawiki.org/wiki/API:Client_code#Java. To demonstrate Java access to a wiki we will use Bliki found at https://bitbucket.org/axelclk/info.bliki.wiki/wiki/Home. It provides good access and is easy to use for most basic operations.

The MediaWiki API is complex and has many features. The intent of this section is to illustrate the basic process of obtaining text from a Wikipedia article using this API. It is not possible to cover the API completely here.

We will use the following classes from the info.bliki.api and info.bliki.wiki.model packages:

Javadocs for Bliki are found at http://www.javadoc.io/doc/info.bliki.wiki/bliki-core/3.1.0.

The following example has been adapted from http://www.integratingstuff.com/2012/04/06/hook-into-wikipedia-using-java-and-the-mediawiki-api/. This example will access the English Wikipedia page for the subject, data science. We start by creating an instance of the User class. The first two arguments of the three-argument constructor are the user ID and password, respectively. In this case, they are empty strings. This combination allows us to read a page without having to set up an account. The third argument is the URL for the MediaWiki API page:

    User user = new User("", "",  
        "http://en.wikipedia.org/w/api.php"); 
    user.login(); 

An account will enable us to modify the document. The queryContent method returns a list of Page objects for the subjects found in a string array. Each string should be the title of a page. In this example, we access a single page:

    String[] titles = {"Data science"}; 
    List<Page> pageList = user.queryContent(titles); 

Each Page object contains the content of a page. There are several methods that will return the contents of the page. For each page, a WikiModel instance is created using the two-argument constructor. The first argument is the image base URL and the second argument is the link base URL. These URLs use Wiki variables called image and title, which will be replaced when creating links:

    for (Page page : pageList) { 
        WikiModel wikiModel = new WikiModel("${image}",  
            "${title}"); 
        ... 
    } 

The render method will take the wiki page and render it to HTML. There is also a method to render the page to a PDF document:

    String htmlText = wikiModel.render(page.toString()); 

The HTML text is then displayed:

    out.println(htmlText); 

A partial listing of the output follows:

<p>PageID: 35458904; NS: 0; Title: Data science; 
Image url: 
Content:
{{distinguish}}
{{Use dmy dates}}
{{Data Visualization}}</p>
<p><b>Data science</b> is an interdisciplinary field about processes and systems to extract <a href="Knowledge" >knowledge</a> 
...

We can also obtain basic information about the article using one of several methods as shown here:

    out.println("Title: " + page.getTitle() + "\n" + 
        "Page ID: " + page.getPageid() + "\n" + 
        "Timestamp: " + page.getCurrentRevision().getTimestamp()); 

It is also possible to obtain a list of references in the article and a list of the headers. Here, a list of the references is displayed:

    List <Reference> referenceList = wikiModel.getReferences(); 
    out.println(referenceList.size()); 
    for(Reference reference : referenceList) { 
        out.println(reference.getRefString()); 
    } 

The following illustrates the process of getting the section headers:

    ITableOfContent toc = wikiModel.getTableOfContent(); 
    List<SectionHeader> sections = toc.getSectionHeaders(); 
    for(SectionHeader sh : sections) { 
        out.println(sh.getFirst()); 
    } 

The entire content of Wikipedia can be downloaded. This process is discussed at https://en.wikipedia.org/wiki/Wikipedia:Database_download.

It may be desirable to set up your own Wikipedia server to handle your request.

Flickr (https://www.flickr.com/) is an online photo management and sharing application. It is a possible source for images and videos. The Flickr Developer Guide (https://www.flickr.com/services/developer/) is a good starting point to learn more about Flickr's API.

One of the first steps to using the Flickr API is to request an API key. This key is used to sign your API requests. The process to obtain a key starts at https://www.flickr.com/services/apps/create/. Both commercial and noncommercial keys are available. When you obtain a key you will also get a "secret." Both of these are required to use the API.

We will illustrate the process of locating and downloading images from Flickr. The process involves:

A FlickrException or IOException may be thrown during this process. There are several APIs that support Flickr access. We will be using Flickr4Java, found at https://github.com/callmeal/Flickr4Java. The Flickr4Java Javadocs is found at http://flickrj.sourceforge.net/api/. We will start with a try block and the apikey and secret declarations:

    try { 
        String apikey = "Your API key"; 
        String secret = "Your secret"; 
 
    } catch (FlickrException | IOException ex) { 
        // Handle exceptions 
    } 

The Flickr instance is created next, where the apikey and secret are supplied as the first two parameters. The last parameter specifies the transfer technique used to access Flickr servers. Currently, the REST transport is supported using the REST class:

    Flickr flickr = new Flickr(apikey, secret, new REST()); 

To search for images, we will use the SearchParameters class. This class supports a number of criteria that will narrow down the number of images returned from a query and includes such criteria as latitude, longitude, media type, and user ID. In the following sequence, the setBBox method specifies the longitude and latitude for the search. The parameters are (in order): minimum longitude, minimum latitude, maximum longitude, and maximum latitude. The setMedia method specifies the type of media. There are three possible arguments — "all", "photos", and "videos":

    SearchParameters searchParameters = new SearchParameters(); 
    searchParameters.setBBox("-180", "-90", "180", "90"); 
    searchParameters.setMedia("photos"); 

The PhotosInterface class possesses a search method that uses the SearchParameters instance to retrieve a list of photos. The getPhotosInterface method returns an instance of the PhotosInterface class, as shown next. The SearchParameters instance is the first parameter. The second parameter determines how many photos are retrieved per page and the third parameter is the offset. A PhotoList class instance is returned:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 
    PhotoList<Photo> list = pi.search(searchParameters, 10, 0); 

The next sequence illustrates the use of several methods to get information about the images retrieved. Each Photo instance is accessed using the get method. The title, image format, public flag, and photo URL are displayed:

    out.println("Image List"); 
    for (int i = 0; i < list.size(); i++) { 
        Photo photo = list.get(i); 
        out.println("Image: " + i + 
            `"\nTitle: " + photo.getTitle() +  
            "\nMedia: " + photo.getOriginalFormat() + 
            "\nPublic: " + photo.isPublicFlag() + 
            "\nUrl: " + photo.getUrl() + 
            "\n"); 
    } 
    out.println(); 

A partial listing is shown here where many of the specific values have been modified to protect the original data:

Image List
Image: 0
Title: XYZ Image
Media: jpg
Public: true
Url: https://flickr.com/photos/7723...@N02/269...
Image: 1
Title: IMG_5555.jpg
Media: jpg
Public: true
Url: https://flickr.com/photos/2665...@N07/264...
Image: 2
Title: DSC05555
Media: jpg
Public: true
Url: https://flickr.com/photos/1179...@N04/264...

The list of images returned by this example will vary since we used a fairly wide search range and images are being added all of the time.

There are two approaches that we can use to download an image. The first uses the image's URL and the second uses a Photo object. The image's URL can be obtained from a number of sources. We use the Photo class getUrl method for this example.

In the following sequence, we obtain an instance of PhotosInterface using its constructor to illustrate an alternate approach:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 

We get the first Photo instance from the previous list and then its getUrl to get the image's URL. The PhotosInterface class's getImage method returns a BufferedImage object representing the image as shown here:

    Photo currentPhoto = list.get(0);  
    BufferedImage bufferedImage =  
        pi.getImage(currentPhoto.getUrl()); 

The image is then saved to a file using the ImageIO class:

    File outputfile = new File("image.jpg"); 
    ImageIO.write(bufferedImage, "jpg", outputfile); 

The getImage method is overloaded. Here, the Photo instance and the size of the image desired are used as arguments to get the BufferedImage instance:

    bufferedImage = pi.getImage(currentPhoto, Size.SMALL); 

The image can be saved to a file using the previous technique.

The Flickr4Java API supports a number of other techniques for working with Flickr images.

YouTube is a popular video site where users can upload and share videos (https://www.youtube.com/). It has been used to share humorous videos, provide instructions on how to do any number of things, and share information among its viewers. It is a useful source of information as it captures the thoughts and ideas of a diverse group of people. This provides an interesting opportunity to analysis and gain insight into human behavior.

YouTube can serve as a useful source of videos and video metadata. A Java API is available to access its contents (https://developers.google.com/youtube/v3/). Detailed documentation of the API is found at https://developers.google.com/youtube/v3/docs/.

In this section, we will demonstrate how to search for videos by keyword and retrieve information of interest. We will also show how to download a video. To use the YouTube API, you will need a Google account, which can be obtained at https://www.google.com/accounts/NewAccount. Next, create an account in the Google Developer's Console (https://console.developers.google.com/). API access is supported using either API keys or OAuth 2.0 credentials. The project creation process and keys are discussed at https://developers.google.com/youtube/registering_an_application#create_project.

    try { 
        YouTube youtube = new YouTube.Builder( 
            Auth.HTTP_TRANSPORT, 
            Auth.JSON_FACTORY, 
            new HttpRequestInitializer() { 
                public void initialize(HttpRequest request)  
                        throws IOException { 
                } 
            }) 
                .setApplicationName("application_name") 
        ... 
    } catch (GoogleJSONResponseException ex) { 
        // Handle exceptions 
    } catch (IOException ex) { 
        // Handle exceptions 
    } 

    String queryTerm = "cats"; 

    YouTube.Search.List search = youtube 
        .search() 
        .list("id,snippet"); 

{ 
  "kind": "youtube#searchResult", 
  "etag": etag, 
  "id": { 
    "kind": string, 
    "videoId": string, 
    "channelId": string, 
    "playlistId": string 
  }, 
  "snippet": { 
    "publishedAt": datetime, 
    "channelId": string, 
    "title": string, 
    "description": string, 
    "thumbnails": { 
      (key): { 
        "url": string, 
        "width": unsigned integer, 
        "height": unsigned integer 
      } 
    }, 
    "channelTitle": string, 
    "liveBroadcastContent": string 
  } 
} 

    String apiKey = "Your API key"; 
    search.setKey(apiKey); 
    search.setQ(queryTerm); 
    search.setType("video"); 

    search.setFields("items(id/kind,id/videoId,snippet/title," +  
        "snippet/description,snippet/thumbnails/default/url)"); 

    search.setMaxResults(10L); 

    SearchListResponse searchResponse = search.execute(); 
    List<SearchResult> searchResultList =  
        searchResponse.getItems(); 

    SearchResult video = searchResultList.iterator().next(); 
    Thumbnail thumbnail = video 
        .getSnippet().getThumbnails().getDefault(); 
 
    out.println("Kind: " + video.getKind()); 
    out.println("Video Id: " + video.getId().getVideoId()); 
    out.println("Title: " + video.getSnippet().getTitle()); 
    out.println("Description: " +  
        video.getSnippet().getDescription()); 
    out.println("Thumbnail: " + thumbnail.getUrl()); 

Kind: null
Video Id: tntO...
Title: Funny Cats ...
Description: Check out the ...
Thumbnail: https://i.ytimg.com/vi/tntO.../default.jpg

    String url = "http://www.youtube.com/watch?v=videoID"; 
    String path = "."; 
    VGet vget = new VGet(new URL(url), new File(path)); 
    vget.download(); 

Now that we have a basic understanding of web crawlers, we are ready to create our own. In this simple web crawler, we will keep track of the pages visited using ArrayList instances. In addition, jsoup will be used to parse a web page and we will limit the number of pages we visit. Jsoup (https://jsoup.org/) is an open source HTML parser. This example demonstrates the basic structure of a web crawler and also highlights some of the issues involved in creating a web crawler.

We will use the SimpleWebCrawler class, as declared here:

public class SimpleWebCrawler { 
 
    private String topic; 
    private String startingURL; 
    private String urlLimiter; 
    private final int pageLimit = 20; 
    private ArrayList<String> visitedList = new ArrayList<>(); 
    private ArrayList<String> pageList = new ArrayList<>(); 
    ... 
    public static void main(String[] args) { 
        new SimpleWebCrawler(); 
    } 
 
} 

The instance variables are detailed here:

In the SimpleWebCrawler constructor, we initialize the instance variables to begin the search from the Wikipedia page for Bishop Rock, an island off the coast of Italy. This was chosen to minimize the number of pages that might be retrieved. As we will see, there are many more Wikipedia pages dealing with Bishop Rock than one might think.

The urlLimiter variable is set to Bishop_Rock, which will restrict the embedded links to follow to just those containing that string. Each page of interest must contain the value stored in the topic variable. The visitPage method performs the actual crawl:

    public SimpleWebCrawler() { 
        startingURL = https://en.wikipedia.org/wiki/Bishop_Rock, " 
            + "Isles_of_Scilly"; 
        urlLimiter = "Bishop_Rock"; 
        topic = "shipping route"; 
        visitPage(startingURL); 
    } 

In the visitPage method, the pageList ArrayList is checked to see whether the maximum number of accepted pages has been exceeded. If the limit has been exceeded, then the search terminates:

    public void visitPage(String url) { 
        if (pageList.size() >= pageLimit) { 
            return; 
        } 
       ... 
    } 

If the page has already been visited, then we ignore it. Otherwise, it is added to the visited list:

    if (visitedList.contains(url)) { 
        // URL already visited 
    } else { 
        visitedList.add(url); 
            ... 
    } 

Jsoup is used to parse the page and return a Document object. There are many different exceptions and problems that can occur such as a malformed URL, retrieval timeouts, or simply bad links. The catch block needs to handle these types of problems. We will provide a more in-depth explanation of jsoup in web scraping in Java:

    try { 
        Document doc = Jsoup.connect(url).get(); 
            ... 
        } 
    } catch (Exception ex) { 
        // Handle exceptions 
    } 

If the document contains the topic text, then the link is displayed and added to the pageList ArrayList. Each embedded link is obtained, and if the link contains the limiting text, then the visitPage method is called recursively:

    if (doc.text().contains(topic)) { 
        out.println((pageList.size() + 1) + ": [" + url + "]"); 
        pageList.add(url); 
 
        // Process page links 
        Elements questions = doc.select("a[href]"); 
        for (Element link : questions) { 
            if (link.attr("href").contains(urlLimiter)) { 
                visitPage(link.attr("abs:href")); 
            } 
        } 
    } 

This approach only examines links in those pages that contain the topic text. Moving the for loop outside of the if statement will test the links for all pages.

The output follows:

1: [https://en.wikipedia.org/wiki/Bishop_Rock,_Isles_of_Scilly]
2: [https://en.wikipedia.org/wiki/Bishop_Rock_Lighthouse]
3: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&oldid=717634231#Lighthouse]
4: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=717634231]
5: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&oldid=716622943]
6: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=716622943]
7: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&oldid=716608512]
8: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=716608512]
...
20: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=716603919]

In this example, we did not save the results of the crawl in an external source. Normally this is necessary and can be stored in a file or database.

Here we will illustrate the use of the crawler4j (https://github.com/yasserg/crawler4j) web crawler. We will use an adapted version of the basic crawler found at https://github.com/yasserg/crawler4j/tree/master/src/test/java/edu/uci/ics/crawler4j/examples/basic. We will create two classes: CrawlerController and SampleCrawler. The former class set ups the crawler while the latter contains the logic that controls what pages will be processed.

As with our previous crawler, we will crawl the Wikipedia article dealing with Bishop Rock. The results using this crawler will be smaller as many extraneous pages are ignored.

Let's look at the CrawlerController class first. There are several parameters that are used with the crawler as detailed here:

The basic class is shown here:

public class CrawlerController { 
 
  public static void main(String[] args) throws Exception { 
    int numberOfCrawlers = 2; 
    CrawlConfig config = new CrawlConfig(); 
    String crawlStorageFolder = "data"; 
     
    config.setCrawlStorageFolder(crawlStorageFolder); 
    config.setPolitenessDelay(500); 
    config.setMaxDepthOfCrawling(2); 
    config.setMaxPagesToFetch(20); 
    config.setIncludeBinaryContentInCrawling(false); 
    ... 
  } 
}

Next, the CrawlController class is created and configured. Notice the RobotstxtConfig and RobotstxtServer classes used to handle robot.txt files. These files contain instructions that are intended to be read by a web crawler. They provide direction to help a crawler to do a better job such as specifying which parts of a site should not be crawled. This is useful for auto generated pages:

    PageFetcher pageFetcher = new PageFetcher(config); 
    RobotstxtConfig robotstxtConfig = new RobotstxtConfig(); 
    RobotstxtServer robotstxtServer =  
        new RobotstxtServer(robotstxtConfig, pageFetcher); 
    CrawlController controller =  
        new CrawlController(config, pageFetcher, robotstxtServer); 

The crawler needs to start at one or more pages. The addSeed method adds the starting pages. While we used the method only once here, it can be used as many times as needed:

    controller.addSeed( 
      "https://en.wikipedia.org/wiki/Bishop_Rock,_Isles_of_Scilly"); 

The start method will begin the crawling process:

    controller.start(SampleCrawler.class, numberOfCrawlers); 

The SampleCrawler class contains two methods of interest. The first is the shouldVisit method that determines whether a page will be visited and the visit method that actually handles the page. We start with the class declaration and the declaration of a Java regular expression class Pattern object. It will be one way of determining whether a page will be visited. In this declaration, standard images are specified and will be ignored:

    public class SampleCrawler extends WebCrawler { 
        private static final Pattern IMAGE_EXTENSIONS =  
            Pattern.compile(".*\\.(bmp|gif|jpg|png)$"); 
 
        ... 
    } 

The shouldVisit method is passed a reference to the page where this URL was found along with the URL. If any of the images match, the method returns false and the page is ignored. In addition, the URL must start with https://en.wikipedia.org/wiki/. We added this to restrict our searches to the Wikipedia website:

    public boolean shouldVisit(Page referringPage, WebURL url) { 
        String href = url.getURL().toLowerCase(); 
        if (IMAGE_EXTENSIONS.matcher(href).matches()) { 
            return false; 
        } 
        return href.startsWith("https://en.wikipedia.org/wiki/"); 
    }

The visit method is passed a Page object representing the page being visited. In this implementation, only those pages containing the string shipping route will be processed. This further restricts the pages visited. When we find such a page, its URL, Text, and Text length are displayed:

    public void visit(Page page) { 
        String url = page.getWebURL().getURL(); 
 
        if (page.getParseData() instanceof HtmlParseData) { 
            HtmlParseData htmlParseData =  
                (HtmlParseData) page.getParseData(); 
            String text = htmlParseData.getText(); 
            if (text.contains("shipping route")) { 
                out.println("\nURL: " + url); 
                out.println("Text: " + text); 
                out.println("Text length: " + text.length()); 
            } 
        } 
    } 

The following is the truncated output of the program when executed:

URL: https://en.wikipedia.org/wiki/Bishop_Rock,_Isles_of_Scilly
Text: Bishop Rock, Isles of Scilly...From Wikipedia, the free encyclopedia ... Jump to: ... navigation, search For the Bishop Rock in the Pacific Ocean, see Cortes Bank. Bishop Rock Bishop Rock Lighthouse (2005)
...
Text length: 14677

Notice that only one page was returned. This web crawler was able to identify and ignore previous versions of the main web page.

We could perform further processing, but this example provides some insight into how the API works. Significant amounts of information can be obtained when visiting a page. In the example, we only used the URL and the length of the text. The following is a sample of other data that you may be interested in obtaining:

OAuth is an open standard used to authenticate users to many different websites. A resource owner effectively delegates access to a server resource without having to share their credentials. It works over HTTPS. OAuth 2.0 succeeded OAuth and is not backwards compatible. It provides client developers a simple way of providing authentication. Several companies use OAuth 2.0 including PayPal, Comcast, and Blizzard Entertainment.

A list of OAuth 2.0 providers is found at https://en.wikipedia.org/wiki/List_of_OAuth_providers. We will use several of these in our discussions.

The sheer volume of data and the popularity of the site, among celebrities and the general public alike, make Twitter a valuable resource for mining social media data. Twitter is a popular social media platform allowing users to read and post short messages called tweets. Twitter provides API support for posting and pulling tweets, including streaming data from all public users. While there are services available for pulling the entire set of public tweet data, we are going to examine other options that, while limiting in the amount of data retrieved at one time, are available at no cost.

We are going to focus on the Twitter API for retrieving streaming data. There are other options for retrieving tweets from a specific user as well as posting data to a specific account but we will not be addressing those in this chapter. The public stream API, at the default access level, allows the user to pull a sample of public tweets currently streaming on Twitter. It is possible to refine the data by specifying parameters to track keywords, specific users, and location.

We are going to use HBC, a Java HTTP client, for this example. You can download a sample HBC application at https://github.com/twitter/hbc. If you prefer to use a different HTTP client, ensure it will return incremental response data. The Apache HTTP client is one option. Before you can create the HTTP connection, you must first create a Twitter account and an application within that account. To get started with the app, visit apps.twitter.com. Once your app is created, you will be assigned a consumer key, consumer secret, access token, and access secret token. We will also use OAuth, as discussed previously in this chapter.

First, we will write a method to perform the authentication and request data from Twitter. The parameters for our method are the authentication information given to us by Twitter when we created our app. We will create a BlockingQueue object to hold our streaming data. For this example, we will set a default capacity of 10,000. We will also specify our endpoint and turn off stall warnings:

    public static void streamTwitter( 
        String consumerKey, String consumerSecret,  
        String accessToken, String accessSecret)  
            throws InterruptedException { 
 
        BlockingQueue<String> statusQueue =  
            new LinkedBlockingQueue<String>(10000); 
        StatusesSampleEndpoint ending =  
            new StatusesSampleEndpoint(); 
        ending.stallWarnings(false); 
        ... 
    } 

Next, we create an Authentication object using OAuth1, a variation of the OAuth class. We can then build our connection client and complete the HTTP connection:

    Authentication twitterAuth = new OAuth1(consumerKey,  
        consumerSecret, accessToken, accessSecret); 
    BasicClient twitterClient = new ClientBuilder() 
            .name("Twitter client") 
            .hosts(Constants.STREAM_HOST) 
            .endpoint(ending) 
            .authentication(twitterAuth) 
            .processor(new StringDelimitedProcessor(statusQueue)) 
            .build(); 
    twitterClient.connect(); 

For the purposes of this example, we will simply read the messages received from the stream and print them to the screen. The messages are returned in JSON format and the decision of how to process them in a real application will depend upon the purpose and limitations of that application:

    for (int msgRead = 0; msgRead < 1000; msgRead++) { 
      if (twitterClient.isDone()) { 
        out.println(twitterClient.getExitEvent().getMessage()); 
        break; 
      } 
 
      String msg = statusQueue.poll(10, TimeUnit.SECONDS); 
      if (msg == null) { 
        out.println("Waited 10 seconds - no message received"); 
      } else { 
        out.println(msg); 
      } 
    } 
    twitterClient.stop(); 

To execute our method, we simply pass our authentication information to the streamTwitter method. For security purposes, we have replaced our personal keys here. Authentication information should always be protected:

    public static void main(String[] args) { 
   
      try { 
        SampleStreamExample.streamTwitter( 
            myKey, mySecret, myToken, myAccess);  
      } catch (InterruptedException e) { 
        out.println(e); 
      } 
    } 

Here is truncated sample data retrieved using the methods listed above. Your data will vary based upon Twitter's live stream, but it should resemble this example:

{"created_at":"Fri May 20 15:47:21 +0000 2016","id":733685552789098496,"id_str":"733685552789098496","text":"bwisit si em bahala sya","source":"\u003ca href="http:\/\/twitter.com" rel="nofollow"\u003eTwitter Web 
...
ntions":[],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"tl","timestamp_ms":"1463759241660"}

Twitter also provides support for pulling all data for one specific user account, as well as posting data directly to an account. A REST API is also available and provides support for specific queries via the search API. These also use the OAuth standard and return data in JSON files.

Wikipedia (https://www.wikipedia.org/) is a useful source of text and image type information. It is an Internet encyclopedia that hosts 38 million articles written in over 250 languages (https://en.wikipedia.org/wiki/Wikipedia). As such, it is useful to know how to programmatically access its contents.

MediaWiki is an open source wiki application that supports wiki type sites. It is used to support Wikipedia and many other sites. The MediaWiki API (http://www.mediawiki.org/wiki/API) provides access to a wiki's data and metadata over HTTP. An application, using this API, can log in, read data, and post changes to a site.

There are several Java APIs that support programmatic access to a wiki site as listed at https://www.mediawiki.org/wiki/API:Client_code#Java. To demonstrate Java access to a wiki we will use Bliki found at https://bitbucket.org/axelclk/info.bliki.wiki/wiki/Home. It provides good access and is easy to use for most basic operations.

The MediaWiki API is complex and has many features. The intent of this section is to illustrate the basic process of obtaining text from a Wikipedia article using this API. It is not possible to cover the API completely here.

We will use the following classes from the info.bliki.api and info.bliki.wiki.model packages:

Javadocs for Bliki are found at http://www.javadoc.io/doc/info.bliki.wiki/bliki-core/3.1.0.

The following example has been adapted from http://www.integratingstuff.com/2012/04/06/hook-into-wikipedia-using-java-and-the-mediawiki-api/. This example will access the English Wikipedia page for the subject, data science. We start by creating an instance of the User class. The first two arguments of the three-argument constructor are the user ID and password, respectively. In this case, they are empty strings. This combination allows us to read a page without having to set up an account. The third argument is the URL for the MediaWiki API page:

    User user = new User("", "",  
        "http://en.wikipedia.org/w/api.php"); 
    user.login(); 

An account will enable us to modify the document. The queryContent method returns a list of Page objects for the subjects found in a string array. Each string should be the title of a page. In this example, we access a single page:

    String[] titles = {"Data science"}; 
    List<Page> pageList = user.queryContent(titles); 

Each Page object contains the content of a page. There are several methods that will return the contents of the page. For each page, a WikiModel instance is created using the two-argument constructor. The first argument is the image base URL and the second argument is the link base URL. These URLs use Wiki variables called image and title, which will be replaced when creating links:

    for (Page page : pageList) { 
        WikiModel wikiModel = new WikiModel("${image}",  
            "${title}"); 
        ... 
    } 

The render method will take the wiki page and render it to HTML. There is also a method to render the page to a PDF document:

    String htmlText = wikiModel.render(page.toString()); 

The HTML text is then displayed:

    out.println(htmlText); 

A partial listing of the output follows:

<p>PageID: 35458904; NS: 0; Title: Data science; 
Image url: 
Content:
{{distinguish}}
{{Use dmy dates}}
{{Data Visualization}}</p>
<p><b>Data science</b> is an interdisciplinary field about processes and systems to extract <a href="Knowledge" >knowledge</a> 
...

We can also obtain basic information about the article using one of several methods as shown here:

    out.println("Title: " + page.getTitle() + "\n" + 
        "Page ID: " + page.getPageid() + "\n" + 
        "Timestamp: " + page.getCurrentRevision().getTimestamp()); 

It is also possible to obtain a list of references in the article and a list of the headers. Here, a list of the references is displayed:

    List <Reference> referenceList = wikiModel.getReferences(); 
    out.println(referenceList.size()); 
    for(Reference reference : referenceList) { 
        out.println(reference.getRefString()); 
    } 

The following illustrates the process of getting the section headers:

    ITableOfContent toc = wikiModel.getTableOfContent(); 
    List<SectionHeader> sections = toc.getSectionHeaders(); 
    for(SectionHeader sh : sections) { 
        out.println(sh.getFirst()); 
    } 

The entire content of Wikipedia can be downloaded. This process is discussed at https://en.wikipedia.org/wiki/Wikipedia:Database_download.

It may be desirable to set up your own Wikipedia server to handle your request.

Flickr (https://www.flickr.com/) is an online photo management and sharing application. It is a possible source for images and videos. The Flickr Developer Guide (https://www.flickr.com/services/developer/) is a good starting point to learn more about Flickr's API.

One of the first steps to using the Flickr API is to request an API key. This key is used to sign your API requests. The process to obtain a key starts at https://www.flickr.com/services/apps/create/. Both commercial and noncommercial keys are available. When you obtain a key you will also get a "secret." Both of these are required to use the API.

We will illustrate the process of locating and downloading images from Flickr. The process involves:

A FlickrException or IOException may be thrown during this process. There are several APIs that support Flickr access. We will be using Flickr4Java, found at https://github.com/callmeal/Flickr4Java. The Flickr4Java Javadocs is found at http://flickrj.sourceforge.net/api/. We will start with a try block and the apikey and secret declarations:

    try { 
        String apikey = "Your API key"; 
        String secret = "Your secret"; 
 
    } catch (FlickrException | IOException ex) { 
        // Handle exceptions 
    } 

The Flickr instance is created next, where the apikey and secret are supplied as the first two parameters. The last parameter specifies the transfer technique used to access Flickr servers. Currently, the REST transport is supported using the REST class:

    Flickr flickr = new Flickr(apikey, secret, new REST()); 

To search for images, we will use the SearchParameters class. This class supports a number of criteria that will narrow down the number of images returned from a query and includes such criteria as latitude, longitude, media type, and user ID. In the following sequence, the setBBox method specifies the longitude and latitude for the search. The parameters are (in order): minimum longitude, minimum latitude, maximum longitude, and maximum latitude. The setMedia method specifies the type of media. There are three possible arguments — "all", "photos", and "videos":

    SearchParameters searchParameters = new SearchParameters(); 
    searchParameters.setBBox("-180", "-90", "180", "90"); 
    searchParameters.setMedia("photos"); 

The PhotosInterface class possesses a search method that uses the SearchParameters instance to retrieve a list of photos. The getPhotosInterface method returns an instance of the PhotosInterface class, as shown next. The SearchParameters instance is the first parameter. The second parameter determines how many photos are retrieved per page and the third parameter is the offset. A PhotoList class instance is returned:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 
    PhotoList<Photo> list = pi.search(searchParameters, 10, 0); 

The next sequence illustrates the use of several methods to get information about the images retrieved. Each Photo instance is accessed using the get method. The title, image format, public flag, and photo URL are displayed:

    out.println("Image List"); 
    for (int i = 0; i < list.size(); i++) { 
        Photo photo = list.get(i); 
        out.println("Image: " + i + 
            `"\nTitle: " + photo.getTitle() +  
            "\nMedia: " + photo.getOriginalFormat() + 
            "\nPublic: " + photo.isPublicFlag() + 
            "\nUrl: " + photo.getUrl() + 
            "\n"); 
    } 
    out.println(); 

A partial listing is shown here where many of the specific values have been modified to protect the original data:

Image List
Image: 0
Title: XYZ Image
Media: jpg
Public: true
Url: https://flickr.com/photos/7723...@N02/269...
Image: 1
Title: IMG_5555.jpg
Media: jpg
Public: true
Url: https://flickr.com/photos/2665...@N07/264...
Image: 2
Title: DSC05555
Media: jpg
Public: true
Url: https://flickr.com/photos/1179...@N04/264...

The list of images returned by this example will vary since we used a fairly wide search range and images are being added all of the time.

There are two approaches that we can use to download an image. The first uses the image's URL and the second uses a Photo object. The image's URL can be obtained from a number of sources. We use the Photo class getUrl method for this example.

In the following sequence, we obtain an instance of PhotosInterface using its constructor to illustrate an alternate approach:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 

We get the first Photo instance from the previous list and then its getUrl to get the image's URL. The PhotosInterface class's getImage method returns a BufferedImage object representing the image as shown here:

    Photo currentPhoto = list.get(0);  
    BufferedImage bufferedImage =  
        pi.getImage(currentPhoto.getUrl()); 

The image is then saved to a file using the ImageIO class:

    File outputfile = new File("image.jpg"); 
    ImageIO.write(bufferedImage, "jpg", outputfile); 

The getImage method is overloaded. Here, the Photo instance and the size of the image desired are used as arguments to get the BufferedImage instance:

    bufferedImage = pi.getImage(currentPhoto, Size.SMALL); 

The image can be saved to a file using the previous technique.

The Flickr4Java API supports a number of other techniques for working with Flickr images.

YouTube is a popular video site where users can upload and share videos (https://www.youtube.com/). It has been used to share humorous videos, provide instructions on how to do any number of things, and share information among its viewers. It is a useful source of information as it captures the thoughts and ideas of a diverse group of people. This provides an interesting opportunity to analysis and gain insight into human behavior.

YouTube can serve as a useful source of videos and video metadata. A Java API is available to access its contents (https://developers.google.com/youtube/v3/). Detailed documentation of the API is found at https://developers.google.com/youtube/v3/docs/.

In this section, we will demonstrate how to search for videos by keyword and retrieve information of interest. We will also show how to download a video. To use the YouTube API, you will need a Google account, which can be obtained at https://www.google.com/accounts/NewAccount. Next, create an account in the Google Developer's Console (https://console.developers.google.com/). API access is supported using either API keys or OAuth 2.0 credentials. The project creation process and keys are discussed at https://developers.google.com/youtube/registering_an_application#create_project.

    try { 
        YouTube youtube = new YouTube.Builder( 
            Auth.HTTP_TRANSPORT, 
            Auth.JSON_FACTORY, 
            new HttpRequestInitializer() { 
                public void initialize(HttpRequest request)  
                        throws IOException { 
                } 
            }) 
                .setApplicationName("application_name") 
        ... 
    } catch (GoogleJSONResponseException ex) { 
        // Handle exceptions 
    } catch (IOException ex) { 
        // Handle exceptions 
    } 

    String queryTerm = "cats"; 

    YouTube.Search.List search = youtube 
        .search() 
        .list("id,snippet"); 

{ 
  "kind": "youtube#searchResult", 
  "etag": etag, 
  "id": { 
    "kind": string, 
    "videoId": string, 
    "channelId": string, 
    "playlistId": string 
  }, 
  "snippet": { 
    "publishedAt": datetime, 
    "channelId": string, 
    "title": string, 
    "description": string, 
    "thumbnails": { 
      (key): { 
        "url": string, 
        "width": unsigned integer, 
        "height": unsigned integer 
      } 
    }, 
    "channelTitle": string, 
    "liveBroadcastContent": string 
  } 
} 

    String apiKey = "Your API key"; 
    search.setKey(apiKey); 
    search.setQ(queryTerm); 
    search.setType("video"); 

    search.setFields("items(id/kind,id/videoId,snippet/title," +  
        "snippet/description,snippet/thumbnails/default/url)"); 

    search.setMaxResults(10L); 

    SearchListResponse searchResponse = search.execute(); 
    List<SearchResult> searchResultList =  
        searchResponse.getItems(); 

    SearchResult video = searchResultList.iterator().next(); 
    Thumbnail thumbnail = video 
        .getSnippet().getThumbnails().getDefault(); 
 
    out.println("Kind: " + video.getKind()); 
    out.println("Video Id: " + video.getId().getVideoId()); 
    out.println("Title: " + video.getSnippet().getTitle()); 
    out.println("Description: " +  
        video.getSnippet().getDescription()); 
    out.println("Thumbnail: " + thumbnail.getUrl()); 

Kind: null
Video Id: tntO...
Title: Funny Cats ...
Description: Check out the ...
Thumbnail: https://i.ytimg.com/vi/tntO.../default.jpg

    String url = "http://www.youtube.com/watch?v=videoID"; 
    String path = "."; 
    VGet vget = new VGet(new URL(url), new File(path)); 
    vget.download(); 

Now that we have a basic understanding of web crawlers, we are ready to create our own. In this simple web crawler, we will keep track of the pages visited using ArrayList instances. In addition, jsoup will be used to parse a web page and we will limit the number of pages we visit. Jsoup (https://jsoup.org/) is an open source HTML parser. This example demonstrates the basic structure of a web crawler and also highlights some of the issues involved in creating a web crawler.

We will use the SimpleWebCrawler class, as declared here:

public class SimpleWebCrawler { 
 
    private String topic; 
    private String startingURL; 
    private String urlLimiter; 
    private final int pageLimit = 20; 
    private ArrayList<String> visitedList = new ArrayList<>(); 
    private ArrayList<String> pageList = new ArrayList<>(); 
    ... 
    public static void main(String[] args) { 
        new SimpleWebCrawler(); 
    } 
 
} 

The instance variables are detailed here:

In the SimpleWebCrawler constructor, we initialize the instance variables to begin the search from the Wikipedia page for Bishop Rock, an island off the coast of Italy. This was chosen to minimize the number of pages that might be retrieved. As we will see, there are many more Wikipedia pages dealing with Bishop Rock than one might think.

The urlLimiter variable is set to Bishop_Rock, which will restrict the embedded links to follow to just those containing that string. Each page of interest must contain the value stored in the topic variable. The visitPage method performs the actual crawl:

    public SimpleWebCrawler() { 
        startingURL = https://en.wikipedia.org/wiki/Bishop_Rock, " 
            + "Isles_of_Scilly"; 
        urlLimiter = "Bishop_Rock"; 
        topic = "shipping route"; 
        visitPage(startingURL); 
    } 

In the visitPage method, the pageList ArrayList is checked to see whether the maximum number of accepted pages has been exceeded. If the limit has been exceeded, then the search terminates:

    public void visitPage(String url) { 
        if (pageList.size() >= pageLimit) { 
            return; 
        } 
       ... 
    } 

If the page has already been visited, then we ignore it. Otherwise, it is added to the visited list:

    if (visitedList.contains(url)) { 
        // URL already visited 
    } else { 
        visitedList.add(url); 
            ... 
    } 

Jsoup is used to parse the page and return a Document object. There are many different exceptions and problems that can occur such as a malformed URL, retrieval timeouts, or simply bad links. The catch block needs to handle these types of problems. We will provide a more in-depth explanation of jsoup in web scraping in Java:

    try { 
        Document doc = Jsoup.connect(url).get(); 
            ... 
        } 
    } catch (Exception ex) { 
        // Handle exceptions 
    } 

If the document contains the topic text, then the link is displayed and added to the pageList ArrayList. Each embedded link is obtained, and if the link contains the limiting text, then the visitPage method is called recursively:

    if (doc.text().contains(topic)) { 
        out.println((pageList.size() + 1) + ": [" + url + "]"); 
        pageList.add(url); 
 
        // Process page links 
        Elements questions = doc.select("a[href]"); 
        for (Element link : questions) { 
            if (link.attr("href").contains(urlLimiter)) { 
                visitPage(link.attr("abs:href")); 
            } 
        } 
    } 

This approach only examines links in those pages that contain the topic text. Moving the for loop outside of the if statement will test the links for all pages.

The output follows:

1: [https://en.wikipedia.org/wiki/Bishop_Rock,_Isles_of_Scilly]
2: [https://en.wikipedia.org/wiki/Bishop_Rock_Lighthouse]
3: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&oldid=717634231#Lighthouse]
4: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=717634231]
5: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&oldid=716622943]
6: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=716622943]
7: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&oldid=716608512]
8: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=716608512]
...
20: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=716603919]

In this example, we did not save the results of the crawl in an external source. Normally this is necessary and can be stored in a file or database.

Here we will illustrate the use of the crawler4j (https://github.com/yasserg/crawler4j) web crawler. We will use an adapted version of the basic crawler found at https://github.com/yasserg/crawler4j/tree/master/src/test/java/edu/uci/ics/crawler4j/examples/basic. We will create two classes: CrawlerController and SampleCrawler. The former class set ups the crawler while the latter contains the logic that controls what pages will be processed.

As with our previous crawler, we will crawl the Wikipedia article dealing with Bishop Rock. The results using this crawler will be smaller as many extraneous pages are ignored.

Let's look at the CrawlerController class first. There are several parameters that are used with the crawler as detailed here:

The basic class is shown here:

public class CrawlerController { 
 
  public static void main(String[] args) throws Exception { 
    int numberOfCrawlers = 2; 
    CrawlConfig config = new CrawlConfig(); 
    String crawlStorageFolder = "data"; 
     
    config.setCrawlStorageFolder(crawlStorageFolder); 
    config.setPolitenessDelay(500); 
    config.setMaxDepthOfCrawling(2); 
    config.setMaxPagesToFetch(20); 
    config.setIncludeBinaryContentInCrawling(false); 
    ... 
  } 
}

Next, the CrawlController class is created and configured. Notice the RobotstxtConfig and RobotstxtServer classes used to handle robot.txt files. These files contain instructions that are intended to be read by a web crawler. They provide direction to help a crawler to do a better job such as specifying which parts of a site should not be crawled. This is useful for auto generated pages:

    PageFetcher pageFetcher = new PageFetcher(config); 
    RobotstxtConfig robotstxtConfig = new RobotstxtConfig(); 
    RobotstxtServer robotstxtServer =  
        new RobotstxtServer(robotstxtConfig, pageFetcher); 
    CrawlController controller =  
        new CrawlController(config, pageFetcher, robotstxtServer); 

The crawler needs to start at one or more pages. The addSeed method adds the starting pages. While we used the method only once here, it can be used as many times as needed:

    controller.addSeed( 
      "https://en.wikipedia.org/wiki/Bishop_Rock,_Isles_of_Scilly"); 

The start method will begin the crawling process:

    controller.start(SampleCrawler.class, numberOfCrawlers); 

The SampleCrawler class contains two methods of interest. The first is the shouldVisit method that determines whether a page will be visited and the visit method that actually handles the page. We start with the class declaration and the declaration of a Java regular expression class Pattern object. It will be one way of determining whether a page will be visited. In this declaration, standard images are specified and will be ignored:

    public class SampleCrawler extends WebCrawler { 
        private static final Pattern IMAGE_EXTENSIONS =  
            Pattern.compile(".*\\.(bmp|gif|jpg|png)$"); 
 
        ... 
    } 

The shouldVisit method is passed a reference to the page where this URL was found along with the URL. If any of the images match, the method returns false and the page is ignored. In addition, the URL must start with https://en.wikipedia.org/wiki/. We added this to restrict our searches to the Wikipedia website:

    public boolean shouldVisit(Page referringPage, WebURL url) { 
        String href = url.getURL().toLowerCase(); 
        if (IMAGE_EXTENSIONS.matcher(href).matches()) { 
            return false; 
        } 
        return href.startsWith("https://en.wikipedia.org/wiki/"); 
    }

The visit method is passed a Page object representing the page being visited. In this implementation, only those pages containing the string shipping route will be processed. This further restricts the pages visited. When we find such a page, its URL, Text, and Text length are displayed:

    public void visit(Page page) { 
        String url = page.getWebURL().getURL(); 
 
        if (page.getParseData() instanceof HtmlParseData) { 
            HtmlParseData htmlParseData =  
                (HtmlParseData) page.getParseData(); 
            String text = htmlParseData.getText(); 
            if (text.contains("shipping route")) { 
                out.println("\nURL: " + url); 
                out.println("Text: " + text); 
                out.println("Text length: " + text.length()); 
            } 
        } 
    } 

The following is the truncated output of the program when executed:

URL: https://en.wikipedia.org/wiki/Bishop_Rock,_Isles_of_Scilly
Text: Bishop Rock, Isles of Scilly...From Wikipedia, the free encyclopedia ... Jump to: ... navigation, search For the Bishop Rock in the Pacific Ocean, see Cortes Bank. Bishop Rock Bishop Rock Lighthouse (2005)
...
Text length: 14677

Notice that only one page was returned. This web crawler was able to identify and ignore previous versions of the main web page.

We could perform further processing, but this example provides some insight into how the API works. Significant amounts of information can be obtained when visiting a page. In the example, we only used the URL and the length of the text. The following is a sample of other data that you may be interested in obtaining:

OAuth is an open standard used to authenticate users to many different websites. A resource owner effectively delegates access to a server resource without having to share their credentials. It works over HTTPS. OAuth 2.0 succeeded OAuth and is not backwards compatible. It provides client developers a simple way of providing authentication. Several companies use OAuth 2.0 including PayPal, Comcast, and Blizzard Entertainment.

A list of OAuth 2.0 providers is found at https://en.wikipedia.org/wiki/List_of_OAuth_providers. We will use several of these in our discussions.

The sheer volume of data and the popularity of the site, among celebrities and the general public alike, make Twitter a valuable resource for mining social media data. Twitter is a popular social media platform allowing users to read and post short messages called tweets. Twitter provides API support for posting and pulling tweets, including streaming data from all public users. While there are services available for pulling the entire set of public tweet data, we are going to examine other options that, while limiting in the amount of data retrieved at one time, are available at no cost.

We are going to focus on the Twitter API for retrieving streaming data. There are other options for retrieving tweets from a specific user as well as posting data to a specific account but we will not be addressing those in this chapter. The public stream API, at the default access level, allows the user to pull a sample of public tweets currently streaming on Twitter. It is possible to refine the data by specifying parameters to track keywords, specific users, and location.

We are going to use HBC, a Java HTTP client, for this example. You can download a sample HBC application at https://github.com/twitter/hbc. If you prefer to use a different HTTP client, ensure it will return incremental response data. The Apache HTTP client is one option. Before you can create the HTTP connection, you must first create a Twitter account and an application within that account. To get started with the app, visit apps.twitter.com. Once your app is created, you will be assigned a consumer key, consumer secret, access token, and access secret token. We will also use OAuth, as discussed previously in this chapter.

First, we will write a method to perform the authentication and request data from Twitter. The parameters for our method are the authentication information given to us by Twitter when we created our app. We will create a BlockingQueue object to hold our streaming data. For this example, we will set a default capacity of 10,000. We will also specify our endpoint and turn off stall warnings:

    public static void streamTwitter( 
        String consumerKey, String consumerSecret,  
        String accessToken, String accessSecret)  
            throws InterruptedException { 
 
        BlockingQueue<String> statusQueue =  
            new LinkedBlockingQueue<String>(10000); 
        StatusesSampleEndpoint ending =  
            new StatusesSampleEndpoint(); 
        ending.stallWarnings(false); 
        ... 
    } 

Next, we create an Authentication object using OAuth1, a variation of the OAuth class. We can then build our connection client and complete the HTTP connection:

    Authentication twitterAuth = new OAuth1(consumerKey,  
        consumerSecret, accessToken, accessSecret); 
    BasicClient twitterClient = new ClientBuilder() 
            .name("Twitter client") 
            .hosts(Constants.STREAM_HOST) 
            .endpoint(ending) 
            .authentication(twitterAuth) 
            .processor(new StringDelimitedProcessor(statusQueue)) 
            .build(); 
    twitterClient.connect(); 

For the purposes of this example, we will simply read the messages received from the stream and print them to the screen. The messages are returned in JSON format and the decision of how to process them in a real application will depend upon the purpose and limitations of that application:

    for (int msgRead = 0; msgRead < 1000; msgRead++) { 
      if (twitterClient.isDone()) { 
        out.println(twitterClient.getExitEvent().getMessage()); 
        break; 
      } 
 
      String msg = statusQueue.poll(10, TimeUnit.SECONDS); 
      if (msg == null) { 
        out.println("Waited 10 seconds - no message received"); 
      } else { 
        out.println(msg); 
      } 
    } 
    twitterClient.stop(); 

To execute our method, we simply pass our authentication information to the streamTwitter method. For security purposes, we have replaced our personal keys here. Authentication information should always be protected:

    public static void main(String[] args) { 
   
      try { 
        SampleStreamExample.streamTwitter( 
            myKey, mySecret, myToken, myAccess);  
      } catch (InterruptedException e) { 
        out.println(e); 
      } 
    } 

Here is truncated sample data retrieved using the methods listed above. Your data will vary based upon Twitter's live stream, but it should resemble this example:

{"created_at":"Fri May 20 15:47:21 +0000 2016","id":733685552789098496,"id_str":"733685552789098496","text":"bwisit si em bahala sya","source":"\u003ca href="http:\/\/twitter.com" rel="nofollow"\u003eTwitter Web 
...
ntions":[],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"tl","timestamp_ms":"1463759241660"}

Twitter also provides support for pulling all data for one specific user account, as well as posting data directly to an account. A REST API is also available and provides support for specific queries via the search API. These also use the OAuth standard and return data in JSON files.

Wikipedia (https://www.wikipedia.org/) is a useful source of text and image type information. It is an Internet encyclopedia that hosts 38 million articles written in over 250 languages (https://en.wikipedia.org/wiki/Wikipedia). As such, it is useful to know how to programmatically access its contents.

MediaWiki is an open source wiki application that supports wiki type sites. It is used to support Wikipedia and many other sites. The MediaWiki API (http://www.mediawiki.org/wiki/API) provides access to a wiki's data and metadata over HTTP. An application, using this API, can log in, read data, and post changes to a site.

There are several Java APIs that support programmatic access to a wiki site as listed at https://www.mediawiki.org/wiki/API:Client_code#Java. To demonstrate Java access to a wiki we will use Bliki found at https://bitbucket.org/axelclk/info.bliki.wiki/wiki/Home. It provides good access and is easy to use for most basic operations.

The MediaWiki API is complex and has many features. The intent of this section is to illustrate the basic process of obtaining text from a Wikipedia article using this API. It is not possible to cover the API completely here.

We will use the following classes from the info.bliki.api and info.bliki.wiki.model packages:

Javadocs for Bliki are found at http://www.javadoc.io/doc/info.bliki.wiki/bliki-core/3.1.0.

The following example has been adapted from http://www.integratingstuff.com/2012/04/06/hook-into-wikipedia-using-java-and-the-mediawiki-api/. This example will access the English Wikipedia page for the subject, data science. We start by creating an instance of the User class. The first two arguments of the three-argument constructor are the user ID and password, respectively. In this case, they are empty strings. This combination allows us to read a page without having to set up an account. The third argument is the URL for the MediaWiki API page:

    User user = new User("", "",  
        "http://en.wikipedia.org/w/api.php"); 
    user.login(); 

An account will enable us to modify the document. The queryContent method returns a list of Page objects for the subjects found in a string array. Each string should be the title of a page. In this example, we access a single page:

    String[] titles = {"Data science"}; 
    List<Page> pageList = user.queryContent(titles); 

Each Page object contains the content of a page. There are several methods that will return the contents of the page. For each page, a WikiModel instance is created using the two-argument constructor. The first argument is the image base URL and the second argument is the link base URL. These URLs use Wiki variables called image and title, which will be replaced when creating links:

    for (Page page : pageList) { 
        WikiModel wikiModel = new WikiModel("${image}",  
            "${title}"); 
        ... 
    } 

The render method will take the wiki page and render it to HTML. There is also a method to render the page to a PDF document:

    String htmlText = wikiModel.render(page.toString()); 

The HTML text is then displayed:

    out.println(htmlText); 

A partial listing of the output follows:

<p>PageID: 35458904; NS: 0; Title: Data science; 
Image url: 
Content:
{{distinguish}}
{{Use dmy dates}}
{{Data Visualization}}</p>
<p><b>Data science</b> is an interdisciplinary field about processes and systems to extract <a href="Knowledge" >knowledge</a> 
...

We can also obtain basic information about the article using one of several methods as shown here:

    out.println("Title: " + page.getTitle() + "\n" + 
        "Page ID: " + page.getPageid() + "\n" + 
        "Timestamp: " + page.getCurrentRevision().getTimestamp()); 

It is also possible to obtain a list of references in the article and a list of the headers. Here, a list of the references is displayed:

    List <Reference> referenceList = wikiModel.getReferences(); 
    out.println(referenceList.size()); 
    for(Reference reference : referenceList) { 
        out.println(reference.getRefString()); 
    } 

The following illustrates the process of getting the section headers:

    ITableOfContent toc = wikiModel.getTableOfContent(); 
    List<SectionHeader> sections = toc.getSectionHeaders(); 
    for(SectionHeader sh : sections) { 
        out.println(sh.getFirst()); 
    } 

The entire content of Wikipedia can be downloaded. This process is discussed at https://en.wikipedia.org/wiki/Wikipedia:Database_download.

It may be desirable to set up your own Wikipedia server to handle your request.

Flickr (https://www.flickr.com/) is an online photo management and sharing application. It is a possible source for images and videos. The Flickr Developer Guide (https://www.flickr.com/services/developer/) is a good starting point to learn more about Flickr's API.

One of the first steps to using the Flickr API is to request an API key. This key is used to sign your API requests. The process to obtain a key starts at https://www.flickr.com/services/apps/create/. Both commercial and noncommercial keys are available. When you obtain a key you will also get a "secret." Both of these are required to use the API.

We will illustrate the process of locating and downloading images from Flickr. The process involves:

A FlickrException or IOException may be thrown during this process. There are several APIs that support Flickr access. We will be using Flickr4Java, found at https://github.com/callmeal/Flickr4Java. The Flickr4Java Javadocs is found at http://flickrj.sourceforge.net/api/. We will start with a try block and the apikey and secret declarations:

    try { 
        String apikey = "Your API key"; 
        String secret = "Your secret"; 
 
    } catch (FlickrException | IOException ex) { 
        // Handle exceptions 
    } 

The Flickr instance is created next, where the apikey and secret are supplied as the first two parameters. The last parameter specifies the transfer technique used to access Flickr servers. Currently, the REST transport is supported using the REST class:

    Flickr flickr = new Flickr(apikey, secret, new REST()); 

To search for images, we will use the SearchParameters class. This class supports a number of criteria that will narrow down the number of images returned from a query and includes such criteria as latitude, longitude, media type, and user ID. In the following sequence, the setBBox method specifies the longitude and latitude for the search. The parameters are (in order): minimum longitude, minimum latitude, maximum longitude, and maximum latitude. The setMedia method specifies the type of media. There are three possible arguments — "all", "photos", and "videos":

    SearchParameters searchParameters = new SearchParameters(); 
    searchParameters.setBBox("-180", "-90", "180", "90"); 
    searchParameters.setMedia("photos"); 

The PhotosInterface class possesses a search method that uses the SearchParameters instance to retrieve a list of photos. The getPhotosInterface method returns an instance of the PhotosInterface class, as shown next. The SearchParameters instance is the first parameter. The second parameter determines how many photos are retrieved per page and the third parameter is the offset. A PhotoList class instance is returned:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 
    PhotoList<Photo> list = pi.search(searchParameters, 10, 0); 

The next sequence illustrates the use of several methods to get information about the images retrieved. Each Photo instance is accessed using the get method. The title, image format, public flag, and photo URL are displayed:

    out.println("Image List"); 
    for (int i = 0; i < list.size(); i++) { 
        Photo photo = list.get(i); 
        out.println("Image: " + i + 
            `"\nTitle: " + photo.getTitle() +  
            "\nMedia: " + photo.getOriginalFormat() + 
            "\nPublic: " + photo.isPublicFlag() + 
            "\nUrl: " + photo.getUrl() + 
            "\n"); 
    } 
    out.println(); 

A partial listing is shown here where many of the specific values have been modified to protect the original data:

Image List
Image: 0
Title: XYZ Image
Media: jpg
Public: true
Url: https://flickr.com/photos/7723...@N02/269...
Image: 1
Title: IMG_5555.jpg
Media: jpg
Public: true
Url: https://flickr.com/photos/2665...@N07/264...
Image: 2
Title: DSC05555
Media: jpg
Public: true
Url: https://flickr.com/photos/1179...@N04/264...

The list of images returned by this example will vary since we used a fairly wide search range and images are being added all of the time.

There are two approaches that we can use to download an image. The first uses the image's URL and the second uses a Photo object. The image's URL can be obtained from a number of sources. We use the Photo class getUrl method for this example.

In the following sequence, we obtain an instance of PhotosInterface using its constructor to illustrate an alternate approach:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 

We get the first Photo instance from the previous list and then its getUrl to get the image's URL. The PhotosInterface class's getImage method returns a BufferedImage object representing the image as shown here:

    Photo currentPhoto = list.get(0);  
    BufferedImage bufferedImage =  
        pi.getImage(currentPhoto.getUrl()); 

The image is then saved to a file using the ImageIO class:

    File outputfile = new File("image.jpg"); 
    ImageIO.write(bufferedImage, "jpg", outputfile); 

The getImage method is overloaded. Here, the Photo instance and the size of the image desired are used as arguments to get the BufferedImage instance:

    bufferedImage = pi.getImage(currentPhoto, Size.SMALL); 

The image can be saved to a file using the previous technique.

The Flickr4Java API supports a number of other techniques for working with Flickr images.

YouTube is a popular video site where users can upload and share videos (https://www.youtube.com/). It has been used to share humorous videos, provide instructions on how to do any number of things, and share information among its viewers. It is a useful source of information as it captures the thoughts and ideas of a diverse group of people. This provides an interesting opportunity to analysis and gain insight into human behavior.

YouTube can serve as a useful source of videos and video metadata. A Java API is available to access its contents (https://developers.google.com/youtube/v3/). Detailed documentation of the API is found at https://developers.google.com/youtube/v3/docs/.

In this section, we will demonstrate how to search for videos by keyword and retrieve information of interest. We will also show how to download a video. To use the YouTube API, you will need a Google account, which can be obtained at https://www.google.com/accounts/NewAccount. Next, create an account in the Google Developer's Console (https://console.developers.google.com/). API access is supported using either API keys or OAuth 2.0 credentials. The project creation process and keys are discussed at https://developers.google.com/youtube/registering_an_application#create_project.

    try { 
        YouTube youtube = new YouTube.Builder( 
            Auth.HTTP_TRANSPORT, 
            Auth.JSON_FACTORY, 
            new HttpRequestInitializer() { 
                public void initialize(HttpRequest request)  
                        throws IOException { 
                } 
            }) 
                .setApplicationName("application_name") 
        ... 
    } catch (GoogleJSONResponseException ex) { 
        // Handle exceptions 
    } catch (IOException ex) { 
        // Handle exceptions 
    } 

    String queryTerm = "cats"; 

    YouTube.Search.List search = youtube 
        .search() 
        .list("id,snippet"); 

{ 
  "kind": "youtube#searchResult", 
  "etag": etag, 
  "id": { 
    "kind": string, 
    "videoId": string, 
    "channelId": string, 
    "playlistId": string 
  }, 
  "snippet": { 
    "publishedAt": datetime, 
    "channelId": string, 
    "title": string, 
    "description": string, 
    "thumbnails": { 
      (key): { 
        "url": string, 
        "width": unsigned integer, 
        "height": unsigned integer 
      } 
    }, 
    "channelTitle": string, 
    "liveBroadcastContent": string 
  } 
} 

    String apiKey = "Your API key"; 
    search.setKey(apiKey); 
    search.setQ(queryTerm); 
    search.setType("video"); 

    search.setFields("items(id/kind,id/videoId,snippet/title," +  
        "snippet/description,snippet/thumbnails/default/url)"); 

    search.setMaxResults(10L); 

    SearchListResponse searchResponse = search.execute(); 
    List<SearchResult> searchResultList =  
        searchResponse.getItems(); 

    SearchResult video = searchResultList.iterator().next(); 
    Thumbnail thumbnail = video 
        .getSnippet().getThumbnails().getDefault(); 
 
    out.println("Kind: " + video.getKind()); 
    out.println("Video Id: " + video.getId().getVideoId()); 
    out.println("Title: " + video.getSnippet().getTitle()); 
    out.println("Description: " +  
        video.getSnippet().getDescription()); 
    out.println("Thumbnail: " + thumbnail.getUrl()); 

Kind: null
Video Id: tntO...
Title: Funny Cats ...
Description: Check out the ...
Thumbnail: https://i.ytimg.com/vi/tntO.../default.jpg

    String url = "http://www.youtube.com/watch?v=videoID"; 
    String path = "."; 
    VGet vget = new VGet(new URL(url), new File(path)); 
    vget.download(); 

Now that we have a basic understanding of web crawlers, we are ready to create our own. In this simple web crawler, we will keep track of the pages visited using ArrayList instances. In addition, jsoup will be used to parse a web page and we will limit the number of pages we visit. Jsoup (https://jsoup.org/) is an open source HTML parser. This example demonstrates the basic structure of a web crawler and also highlights some of the issues involved in creating a web crawler.

We will use the SimpleWebCrawler class, as declared here:

public class SimpleWebCrawler { 
 
    private String topic; 
    private String startingURL; 
    private String urlLimiter; 
    private final int pageLimit = 20; 
    private ArrayList<String> visitedList = new ArrayList<>(); 
    private ArrayList<String> pageList = new ArrayList<>(); 
    ... 
    public static void main(String[] args) { 
        new SimpleWebCrawler(); 
    } 
 
} 

The instance variables are detailed here:

In the SimpleWebCrawler constructor, we initialize the instance variables to begin the search from the Wikipedia page for Bishop Rock, an island off the coast of Italy. This was chosen to minimize the number of pages that might be retrieved. As we will see, there are many more Wikipedia pages dealing with Bishop Rock than one might think.

The urlLimiter variable is set to Bishop_Rock, which will restrict the embedded links to follow to just those containing that string. Each page of interest must contain the value stored in the topic variable. The visitPage method performs the actual crawl:

    public SimpleWebCrawler() { 
        startingURL = https://en.wikipedia.org/wiki/Bishop_Rock, " 
            + "Isles_of_Scilly"; 
        urlLimiter = "Bishop_Rock"; 
        topic = "shipping route"; 
        visitPage(startingURL); 
    } 

In the visitPage method, the pageList ArrayList is checked to see whether the maximum number of accepted pages has been exceeded. If the limit has been exceeded, then the search terminates:

    public void visitPage(String url) { 
        if (pageList.size() >= pageLimit) { 
            return; 
        } 
       ... 
    } 

If the page has already been visited, then we ignore it. Otherwise, it is added to the visited list:

    if (visitedList.contains(url)) { 
        // URL already visited 
    } else { 
        visitedList.add(url); 
            ... 
    } 

Jsoup is used to parse the page and return a Document object. There are many different exceptions and problems that can occur such as a malformed URL, retrieval timeouts, or simply bad links. The catch block needs to handle these types of problems. We will provide a more in-depth explanation of jsoup in web scraping in Java:

    try { 
        Document doc = Jsoup.connect(url).get(); 
            ... 
        } 
    } catch (Exception ex) { 
        // Handle exceptions 
    } 

If the document contains the topic text, then the link is displayed and added to the pageList ArrayList. Each embedded link is obtained, and if the link contains the limiting text, then the visitPage method is called recursively:

    if (doc.text().contains(topic)) { 
        out.println((pageList.size() + 1) + ": [" + url + "]"); 
        pageList.add(url); 
 
        // Process page links 
        Elements questions = doc.select("a[href]"); 
        for (Element link : questions) { 
            if (link.attr("href").contains(urlLimiter)) { 
                visitPage(link.attr("abs:href")); 
            } 
        } 
    } 

This approach only examines links in those pages that contain the topic text. Moving the for loop outside of the if statement will test the links for all pages.

The output follows:

1: [https://en.wikipedia.org/wiki/Bishop_Rock,_Isles_of_Scilly]
2: [https://en.wikipedia.org/wiki/Bishop_Rock_Lighthouse]
3: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&oldid=717634231#Lighthouse]
4: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=717634231]
5: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&oldid=716622943]
6: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=716622943]
7: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&oldid=716608512]
8: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=716608512]
...
20: [https://en.wikipedia.org/w/index.php?title=Bishop_Rock,_Isles_of_Scilly&diff=prev&oldid=716603919]

In this example, we did not save the results of the crawl in an external source. Normally this is necessary and can be stored in a file or database.

Here we will illustrate the use of the crawler4j (https://github.com/yasserg/crawler4j) web crawler. We will use an adapted version of the basic crawler found at https://github.com/yasserg/crawler4j/tree/master/src/test/java/edu/uci/ics/crawler4j/examples/basic. We will create two classes: CrawlerController and SampleCrawler. The former class set ups the crawler while the latter contains the logic that controls what pages will be processed.

As with our previous crawler, we will crawl the Wikipedia article dealing with Bishop Rock. The results using this crawler will be smaller as many extraneous pages are ignored.

Let's look at the CrawlerController class first. There are several parameters that are used with the crawler as detailed here:

The basic class is shown here:

public class CrawlerController { 
 
  public static void main(String[] args) throws Exception { 
    int numberOfCrawlers = 2; 
    CrawlConfig config = new CrawlConfig(); 
    String crawlStorageFolder = "data"; 
     
    config.setCrawlStorageFolder(crawlStorageFolder); 
    config.setPolitenessDelay(500); 
    config.setMaxDepthOfCrawling(2); 
    config.setMaxPagesToFetch(20); 
    config.setIncludeBinaryContentInCrawling(false); 
    ... 
  } 
}

Next, the CrawlController class is created and configured. Notice the RobotstxtConfig and RobotstxtServer classes used to handle robot.txt files. These files contain instructions that are intended to be read by a web crawler. They provide direction to help a crawler to do a better job such as specifying which parts of a site should not be crawled. This is useful for auto generated pages:

    PageFetcher pageFetcher = new PageFetcher(config); 
    RobotstxtConfig robotstxtConfig = new RobotstxtConfig(); 
    RobotstxtServer robotstxtServer =  
        new RobotstxtServer(robotstxtConfig, pageFetcher); 
    CrawlController controller =  
        new CrawlController(config, pageFetcher, robotstxtServer); 

The crawler needs to start at one or more pages. The addSeed method adds the starting pages. While we used the method only once here, it can be used as many times as needed:

    controller.addSeed( 
      "https://en.wikipedia.org/wiki/Bishop_Rock,_Isles_of_Scilly"); 

The start method will begin the crawling process:

    controller.start(SampleCrawler.class, numberOfCrawlers); 

The SampleCrawler class contains two methods of interest. The first is the shouldVisit method that determines whether a page will be visited and the visit method that actually handles the page. We start with the class declaration and the declaration of a Java regular expression class Pattern object. It will be one way of determining whether a page will be visited. In this declaration, standard images are specified and will be ignored:

    public class SampleCrawler extends WebCrawler { 
        private static final Pattern IMAGE_EXTENSIONS =  
            Pattern.compile(".*\\.(bmp|gif|jpg|png)$"); 
 
        ... 
    } 

The shouldVisit method is passed a reference to the page where this URL was found along with the URL. If any of the images match, the method returns false and the page is ignored. In addition, the URL must start with https://en.wikipedia.org/wiki/. We added this to restrict our searches to the Wikipedia website:

    public boolean shouldVisit(Page referringPage, WebURL url) { 
        String href = url.getURL().toLowerCase(); 
        if (IMAGE_EXTENSIONS.matcher(href).matches()) { 
            return false; 
        } 
        return href.startsWith("https://en.wikipedia.org/wiki/"); 
    }

The visit method is passed a Page object representing the page being visited. In this implementation, only those pages containing the string shipping route will be processed. This further restricts the pages visited. When we find such a page, its URL, Text, and Text length are displayed:

    public void visit(Page page) { 
        String url = page.getWebURL().getURL(); 
 
        if (page.getParseData() instanceof HtmlParseData) { 
            HtmlParseData htmlParseData =  
                (HtmlParseData) page.getParseData(); 
            String text = htmlParseData.getText(); 
            if (text.contains("shipping route")) { 
                out.println("\nURL: " + url); 
                out.println("Text: " + text); 
                out.println("Text length: " + text.length()); 
            } 
        } 
    } 

The following is the truncated output of the program when executed:

URL: https://en.wikipedia.org/wiki/Bishop_Rock,_Isles_of_Scilly
Text: Bishop Rock, Isles of Scilly...From Wikipedia, the free encyclopedia ... Jump to: ... navigation, search For the Bishop Rock in the Pacific Ocean, see Cortes Bank. Bishop Rock Bishop Rock Lighthouse (2005)
...
Text length: 14677

Notice that only one page was returned. This web crawler was able to identify and ignore previous versions of the main web page.

We could perform further processing, but this example provides some insight into how the API works. Significant amounts of information can be obtained when visiting a page. In the example, we only used the URL and the length of the text. The following is a sample of other data that you may be interested in obtaining:

OAuth is an open standard used to authenticate users to many different websites. A resource owner effectively delegates access to a server resource without having to share their credentials. It works over HTTPS. OAuth 2.0 succeeded OAuth and is not backwards compatible. It provides client developers a simple way of providing authentication. Several companies use OAuth 2.0 including PayPal, Comcast, and Blizzard Entertainment.

A list of OAuth 2.0 providers is found at https://en.wikipedia.org/wiki/List_of_OAuth_providers. We will use several of these in our discussions.

The sheer volume of data and the popularity of the site, among celebrities and the general public alike, make Twitter a valuable resource for mining social media data. Twitter is a popular social media platform allowing users to read and post short messages called tweets. Twitter provides API support for posting and pulling tweets, including streaming data from all public users. While there are services available for pulling the entire set of public tweet data, we are going to examine other options that, while limiting in the amount of data retrieved at one time, are available at no cost.

We are going to focus on the Twitter API for retrieving streaming data. There are other options for retrieving tweets from a specific user as well as posting data to a specific account but we will not be addressing those in this chapter. The public stream API, at the default access level, allows the user to pull a sample of public tweets currently streaming on Twitter. It is possible to refine the data by specifying parameters to track keywords, specific users, and location.

We are going to use HBC, a Java HTTP client, for this example. You can download a sample HBC application at https://github.com/twitter/hbc. If you prefer to use a different HTTP client, ensure it will return incremental response data. The Apache HTTP client is one option. Before you can create the HTTP connection, you must first create a Twitter account and an application within that account. To get started with the app, visit apps.twitter.com. Once your app is created, you will be assigned a consumer key, consumer secret, access token, and access secret token. We will also use OAuth, as discussed previously in this chapter.

First, we will write a method to perform the authentication and request data from Twitter. The parameters for our method are the authentication information given to us by Twitter when we created our app. We will create a BlockingQueue object to hold our streaming data. For this example, we will set a default capacity of 10,000. We will also specify our endpoint and turn off stall warnings:

    public static void streamTwitter( 
        String consumerKey, String consumerSecret,  
        String accessToken, String accessSecret)  
            throws InterruptedException { 
 
        BlockingQueue<String> statusQueue =  
            new LinkedBlockingQueue<String>(10000); 
        StatusesSampleEndpoint ending =  
            new StatusesSampleEndpoint(); 
        ending.stallWarnings(false); 
        ... 
    } 

Next, we create an Authentication object using OAuth1, a variation of the OAuth class. We can then build our connection client and complete the HTTP connection:

    Authentication twitterAuth = new OAuth1(consumerKey,  
        consumerSecret, accessToken, accessSecret); 
    BasicClient twitterClient = new ClientBuilder() 
            .name("Twitter client") 
            .hosts(Constants.STREAM_HOST) 
            .endpoint(ending) 
            .authentication(twitterAuth) 
            .processor(new StringDelimitedProcessor(statusQueue)) 
            .build(); 
    twitterClient.connect(); 

For the purposes of this example, we will simply read the messages received from the stream and print them to the screen. The messages are returned in JSON format and the decision of how to process them in a real application will depend upon the purpose and limitations of that application:

    for (int msgRead = 0; msgRead < 1000; msgRead++) { 
      if (twitterClient.isDone()) { 
        out.println(twitterClient.getExitEvent().getMessage()); 
        break; 
      } 
 
      String msg = statusQueue.poll(10, TimeUnit.SECONDS); 
      if (msg == null) { 
        out.println("Waited 10 seconds - no message received"); 
      } else { 
        out.println(msg); 
      } 
    } 
    twitterClient.stop(); 

To execute our method, we simply pass our authentication information to the streamTwitter method. For security purposes, we have replaced our personal keys here. Authentication information should always be protected:

    public static void main(String[] args) { 
   
      try { 
        SampleStreamExample.streamTwitter( 
            myKey, mySecret, myToken, myAccess);  
      } catch (InterruptedException e) { 
        out.println(e); 
      } 
    } 

Here is truncated sample data retrieved using the methods listed above. Your data will vary based upon Twitter's live stream, but it should resemble this example:

{"created_at":"Fri May 20 15:47:21 +0000 2016","id":733685552789098496,"id_str":"733685552789098496","text":"bwisit si em bahala sya","source":"\u003ca href="http:\/\/twitter.com" rel="nofollow"\u003eTwitter Web 
...
ntions":[],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"tl","timestamp_ms":"1463759241660"}

Twitter also provides support for pulling all data for one specific user account, as well as posting data directly to an account. A REST API is also available and provides support for specific queries via the search API. These also use the OAuth standard and return data in JSON files.

Wikipedia (https://www.wikipedia.org/) is a useful source of text and image type information. It is an Internet encyclopedia that hosts 38 million articles written in over 250 languages (https://en.wikipedia.org/wiki/Wikipedia). As such, it is useful to know how to programmatically access its contents.

MediaWiki is an open source wiki application that supports wiki type sites. It is used to support Wikipedia and many other sites. The MediaWiki API (http://www.mediawiki.org/wiki/API) provides access to a wiki's data and metadata over HTTP. An application, using this API, can log in, read data, and post changes to a site.

There are several Java APIs that support programmatic access to a wiki site as listed at https://www.mediawiki.org/wiki/API:Client_code#Java. To demonstrate Java access to a wiki we will use Bliki found at https://bitbucket.org/axelclk/info.bliki.wiki/wiki/Home. It provides good access and is easy to use for most basic operations.

The MediaWiki API is complex and has many features. The intent of this section is to illustrate the basic process of obtaining text from a Wikipedia article using this API. It is not possible to cover the API completely here.

We will use the following classes from the info.bliki.api and info.bliki.wiki.model packages:

Javadocs for Bliki are found at http://www.javadoc.io/doc/info.bliki.wiki/bliki-core/3.1.0.

The following example has been adapted from http://www.integratingstuff.com/2012/04/06/hook-into-wikipedia-using-java-and-the-mediawiki-api/. This example will access the English Wikipedia page for the subject, data science. We start by creating an instance of the User class. The first two arguments of the three-argument constructor are the user ID and password, respectively. In this case, they are empty strings. This combination allows us to read a page without having to set up an account. The third argument is the URL for the MediaWiki API page:

    User user = new User("", "",  
        "http://en.wikipedia.org/w/api.php"); 
    user.login(); 

An account will enable us to modify the document. The queryContent method returns a list of Page objects for the subjects found in a string array. Each string should be the title of a page. In this example, we access a single page:

    String[] titles = {"Data science"}; 
    List<Page> pageList = user.queryContent(titles); 

Each Page object contains the content of a page. There are several methods that will return the contents of the page. For each page, a WikiModel instance is created using the two-argument constructor. The first argument is the image base URL and the second argument is the link base URL. These URLs use Wiki variables called image and title, which will be replaced when creating links:

    for (Page page : pageList) { 
        WikiModel wikiModel = new WikiModel("${image}",  
            "${title}"); 
        ... 
    } 

The render method will take the wiki page and render it to HTML. There is also a method to render the page to a PDF document:

    String htmlText = wikiModel.render(page.toString()); 

The HTML text is then displayed:

    out.println(htmlText); 

A partial listing of the output follows:

<p>PageID: 35458904; NS: 0; Title: Data science; 
Image url: 
Content:
{{distinguish}}
{{Use dmy dates}}
{{Data Visualization}}</p>
<p><b>Data science</b> is an interdisciplinary field about processes and systems to extract <a href="Knowledge" >knowledge</a> 
...

We can also obtain basic information about the article using one of several methods as shown here:

    out.println("Title: " + page.getTitle() + "\n" + 
        "Page ID: " + page.getPageid() + "\n" + 
        "Timestamp: " + page.getCurrentRevision().getTimestamp()); 

It is also possible to obtain a list of references in the article and a list of the headers. Here, a list of the references is displayed:

    List <Reference> referenceList = wikiModel.getReferences(); 
    out.println(referenceList.size()); 
    for(Reference reference : referenceList) { 
        out.println(reference.getRefString()); 
    } 

The following illustrates the process of getting the section headers:

    ITableOfContent toc = wikiModel.getTableOfContent(); 
    List<SectionHeader> sections = toc.getSectionHeaders(); 
    for(SectionHeader sh : sections) { 
        out.println(sh.getFirst()); 
    } 

The entire content of Wikipedia can be downloaded. This process is discussed at https://en.wikipedia.org/wiki/Wikipedia:Database_download.

It may be desirable to set up your own Wikipedia server to handle your request.

Flickr (https://www.flickr.com/) is an online photo management and sharing application. It is a possible source for images and videos. The Flickr Developer Guide (https://www.flickr.com/services/developer/) is a good starting point to learn more about Flickr's API.

One of the first steps to using the Flickr API is to request an API key. This key is used to sign your API requests. The process to obtain a key starts at https://www.flickr.com/services/apps/create/. Both commercial and noncommercial keys are available. When you obtain a key you will also get a "secret." Both of these are required to use the API.

We will illustrate the process of locating and downloading images from Flickr. The process involves:

A FlickrException or IOException may be thrown during this process. There are several APIs that support Flickr access. We will be using Flickr4Java, found at https://github.com/callmeal/Flickr4Java. The Flickr4Java Javadocs is found at http://flickrj.sourceforge.net/api/. We will start with a try block and the apikey and secret declarations:

    try { 
        String apikey = "Your API key"; 
        String secret = "Your secret"; 
 
    } catch (FlickrException | IOException ex) { 
        // Handle exceptions 
    } 

The Flickr instance is created next, where the apikey and secret are supplied as the first two parameters. The last parameter specifies the transfer technique used to access Flickr servers. Currently, the REST transport is supported using the REST class:

    Flickr flickr = new Flickr(apikey, secret, new REST()); 

To search for images, we will use the SearchParameters class. This class supports a number of criteria that will narrow down the number of images returned from a query and includes such criteria as latitude, longitude, media type, and user ID. In the following sequence, the setBBox method specifies the longitude and latitude for the search. The parameters are (in order): minimum longitude, minimum latitude, maximum longitude, and maximum latitude. The setMedia method specifies the type of media. There are three possible arguments — "all", "photos", and "videos":

    SearchParameters searchParameters = new SearchParameters(); 
    searchParameters.setBBox("-180", "-90", "180", "90"); 
    searchParameters.setMedia("photos"); 

The PhotosInterface class possesses a search method that uses the SearchParameters instance to retrieve a list of photos. The getPhotosInterface method returns an instance of the PhotosInterface class, as shown next. The SearchParameters instance is the first parameter. The second parameter determines how many photos are retrieved per page and the third parameter is the offset. A PhotoList class instance is returned:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 
    PhotoList<Photo> list = pi.search(searchParameters, 10, 0); 

The next sequence illustrates the use of several methods to get information about the images retrieved. Each Photo instance is accessed using the get method. The title, image format, public flag, and photo URL are displayed:

    out.println("Image List"); 
    for (int i = 0; i < list.size(); i++) { 
        Photo photo = list.get(i); 
        out.println("Image: " + i + 
            `"\nTitle: " + photo.getTitle() +  
            "\nMedia: " + photo.getOriginalFormat() + 
            "\nPublic: " + photo.isPublicFlag() + 
            "\nUrl: " + photo.getUrl() + 
            "\n"); 
    } 
    out.println(); 

A partial listing is shown here where many of the specific values have been modified to protect the original data:

Image List
Image: 0
Title: XYZ Image
Media: jpg
Public: true
Url: https://flickr.com/photos/7723...@N02/269...
Image: 1
Title: IMG_5555.jpg
Media: jpg
Public: true
Url: https://flickr.com/photos/2665...@N07/264...
Image: 2
Title: DSC05555
Media: jpg
Public: true
Url: https://flickr.com/photos/1179...@N04/264...

The list of images returned by this example will vary since we used a fairly wide search range and images are being added all of the time.

There are two approaches that we can use to download an image. The first uses the image's URL and the second uses a Photo object. The image's URL can be obtained from a number of sources. We use the Photo class getUrl method for this example.

In the following sequence, we obtain an instance of PhotosInterface using its constructor to illustrate an alternate approach:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 

We get the first Photo instance from the previous list and then its getUrl to get the image's URL. The PhotosInterface class's getImage method returns a BufferedImage object representing the image as shown here:

    Photo currentPhoto = list.get(0);  
    BufferedImage bufferedImage =  
        pi.getImage(currentPhoto.getUrl()); 

The image is then saved to a file using the ImageIO class:

    File outputfile = new File("image.jpg"); 
    ImageIO.write(bufferedImage, "jpg", outputfile); 

The getImage method is overloaded. Here, the Photo instance and the size of the image desired are used as arguments to get the BufferedImage instance:

    bufferedImage = pi.getImage(currentPhoto, Size.SMALL); 

The image can be saved to a file using the previous technique.

The Flickr4Java API supports a number of other techniques for working with Flickr images.

YouTube is a popular video site where users can upload and share videos (https://www.youtube.com/). It has been used to share humorous videos, provide instructions on how to do any number of things, and share information among its viewers. It is a useful source of information as it captures the thoughts and ideas of a diverse group of people. This provides an interesting opportunity to analysis and gain insight into human behavior.

YouTube can serve as a useful source of videos and video metadata. A Java API is available to access its contents (https://developers.google.com/youtube/v3/). Detailed documentation of the API is found at https://developers.google.com/youtube/v3/docs/.

In this section, we will demonstrate how to search for videos by keyword and retrieve information of interest. We will also show how to download a video. To use the YouTube API, you will need a Google account, which can be obtained at https://www.google.com/accounts/NewAccount. Next, create an account in the Google Developer's Console (https://console.developers.google.com/). API access is supported using either API keys or OAuth 2.0 credentials. The project creation process and keys are discussed at https://developers.google.com/youtube/registering_an_application#create_project.

    try { 
        YouTube youtube = new YouTube.Builder( 
            Auth.HTTP_TRANSPORT, 
            Auth.JSON_FACTORY, 
            new HttpRequestInitializer() { 
                public void initialize(HttpRequest request)  
                        throws IOException { 
                } 
            }) 
                .setApplicationName("application_name") 
        ... 
    } catch (GoogleJSONResponseException ex) { 
        // Handle exceptions 
    } catch (IOException ex) { 
        // Handle exceptions 
    } 

    String queryTerm = "cats"; 

    YouTube.Search.List search = youtube 
        .search() 
        .list("id,snippet"); 

{ 
  "kind": "youtube#searchResult", 
  "etag": etag, 
  "id": { 
    "kind": string, 
    "videoId": string, 
    "channelId": string, 
    "playlistId": string 
  }, 
  "snippet": { 
    "publishedAt": datetime, 
    "channelId": string, 
    "title": string, 
    "description": string, 
    "thumbnails": { 
      (key): { 
        "url": string, 
        "width": unsigned integer, 
        "height": unsigned integer 
      } 
    }, 
    "channelTitle": string, 
    "liveBroadcastContent": string 
  } 
} 

    String apiKey = "Your API key"; 
    search.setKey(apiKey); 
    search.setQ(queryTerm); 
    search.setType("video"); 

    search.setFields("items(id/kind,id/videoId,snippet/title," +  
        "snippet/description,snippet/thumbnails/default/url)"); 

    search.setMaxResults(10L); 

    SearchListResponse searchResponse = search.execute(); 
    List<SearchResult> searchResultList =  
        searchResponse.getItems(); 

    SearchResult video = searchResultList.iterator().next(); 
    Thumbnail thumbnail = video 
        .getSnippet().getThumbnails().getDefault(); 
 
    out.println("Kind: " + video.getKind()); 
    out.println("Video Id: " + video.getId().getVideoId()); 
    out.println("Title: " + video.getSnippet().getTitle()); 
    out.println("Description: " +  
        video.getSnippet().getDescription()); 
    out.println("Thumbnail: " + thumbnail.getUrl()); 

Kind: null
Video Id: tntO...
Title: Funny Cats ...
Description: Check out the ...
Thumbnail: https://i.ytimg.com/vi/tntO.../default.jpg

    String url = "http://www.youtube.com/watch?v=videoID"; 
    String path = "."; 
    VGet vget = new VGet(new URL(url), new File(path)); 
    vget.download(); 

Here we will illustrate the use of the crawler4j (https://github.com/yasserg/crawler4j) web crawler. We will use an adapted version of the basic crawler found at https://github.com/yasserg/crawler4j/tree/master/src/test/java/edu/uci/ics/crawler4j/examples/basic. We will create two classes: CrawlerController and SampleCrawler. The former class set ups the crawler while the latter contains the logic that controls what pages will be processed.

As with our previous crawler, we will crawl the Wikipedia article dealing with Bishop Rock. The results using this crawler will be smaller as many extraneous pages are ignored.

Let's look at the CrawlerController class first. There are several parameters that are used with the crawler as detailed here:

The basic class is shown here:

public class CrawlerController { 
 
  public static void main(String[] args) throws Exception { 
    int numberOfCrawlers = 2; 
    CrawlConfig config = new CrawlConfig(); 
    String crawlStorageFolder = "data"; 
     
    config.setCrawlStorageFolder(crawlStorageFolder); 
    config.setPolitenessDelay(500); 
    config.setMaxDepthOfCrawling(2); 
    config.setMaxPagesToFetch(20); 
    config.setIncludeBinaryContentInCrawling(false); 
    ... 
  } 
}

Next, the CrawlController class is created and configured. Notice the RobotstxtConfig and RobotstxtServer classes used to handle robot.txt files. These files contain instructions that are intended to be read by a web crawler. They provide direction to help a crawler to do a better job such as specifying which parts of a site should not be crawled. This is useful for auto generated pages:

    PageFetcher pageFetcher = new PageFetcher(config); 
    RobotstxtConfig robotstxtConfig = new RobotstxtConfig(); 
    RobotstxtServer robotstxtServer =  
        new RobotstxtServer(robotstxtConfig, pageFetcher); 
    CrawlController controller =  
        new CrawlController(config, pageFetcher, robotstxtServer); 

The crawler needs to start at one or more pages. The addSeed method adds the starting pages. While we used the method only once here, it can be used as many times as needed:

    controller.addSeed( 
      "https://en.wikipedia.org/wiki/Bishop_Rock,_Isles_of_Scilly"); 

The start method will begin the crawling process:

    controller.start(SampleCrawler.class, numberOfCrawlers); 

The SampleCrawler class contains two methods of interest. The first is the shouldVisit method that determines whether a page will be visited and the visit method that actually handles the page. We start with the class declaration and the declaration of a Java regular expression class Pattern object. It will be one way of determining whether a page will be visited. In this declaration, standard images are specified and will be ignored:

    public class SampleCrawler extends WebCrawler { 
        private static final Pattern IMAGE_EXTENSIONS =  
            Pattern.compile(".*\\.(bmp|gif|jpg|png)$"); 
 
        ... 
    } 

The shouldVisit method is passed a reference to the page where this URL was found along with the URL. If any of the images match, the method returns false and the page is ignored. In addition, the URL must start with https://en.wikipedia.org/wiki/. We added this to restrict our searches to the Wikipedia website:

    public boolean shouldVisit(Page referringPage, WebURL url) { 
        String href = url.getURL().toLowerCase(); 
        if (IMAGE_EXTENSIONS.matcher(href).matches()) { 
            return false; 
        } 
        return href.startsWith("https://en.wikipedia.org/wiki/"); 
    }

The visit method is passed a Page object representing the page being visited. In this implementation, only those pages containing the string shipping route will be processed. This further restricts the pages visited. When we find such a page, its URL, Text, and Text length are displayed:

    public void visit(Page page) { 
        String url = page.getWebURL().getURL(); 
 
        if (page.getParseData() instanceof HtmlParseData) { 
            HtmlParseData htmlParseData =  
                (HtmlParseData) page.getParseData(); 
            String text = htmlParseData.getText(); 
            if (text.contains("shipping route")) { 
                out.println("\nURL: " + url); 
                out.println("Text: " + text); 
                out.println("Text length: " + text.length()); 
            } 
        } 
    } 

The following is the truncated output of the program when executed:

URL: https://en.wikipedia.org/wiki/Bishop_Rock,_Isles_of_Scilly
Text: Bishop Rock, Isles of Scilly...From Wikipedia, the free encyclopedia ... Jump to: ... navigation, search For the Bishop Rock in the Pacific Ocean, see Cortes Bank. Bishop Rock Bishop Rock Lighthouse (2005)
...
Text length: 14677

Notice that only one page was returned. This web crawler was able to identify and ignore previous versions of the main web page.

We could perform further processing, but this example provides some insight into how the API works. Significant amounts of information can be obtained when visiting a page. In the example, we only used the URL and the length of the text. The following is a sample of other data that you may be interested in obtaining:

OAuth is an open standard used to authenticate users to many different websites. A resource owner effectively delegates access to a server resource without having to share their credentials. It works over HTTPS. OAuth 2.0 succeeded OAuth and is not backwards compatible. It provides client developers a simple way of providing authentication. Several companies use OAuth 2.0 including PayPal, Comcast, and Blizzard Entertainment.

A list of OAuth 2.0 providers is found at https://en.wikipedia.org/wiki/List_of_OAuth_providers. We will use several of these in our discussions.

The sheer volume of data and the popularity of the site, among celebrities and the general public alike, make Twitter a valuable resource for mining social media data. Twitter is a popular social media platform allowing users to read and post short messages called tweets. Twitter provides API support for posting and pulling tweets, including streaming data from all public users. While there are services available for pulling the entire set of public tweet data, we are going to examine other options that, while limiting in the amount of data retrieved at one time, are available at no cost.

We are going to focus on the Twitter API for retrieving streaming data. There are other options for retrieving tweets from a specific user as well as posting data to a specific account but we will not be addressing those in this chapter. The public stream API, at the default access level, allows the user to pull a sample of public tweets currently streaming on Twitter. It is possible to refine the data by specifying parameters to track keywords, specific users, and location.

We are going to use HBC, a Java HTTP client, for this example. You can download a sample HBC application at https://github.com/twitter/hbc. If you prefer to use a different HTTP client, ensure it will return incremental response data. The Apache HTTP client is one option. Before you can create the HTTP connection, you must first create a Twitter account and an application within that account. To get started with the app, visit apps.twitter.com. Once your app is created, you will be assigned a consumer key, consumer secret, access token, and access secret token. We will also use OAuth, as discussed previously in this chapter.

First, we will write a method to perform the authentication and request data from Twitter. The parameters for our method are the authentication information given to us by Twitter when we created our app. We will create a BlockingQueue object to hold our streaming data. For this example, we will set a default capacity of 10,000. We will also specify our endpoint and turn off stall warnings:

    public static void streamTwitter( 
        String consumerKey, String consumerSecret,  
        String accessToken, String accessSecret)  
            throws InterruptedException { 
 
        BlockingQueue<String> statusQueue =  
            new LinkedBlockingQueue<String>(10000); 
        StatusesSampleEndpoint ending =  
            new StatusesSampleEndpoint(); 
        ending.stallWarnings(false); 
        ... 
    } 

Next, we create an Authentication object using OAuth1, a variation of the OAuth class. We can then build our connection client and complete the HTTP connection:

    Authentication twitterAuth = new OAuth1(consumerKey,  
        consumerSecret, accessToken, accessSecret); 
    BasicClient twitterClient = new ClientBuilder() 
            .name("Twitter client") 
            .hosts(Constants.STREAM_HOST) 
            .endpoint(ending) 
            .authentication(twitterAuth) 
            .processor(new StringDelimitedProcessor(statusQueue)) 
            .build(); 
    twitterClient.connect(); 

For the purposes of this example, we will simply read the messages received from the stream and print them to the screen. The messages are returned in JSON format and the decision of how to process them in a real application will depend upon the purpose and limitations of that application:

    for (int msgRead = 0; msgRead < 1000; msgRead++) { 
      if (twitterClient.isDone()) { 
        out.println(twitterClient.getExitEvent().getMessage()); 
        break; 
      } 
 
      String msg = statusQueue.poll(10, TimeUnit.SECONDS); 
      if (msg == null) { 
        out.println("Waited 10 seconds - no message received"); 
      } else { 
        out.println(msg); 
      } 
    } 
    twitterClient.stop(); 

To execute our method, we simply pass our authentication information to the streamTwitter method. For security purposes, we have replaced our personal keys here. Authentication information should always be protected:

    public static void main(String[] args) { 
   
      try { 
        SampleStreamExample.streamTwitter( 
            myKey, mySecret, myToken, myAccess);  
      } catch (InterruptedException e) { 
        out.println(e); 
      } 
    } 

Here is truncated sample data retrieved using the methods listed above. Your data will vary based upon Twitter's live stream, but it should resemble this example:

{"created_at":"Fri May 20 15:47:21 +0000 2016","id":733685552789098496,"id_str":"733685552789098496","text":"bwisit si em bahala sya","source":"\u003ca href="http:\/\/twitter.com" rel="nofollow"\u003eTwitter Web 
...
ntions":[],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"tl","timestamp_ms":"1463759241660"}

Twitter also provides support for pulling all data for one specific user account, as well as posting data directly to an account. A REST API is also available and provides support for specific queries via the search API. These also use the OAuth standard and return data in JSON files.

Wikipedia (https://www.wikipedia.org/) is a useful source of text and image type information. It is an Internet encyclopedia that hosts 38 million articles written in over 250 languages (https://en.wikipedia.org/wiki/Wikipedia). As such, it is useful to know how to programmatically access its contents.

MediaWiki is an open source wiki application that supports wiki type sites. It is used to support Wikipedia and many other sites. The MediaWiki API (http://www.mediawiki.org/wiki/API) provides access to a wiki's data and metadata over HTTP. An application, using this API, can log in, read data, and post changes to a site.

There are several Java APIs that support programmatic access to a wiki site as listed at https://www.mediawiki.org/wiki/API:Client_code#Java. To demonstrate Java access to a wiki we will use Bliki found at https://bitbucket.org/axelclk/info.bliki.wiki/wiki/Home. It provides good access and is easy to use for most basic operations.

The MediaWiki API is complex and has many features. The intent of this section is to illustrate the basic process of obtaining text from a Wikipedia article using this API. It is not possible to cover the API completely here.

We will use the following classes from the info.bliki.api and info.bliki.wiki.model packages:

Javadocs for Bliki are found at http://www.javadoc.io/doc/info.bliki.wiki/bliki-core/3.1.0.

The following example has been adapted from http://www.integratingstuff.com/2012/04/06/hook-into-wikipedia-using-java-and-the-mediawiki-api/. This example will access the English Wikipedia page for the subject, data science. We start by creating an instance of the User class. The first two arguments of the three-argument constructor are the user ID and password, respectively. In this case, they are empty strings. This combination allows us to read a page without having to set up an account. The third argument is the URL for the MediaWiki API page:

    User user = new User("", "",  
        "http://en.wikipedia.org/w/api.php"); 
    user.login(); 

An account will enable us to modify the document. The queryContent method returns a list of Page objects for the subjects found in a string array. Each string should be the title of a page. In this example, we access a single page:

    String[] titles = {"Data science"}; 
    List<Page> pageList = user.queryContent(titles); 

Each Page object contains the content of a page. There are several methods that will return the contents of the page. For each page, a WikiModel instance is created using the two-argument constructor. The first argument is the image base URL and the second argument is the link base URL. These URLs use Wiki variables called image and title, which will be replaced when creating links:

    for (Page page : pageList) { 
        WikiModel wikiModel = new WikiModel("${image}",  
            "${title}"); 
        ... 
    } 

The render method will take the wiki page and render it to HTML. There is also a method to render the page to a PDF document:

    String htmlText = wikiModel.render(page.toString()); 

The HTML text is then displayed:

    out.println(htmlText); 

A partial listing of the output follows:

<p>PageID: 35458904; NS: 0; Title: Data science; 
Image url: 
Content:
{{distinguish}}
{{Use dmy dates}}
{{Data Visualization}}</p>
<p><b>Data science</b> is an interdisciplinary field about processes and systems to extract <a href="Knowledge" >knowledge</a> 
...

We can also obtain basic information about the article using one of several methods as shown here:

    out.println("Title: " + page.getTitle() + "\n" + 
        "Page ID: " + page.getPageid() + "\n" + 
        "Timestamp: " + page.getCurrentRevision().getTimestamp()); 

It is also possible to obtain a list of references in the article and a list of the headers. Here, a list of the references is displayed:

    List <Reference> referenceList = wikiModel.getReferences(); 
    out.println(referenceList.size()); 
    for(Reference reference : referenceList) { 
        out.println(reference.getRefString()); 
    } 

The following illustrates the process of getting the section headers:

    ITableOfContent toc = wikiModel.getTableOfContent(); 
    List<SectionHeader> sections = toc.getSectionHeaders(); 
    for(SectionHeader sh : sections) { 
        out.println(sh.getFirst()); 
    } 

The entire content of Wikipedia can be downloaded. This process is discussed at https://en.wikipedia.org/wiki/Wikipedia:Database_download.

It may be desirable to set up your own Wikipedia server to handle your request.

Flickr (https://www.flickr.com/) is an online photo management and sharing application. It is a possible source for images and videos. The Flickr Developer Guide (https://www.flickr.com/services/developer/) is a good starting point to learn more about Flickr's API.

One of the first steps to using the Flickr API is to request an API key. This key is used to sign your API requests. The process to obtain a key starts at https://www.flickr.com/services/apps/create/. Both commercial and noncommercial keys are available. When you obtain a key you will also get a "secret." Both of these are required to use the API.

We will illustrate the process of locating and downloading images from Flickr. The process involves:

A FlickrException or IOException may be thrown during this process. There are several APIs that support Flickr access. We will be using Flickr4Java, found at https://github.com/callmeal/Flickr4Java. The Flickr4Java Javadocs is found at http://flickrj.sourceforge.net/api/. We will start with a try block and the apikey and secret declarations:

    try { 
        String apikey = "Your API key"; 
        String secret = "Your secret"; 
 
    } catch (FlickrException | IOException ex) { 
        // Handle exceptions 
    } 

The Flickr instance is created next, where the apikey and secret are supplied as the first two parameters. The last parameter specifies the transfer technique used to access Flickr servers. Currently, the REST transport is supported using the REST class:

    Flickr flickr = new Flickr(apikey, secret, new REST()); 

To search for images, we will use the SearchParameters class. This class supports a number of criteria that will narrow down the number of images returned from a query and includes such criteria as latitude, longitude, media type, and user ID. In the following sequence, the setBBox method specifies the longitude and latitude for the search. The parameters are (in order): minimum longitude, minimum latitude, maximum longitude, and maximum latitude. The setMedia method specifies the type of media. There are three possible arguments — "all", "photos", and "videos":

    SearchParameters searchParameters = new SearchParameters(); 
    searchParameters.setBBox("-180", "-90", "180", "90"); 
    searchParameters.setMedia("photos"); 

The PhotosInterface class possesses a search method that uses the SearchParameters instance to retrieve a list of photos. The getPhotosInterface method returns an instance of the PhotosInterface class, as shown next. The SearchParameters instance is the first parameter. The second parameter determines how many photos are retrieved per page and the third parameter is the offset. A PhotoList class instance is returned:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 
    PhotoList<Photo> list = pi.search(searchParameters, 10, 0); 

The next sequence illustrates the use of several methods to get information about the images retrieved. Each Photo instance is accessed using the get method. The title, image format, public flag, and photo URL are displayed:

    out.println("Image List"); 
    for (int i = 0; i < list.size(); i++) { 
        Photo photo = list.get(i); 
        out.println("Image: " + i + 
            `"\nTitle: " + photo.getTitle() +  
            "\nMedia: " + photo.getOriginalFormat() + 
            "\nPublic: " + photo.isPublicFlag() + 
            "\nUrl: " + photo.getUrl() + 
            "\n"); 
    } 
    out.println(); 

A partial listing is shown here where many of the specific values have been modified to protect the original data:

Image List
Image: 0
Title: XYZ Image
Media: jpg
Public: true
Url: https://flickr.com/photos/7723...@N02/269...
Image: 1
Title: IMG_5555.jpg
Media: jpg
Public: true
Url: https://flickr.com/photos/2665...@N07/264...
Image: 2
Title: DSC05555
Media: jpg
Public: true
Url: https://flickr.com/photos/1179...@N04/264...

The list of images returned by this example will vary since we used a fairly wide search range and images are being added all of the time.

There are two approaches that we can use to download an image. The first uses the image's URL and the second uses a Photo object. The image's URL can be obtained from a number of sources. We use the Photo class getUrl method for this example.

In the following sequence, we obtain an instance of PhotosInterface using its constructor to illustrate an alternate approach:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 

We get the first Photo instance from the previous list and then its getUrl to get the image's URL. The PhotosInterface class's getImage method returns a BufferedImage object representing the image as shown here:

    Photo currentPhoto = list.get(0);  
    BufferedImage bufferedImage =  
        pi.getImage(currentPhoto.getUrl()); 

The image is then saved to a file using the ImageIO class:

    File outputfile = new File("image.jpg"); 
    ImageIO.write(bufferedImage, "jpg", outputfile); 

The getImage method is overloaded. Here, the Photo instance and the size of the image desired are used as arguments to get the BufferedImage instance:

    bufferedImage = pi.getImage(currentPhoto, Size.SMALL); 

The image can be saved to a file using the previous technique.

The Flickr4Java API supports a number of other techniques for working with Flickr images.

YouTube is a popular video site where users can upload and share videos (https://www.youtube.com/). It has been used to share humorous videos, provide instructions on how to do any number of things, and share information among its viewers. It is a useful source of information as it captures the thoughts and ideas of a diverse group of people. This provides an interesting opportunity to analysis and gain insight into human behavior.

YouTube can serve as a useful source of videos and video metadata. A Java API is available to access its contents (https://developers.google.com/youtube/v3/). Detailed documentation of the API is found at https://developers.google.com/youtube/v3/docs/.

In this section, we will demonstrate how to search for videos by keyword and retrieve information of interest. We will also show how to download a video. To use the YouTube API, you will need a Google account, which can be obtained at https://www.google.com/accounts/NewAccount. Next, create an account in the Google Developer's Console (https://console.developers.google.com/). API access is supported using either API keys or OAuth 2.0 credentials. The project creation process and keys are discussed at https://developers.google.com/youtube/registering_an_application#create_project.

    try { 
        YouTube youtube = new YouTube.Builder( 
            Auth.HTTP_TRANSPORT, 
            Auth.JSON_FACTORY, 
            new HttpRequestInitializer() { 
                public void initialize(HttpRequest request)  
                        throws IOException { 
                } 
            }) 
                .setApplicationName("application_name") 
        ... 
    } catch (GoogleJSONResponseException ex) { 
        // Handle exceptions 
    } catch (IOException ex) { 
        // Handle exceptions 
    } 

    String queryTerm = "cats"; 

    YouTube.Search.List search = youtube 
        .search() 
        .list("id,snippet"); 

{ 
  "kind": "youtube#searchResult", 
  "etag": etag, 
  "id": { 
    "kind": string, 
    "videoId": string, 
    "channelId": string, 
    "playlistId": string 
  }, 
  "snippet": { 
    "publishedAt": datetime, 
    "channelId": string, 
    "title": string, 
    "description": string, 
    "thumbnails": { 
      (key): { 
        "url": string, 
        "width": unsigned integer, 
        "height": unsigned integer 
      } 
    }, 
    "channelTitle": string, 
    "liveBroadcastContent": string 
  } 
} 

    String apiKey = "Your API key"; 
    search.setKey(apiKey); 
    search.setQ(queryTerm); 
    search.setType("video"); 

    search.setFields("items(id/kind,id/videoId,snippet/title," +  
        "snippet/description,snippet/thumbnails/default/url)"); 

    search.setMaxResults(10L); 

    SearchListResponse searchResponse = search.execute(); 
    List<SearchResult> searchResultList =  
        searchResponse.getItems(); 

    SearchResult video = searchResultList.iterator().next(); 
    Thumbnail thumbnail = video 
        .getSnippet().getThumbnails().getDefault(); 
 
    out.println("Kind: " + video.getKind()); 
    out.println("Video Id: " + video.getId().getVideoId()); 
    out.println("Title: " + video.getSnippet().getTitle()); 
    out.println("Description: " +  
        video.getSnippet().getDescription()); 
    out.println("Thumbnail: " + thumbnail.getUrl()); 

Kind: null
Video Id: tntO...
Title: Funny Cats ...
Description: Check out the ...
Thumbnail: https://i.ytimg.com/vi/tntO.../default.jpg

    String url = "http://www.youtube.com/watch?v=videoID"; 
    String path = "."; 
    VGet vget = new VGet(new URL(url), new File(path)); 
    vget.download(); 

OAuth is an open standard used to authenticate users to many different websites. A resource owner effectively delegates access to a server resource without having to share their credentials. It works over HTTPS. OAuth 2.0 succeeded OAuth and is not backwards compatible. It provides client developers a simple way of providing authentication. Several companies use OAuth 2.0 including PayPal, Comcast, and Blizzard Entertainment.

A list of OAuth 2.0 providers is found at https://en.wikipedia.org/wiki/List_of_OAuth_providers. We will use several of these in our discussions.

The sheer volume of data and the popularity of the site, among celebrities and the general public alike, make Twitter a valuable resource for mining social media data. Twitter is a popular social media platform allowing users to read and post short messages called tweets. Twitter provides API support for posting and pulling tweets, including streaming data from all public users. While there are services available for pulling the entire set of public tweet data, we are going to examine other options that, while limiting in the amount of data retrieved at one time, are available at no cost.

We are going to focus on the Twitter API for retrieving streaming data. There are other options for retrieving tweets from a specific user as well as posting data to a specific account but we will not be addressing those in this chapter. The public stream API, at the default access level, allows the user to pull a sample of public tweets currently streaming on Twitter. It is possible to refine the data by specifying parameters to track keywords, specific users, and location.

We are going to use HBC, a Java HTTP client, for this example. You can download a sample HBC application at https://github.com/twitter/hbc. If you prefer to use a different HTTP client, ensure it will return incremental response data. The Apache HTTP client is one option. Before you can create the HTTP connection, you must first create a Twitter account and an application within that account. To get started with the app, visit apps.twitter.com. Once your app is created, you will be assigned a consumer key, consumer secret, access token, and access secret token. We will also use OAuth, as discussed previously in this chapter.

First, we will write a method to perform the authentication and request data from Twitter. The parameters for our method are the authentication information given to us by Twitter when we created our app. We will create a BlockingQueue object to hold our streaming data. For this example, we will set a default capacity of 10,000. We will also specify our endpoint and turn off stall warnings:

    public static void streamTwitter( 
        String consumerKey, String consumerSecret,  
        String accessToken, String accessSecret)  
            throws InterruptedException { 
 
        BlockingQueue<String> statusQueue =  
            new LinkedBlockingQueue<String>(10000); 
        StatusesSampleEndpoint ending =  
            new StatusesSampleEndpoint(); 
        ending.stallWarnings(false); 
        ... 
    } 

Next, we create an Authentication object using OAuth1, a variation of the OAuth class. We can then build our connection client and complete the HTTP connection:

    Authentication twitterAuth = new OAuth1(consumerKey,  
        consumerSecret, accessToken, accessSecret); 
    BasicClient twitterClient = new ClientBuilder() 
            .name("Twitter client") 
            .hosts(Constants.STREAM_HOST) 
            .endpoint(ending) 
            .authentication(twitterAuth) 
            .processor(new StringDelimitedProcessor(statusQueue)) 
            .build(); 
    twitterClient.connect(); 

For the purposes of this example, we will simply read the messages received from the stream and print them to the screen. The messages are returned in JSON format and the decision of how to process them in a real application will depend upon the purpose and limitations of that application:

    for (int msgRead = 0; msgRead < 1000; msgRead++) { 
      if (twitterClient.isDone()) { 
        out.println(twitterClient.getExitEvent().getMessage()); 
        break; 
      } 
 
      String msg = statusQueue.poll(10, TimeUnit.SECONDS); 
      if (msg == null) { 
        out.println("Waited 10 seconds - no message received"); 
      } else { 
        out.println(msg); 
      } 
    } 
    twitterClient.stop(); 

To execute our method, we simply pass our authentication information to the streamTwitter method. For security purposes, we have replaced our personal keys here. Authentication information should always be protected:

    public static void main(String[] args) { 
   
      try { 
        SampleStreamExample.streamTwitter( 
            myKey, mySecret, myToken, myAccess);  
      } catch (InterruptedException e) { 
        out.println(e); 
      } 
    } 

Here is truncated sample data retrieved using the methods listed above. Your data will vary based upon Twitter's live stream, but it should resemble this example:

{"created_at":"Fri May 20 15:47:21 +0000 2016","id":733685552789098496,"id_str":"733685552789098496","text":"bwisit si em bahala sya","source":"\u003ca href="http:\/\/twitter.com" rel="nofollow"\u003eTwitter Web 
...
ntions":[],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"tl","timestamp_ms":"1463759241660"}

Twitter also provides support for pulling all data for one specific user account, as well as posting data directly to an account. A REST API is also available and provides support for specific queries via the search API. These also use the OAuth standard and return data in JSON files.

Wikipedia (https://www.wikipedia.org/) is a useful source of text and image type information. It is an Internet encyclopedia that hosts 38 million articles written in over 250 languages (https://en.wikipedia.org/wiki/Wikipedia). As such, it is useful to know how to programmatically access its contents.

MediaWiki is an open source wiki application that supports wiki type sites. It is used to support Wikipedia and many other sites. The MediaWiki API (http://www.mediawiki.org/wiki/API) provides access to a wiki's data and metadata over HTTP. An application, using this API, can log in, read data, and post changes to a site.

There are several Java APIs that support programmatic access to a wiki site as listed at https://www.mediawiki.org/wiki/API:Client_code#Java. To demonstrate Java access to a wiki we will use Bliki found at https://bitbucket.org/axelclk/info.bliki.wiki/wiki/Home. It provides good access and is easy to use for most basic operations.

The MediaWiki API is complex and has many features. The intent of this section is to illustrate the basic process of obtaining text from a Wikipedia article using this API. It is not possible to cover the API completely here.

We will use the following classes from the info.bliki.api and info.bliki.wiki.model packages:

Javadocs for Bliki are found at http://www.javadoc.io/doc/info.bliki.wiki/bliki-core/3.1.0.

The following example has been adapted from http://www.integratingstuff.com/2012/04/06/hook-into-wikipedia-using-java-and-the-mediawiki-api/. This example will access the English Wikipedia page for the subject, data science. We start by creating an instance of the User class. The first two arguments of the three-argument constructor are the user ID and password, respectively. In this case, they are empty strings. This combination allows us to read a page without having to set up an account. The third argument is the URL for the MediaWiki API page:

    User user = new User("", "",  
        "http://en.wikipedia.org/w/api.php"); 
    user.login(); 

An account will enable us to modify the document. The queryContent method returns a list of Page objects for the subjects found in a string array. Each string should be the title of a page. In this example, we access a single page:

    String[] titles = {"Data science"}; 
    List<Page> pageList = user.queryContent(titles); 

Each Page object contains the content of a page. There are several methods that will return the contents of the page. For each page, a WikiModel instance is created using the two-argument constructor. The first argument is the image base URL and the second argument is the link base URL. These URLs use Wiki variables called image and title, which will be replaced when creating links:

    for (Page page : pageList) { 
        WikiModel wikiModel = new WikiModel("${image}",  
            "${title}"); 
        ... 
    } 

The render method will take the wiki page and render it to HTML. There is also a method to render the page to a PDF document:

    String htmlText = wikiModel.render(page.toString()); 

The HTML text is then displayed:

    out.println(htmlText); 

A partial listing of the output follows:

<p>PageID: 35458904; NS: 0; Title: Data science; 
Image url: 
Content:
{{distinguish}}
{{Use dmy dates}}
{{Data Visualization}}</p>
<p><b>Data science</b> is an interdisciplinary field about processes and systems to extract <a href="Knowledge" >knowledge</a> 
...

We can also obtain basic information about the article using one of several methods as shown here:

    out.println("Title: " + page.getTitle() + "\n" + 
        "Page ID: " + page.getPageid() + "\n" + 
        "Timestamp: " + page.getCurrentRevision().getTimestamp()); 

It is also possible to obtain a list of references in the article and a list of the headers. Here, a list of the references is displayed:

    List <Reference> referenceList = wikiModel.getReferences(); 
    out.println(referenceList.size()); 
    for(Reference reference : referenceList) { 
        out.println(reference.getRefString()); 
    } 

The following illustrates the process of getting the section headers:

    ITableOfContent toc = wikiModel.getTableOfContent(); 
    List<SectionHeader> sections = toc.getSectionHeaders(); 
    for(SectionHeader sh : sections) { 
        out.println(sh.getFirst()); 
    } 

The entire content of Wikipedia can be downloaded. This process is discussed at https://en.wikipedia.org/wiki/Wikipedia:Database_download.

It may be desirable to set up your own Wikipedia server to handle your request.

Flickr (https://www.flickr.com/) is an online photo management and sharing application. It is a possible source for images and videos. The Flickr Developer Guide (https://www.flickr.com/services/developer/) is a good starting point to learn more about Flickr's API.

One of the first steps to using the Flickr API is to request an API key. This key is used to sign your API requests. The process to obtain a key starts at https://www.flickr.com/services/apps/create/. Both commercial and noncommercial keys are available. When you obtain a key you will also get a "secret." Both of these are required to use the API.

We will illustrate the process of locating and downloading images from Flickr. The process involves:

A FlickrException or IOException may be thrown during this process. There are several APIs that support Flickr access. We will be using Flickr4Java, found at https://github.com/callmeal/Flickr4Java. The Flickr4Java Javadocs is found at http://flickrj.sourceforge.net/api/. We will start with a try block and the apikey and secret declarations:

    try { 
        String apikey = "Your API key"; 
        String secret = "Your secret"; 
 
    } catch (FlickrException | IOException ex) { 
        // Handle exceptions 
    } 

The Flickr instance is created next, where the apikey and secret are supplied as the first two parameters. The last parameter specifies the transfer technique used to access Flickr servers. Currently, the REST transport is supported using the REST class:

    Flickr flickr = new Flickr(apikey, secret, new REST()); 

To search for images, we will use the SearchParameters class. This class supports a number of criteria that will narrow down the number of images returned from a query and includes such criteria as latitude, longitude, media type, and user ID. In the following sequence, the setBBox method specifies the longitude and latitude for the search. The parameters are (in order): minimum longitude, minimum latitude, maximum longitude, and maximum latitude. The setMedia method specifies the type of media. There are three possible arguments — "all", "photos", and "videos":

    SearchParameters searchParameters = new SearchParameters(); 
    searchParameters.setBBox("-180", "-90", "180", "90"); 
    searchParameters.setMedia("photos"); 

The PhotosInterface class possesses a search method that uses the SearchParameters instance to retrieve a list of photos. The getPhotosInterface method returns an instance of the PhotosInterface class, as shown next. The SearchParameters instance is the first parameter. The second parameter determines how many photos are retrieved per page and the third parameter is the offset. A PhotoList class instance is returned:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 
    PhotoList<Photo> list = pi.search(searchParameters, 10, 0); 

The next sequence illustrates the use of several methods to get information about the images retrieved. Each Photo instance is accessed using the get method. The title, image format, public flag, and photo URL are displayed:

    out.println("Image List"); 
    for (int i = 0; i < list.size(); i++) { 
        Photo photo = list.get(i); 
        out.println("Image: " + i + 
            `"\nTitle: " + photo.getTitle() +  
            "\nMedia: " + photo.getOriginalFormat() + 
            "\nPublic: " + photo.isPublicFlag() + 
            "\nUrl: " + photo.getUrl() + 
            "\n"); 
    } 
    out.println(); 

A partial listing is shown here where many of the specific values have been modified to protect the original data:

Image List
Image: 0
Title: XYZ Image
Media: jpg
Public: true
Url: https://flickr.com/photos/7723...@N02/269...
Image: 1
Title: IMG_5555.jpg
Media: jpg
Public: true
Url: https://flickr.com/photos/2665...@N07/264...
Image: 2
Title: DSC05555
Media: jpg
Public: true
Url: https://flickr.com/photos/1179...@N04/264...

The list of images returned by this example will vary since we used a fairly wide search range and images are being added all of the time.

There are two approaches that we can use to download an image. The first uses the image's URL and the second uses a Photo object. The image's URL can be obtained from a number of sources. We use the Photo class getUrl method for this example.

In the following sequence, we obtain an instance of PhotosInterface using its constructor to illustrate an alternate approach:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 

We get the first Photo instance from the previous list and then its getUrl to get the image's URL. The PhotosInterface class's getImage method returns a BufferedImage object representing the image as shown here:

    Photo currentPhoto = list.get(0);  
    BufferedImage bufferedImage =  
        pi.getImage(currentPhoto.getUrl()); 

The image is then saved to a file using the ImageIO class:

    File outputfile = new File("image.jpg"); 
    ImageIO.write(bufferedImage, "jpg", outputfile); 

The getImage method is overloaded. Here, the Photo instance and the size of the image desired are used as arguments to get the BufferedImage instance:

    bufferedImage = pi.getImage(currentPhoto, Size.SMALL); 

The image can be saved to a file using the previous technique.

The Flickr4Java API supports a number of other techniques for working with Flickr images.

YouTube is a popular video site where users can upload and share videos (https://www.youtube.com/). It has been used to share humorous videos, provide instructions on how to do any number of things, and share information among its viewers. It is a useful source of information as it captures the thoughts and ideas of a diverse group of people. This provides an interesting opportunity to analysis and gain insight into human behavior.

YouTube can serve as a useful source of videos and video metadata. A Java API is available to access its contents (https://developers.google.com/youtube/v3/). Detailed documentation of the API is found at https://developers.google.com/youtube/v3/docs/.

In this section, we will demonstrate how to search for videos by keyword and retrieve information of interest. We will also show how to download a video. To use the YouTube API, you will need a Google account, which can be obtained at https://www.google.com/accounts/NewAccount. Next, create an account in the Google Developer's Console (https://console.developers.google.com/). API access is supported using either API keys or OAuth 2.0 credentials. The project creation process and keys are discussed at https://developers.google.com/youtube/registering_an_application#create_project.

    try { 
        YouTube youtube = new YouTube.Builder( 
            Auth.HTTP_TRANSPORT, 
            Auth.JSON_FACTORY, 
            new HttpRequestInitializer() { 
                public void initialize(HttpRequest request)  
                        throws IOException { 
                } 
            }) 
                .setApplicationName("application_name") 
        ... 
    } catch (GoogleJSONResponseException ex) { 
        // Handle exceptions 
    } catch (IOException ex) { 
        // Handle exceptions 
    } 

    String queryTerm = "cats"; 

    YouTube.Search.List search = youtube 
        .search() 
        .list("id,snippet"); 

{ 
  "kind": "youtube#searchResult", 
  "etag": etag, 
  "id": { 
    "kind": string, 
    "videoId": string, 
    "channelId": string, 
    "playlistId": string 
  }, 
  "snippet": { 
    "publishedAt": datetime, 
    "channelId": string, 
    "title": string, 
    "description": string, 
    "thumbnails": { 
      (key): { 
        "url": string, 
        "width": unsigned integer, 
        "height": unsigned integer 
      } 
    }, 
    "channelTitle": string, 
    "liveBroadcastContent": string 
  } 
} 

    String apiKey = "Your API key"; 
    search.setKey(apiKey); 
    search.setQ(queryTerm); 
    search.setType("video"); 

    search.setFields("items(id/kind,id/videoId,snippet/title," +  
        "snippet/description,snippet/thumbnails/default/url)"); 

    search.setMaxResults(10L); 

    SearchListResponse searchResponse = search.execute(); 
    List<SearchResult> searchResultList =  
        searchResponse.getItems(); 

    SearchResult video = searchResultList.iterator().next(); 
    Thumbnail thumbnail = video 
        .getSnippet().getThumbnails().getDefault(); 
 
    out.println("Kind: " + video.getKind()); 
    out.println("Video Id: " + video.getId().getVideoId()); 
    out.println("Title: " + video.getSnippet().getTitle()); 
    out.println("Description: " +  
        video.getSnippet().getDescription()); 
    out.println("Thumbnail: " + thumbnail.getUrl()); 

Kind: null
Video Id: tntO...
Title: Funny Cats ...
Description: Check out the ...
Thumbnail: https://i.ytimg.com/vi/tntO.../default.jpg

    String url = "http://www.youtube.com/watch?v=videoID"; 
    String path = "."; 
    VGet vget = new VGet(new URL(url), new File(path)); 
    vget.download(); 

OAuth is an open standard used to authenticate users to many different websites. A resource owner effectively delegates access to a server resource without having to share their credentials. It works over HTTPS. OAuth 2.0 succeeded OAuth and is not backwards compatible. It provides client developers a simple way of providing authentication. Several companies use OAuth 2.0 including PayPal, Comcast, and Blizzard Entertainment.

A list of OAuth 2.0 providers is found at https://en.wikipedia.org/wiki/List_of_OAuth_providers. We will use several of these in our discussions.

The sheer volume of data and the popularity of the site, among celebrities and the general public alike, make Twitter a valuable resource for mining social media data. Twitter is a popular social media platform allowing users to read and post short messages called tweets. Twitter provides API support for posting and pulling tweets, including streaming data from all public users. While there are services available for pulling the entire set of public tweet data, we are going to examine other options that, while limiting in the amount of data retrieved at one time, are available at no cost.

We are going to focus on the Twitter API for retrieving streaming data. There are other options for retrieving tweets from a specific user as well as posting data to a specific account but we will not be addressing those in this chapter. The public stream API, at the default access level, allows the user to pull a sample of public tweets currently streaming on Twitter. It is possible to refine the data by specifying parameters to track keywords, specific users, and location.

We are going to use HBC, a Java HTTP client, for this example. You can download a sample HBC application at https://github.com/twitter/hbc. If you prefer to use a different HTTP client, ensure it will return incremental response data. The Apache HTTP client is one option. Before you can create the HTTP connection, you must first create a Twitter account and an application within that account. To get started with the app, visit apps.twitter.com. Once your app is created, you will be assigned a consumer key, consumer secret, access token, and access secret token. We will also use OAuth, as discussed previously in this chapter.

First, we will write a method to perform the authentication and request data from Twitter. The parameters for our method are the authentication information given to us by Twitter when we created our app. We will create a BlockingQueue object to hold our streaming data. For this example, we will set a default capacity of 10,000. We will also specify our endpoint and turn off stall warnings:

    public static void streamTwitter( 
        String consumerKey, String consumerSecret,  
        String accessToken, String accessSecret)  
            throws InterruptedException { 
 
        BlockingQueue<String> statusQueue =  
            new LinkedBlockingQueue<String>(10000); 
        StatusesSampleEndpoint ending =  
            new StatusesSampleEndpoint(); 
        ending.stallWarnings(false); 
        ... 
    } 

Next, we create an Authentication object using OAuth1, a variation of the OAuth class. We can then build our connection client and complete the HTTP connection:

    Authentication twitterAuth = new OAuth1(consumerKey,  
        consumerSecret, accessToken, accessSecret); 
    BasicClient twitterClient = new ClientBuilder() 
            .name("Twitter client") 
            .hosts(Constants.STREAM_HOST) 
            .endpoint(ending) 
            .authentication(twitterAuth) 
            .processor(new StringDelimitedProcessor(statusQueue)) 
            .build(); 
    twitterClient.connect(); 

For the purposes of this example, we will simply read the messages received from the stream and print them to the screen. The messages are returned in JSON format and the decision of how to process them in a real application will depend upon the purpose and limitations of that application:

    for (int msgRead = 0; msgRead < 1000; msgRead++) { 
      if (twitterClient.isDone()) { 
        out.println(twitterClient.getExitEvent().getMessage()); 
        break; 
      } 
 
      String msg = statusQueue.poll(10, TimeUnit.SECONDS); 
      if (msg == null) { 
        out.println("Waited 10 seconds - no message received"); 
      } else { 
        out.println(msg); 
      } 
    } 
    twitterClient.stop(); 

To execute our method, we simply pass our authentication information to the streamTwitter method. For security purposes, we have replaced our personal keys here. Authentication information should always be protected:

    public static void main(String[] args) { 
   
      try { 
        SampleStreamExample.streamTwitter( 
            myKey, mySecret, myToken, myAccess);  
      } catch (InterruptedException e) { 
        out.println(e); 
      } 
    } 

Here is truncated sample data retrieved using the methods listed above. Your data will vary based upon Twitter's live stream, but it should resemble this example:

{"created_at":"Fri May 20 15:47:21 +0000 2016","id":733685552789098496,"id_str":"733685552789098496","text":"bwisit si em bahala sya","source":"\u003ca href="http:\/\/twitter.com" rel="nofollow"\u003eTwitter Web 
...
ntions":[],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"tl","timestamp_ms":"1463759241660"}

Twitter also provides support for pulling all data for one specific user account, as well as posting data directly to an account. A REST API is also available and provides support for specific queries via the search API. These also use the OAuth standard and return data in JSON files.

Wikipedia (https://www.wikipedia.org/) is a useful source of text and image type information. It is an Internet encyclopedia that hosts 38 million articles written in over 250 languages (https://en.wikipedia.org/wiki/Wikipedia). As such, it is useful to know how to programmatically access its contents.

MediaWiki is an open source wiki application that supports wiki type sites. It is used to support Wikipedia and many other sites. The MediaWiki API (http://www.mediawiki.org/wiki/API) provides access to a wiki's data and metadata over HTTP. An application, using this API, can log in, read data, and post changes to a site.

There are several Java APIs that support programmatic access to a wiki site as listed at https://www.mediawiki.org/wiki/API:Client_code#Java. To demonstrate Java access to a wiki we will use Bliki found at https://bitbucket.org/axelclk/info.bliki.wiki/wiki/Home. It provides good access and is easy to use for most basic operations.

The MediaWiki API is complex and has many features. The intent of this section is to illustrate the basic process of obtaining text from a Wikipedia article using this API. It is not possible to cover the API completely here.

We will use the following classes from the info.bliki.api and info.bliki.wiki.model packages:

Javadocs for Bliki are found at http://www.javadoc.io/doc/info.bliki.wiki/bliki-core/3.1.0.

The following example has been adapted from http://www.integratingstuff.com/2012/04/06/hook-into-wikipedia-using-java-and-the-mediawiki-api/. This example will access the English Wikipedia page for the subject, data science. We start by creating an instance of the User class. The first two arguments of the three-argument constructor are the user ID and password, respectively. In this case, they are empty strings. This combination allows us to read a page without having to set up an account. The third argument is the URL for the MediaWiki API page:

    User user = new User("", "",  
        "http://en.wikipedia.org/w/api.php"); 
    user.login(); 

An account will enable us to modify the document. The queryContent method returns a list of Page objects for the subjects found in a string array. Each string should be the title of a page. In this example, we access a single page:

    String[] titles = {"Data science"}; 
    List<Page> pageList = user.queryContent(titles); 

Each Page object contains the content of a page. There are several methods that will return the contents of the page. For each page, a WikiModel instance is created using the two-argument constructor. The first argument is the image base URL and the second argument is the link base URL. These URLs use Wiki variables called image and title, which will be replaced when creating links:

    for (Page page : pageList) { 
        WikiModel wikiModel = new WikiModel("${image}",  
            "${title}"); 
        ... 
    } 

The render method will take the wiki page and render it to HTML. There is also a method to render the page to a PDF document:

    String htmlText = wikiModel.render(page.toString()); 

The HTML text is then displayed:

    out.println(htmlText); 

A partial listing of the output follows:

<p>PageID: 35458904; NS: 0; Title: Data science; 
Image url: 
Content:
{{distinguish}}
{{Use dmy dates}}
{{Data Visualization}}</p>
<p><b>Data science</b> is an interdisciplinary field about processes and systems to extract <a href="Knowledge" >knowledge</a> 
...

We can also obtain basic information about the article using one of several methods as shown here:

    out.println("Title: " + page.getTitle() + "\n" + 
        "Page ID: " + page.getPageid() + "\n" + 
        "Timestamp: " + page.getCurrentRevision().getTimestamp()); 

It is also possible to obtain a list of references in the article and a list of the headers. Here, a list of the references is displayed:

    List <Reference> referenceList = wikiModel.getReferences(); 
    out.println(referenceList.size()); 
    for(Reference reference : referenceList) { 
        out.println(reference.getRefString()); 
    } 

The following illustrates the process of getting the section headers:

    ITableOfContent toc = wikiModel.getTableOfContent(); 
    List<SectionHeader> sections = toc.getSectionHeaders(); 
    for(SectionHeader sh : sections) { 
        out.println(sh.getFirst()); 
    } 

The entire content of Wikipedia can be downloaded. This process is discussed at https://en.wikipedia.org/wiki/Wikipedia:Database_download.

It may be desirable to set up your own Wikipedia server to handle your request.

Flickr (https://www.flickr.com/) is an online photo management and sharing application. It is a possible source for images and videos. The Flickr Developer Guide (https://www.flickr.com/services/developer/) is a good starting point to learn more about Flickr's API.

One of the first steps to using the Flickr API is to request an API key. This key is used to sign your API requests. The process to obtain a key starts at https://www.flickr.com/services/apps/create/. Both commercial and noncommercial keys are available. When you obtain a key you will also get a "secret." Both of these are required to use the API.

We will illustrate the process of locating and downloading images from Flickr. The process involves:

A FlickrException or IOException may be thrown during this process. There are several APIs that support Flickr access. We will be using Flickr4Java, found at https://github.com/callmeal/Flickr4Java. The Flickr4Java Javadocs is found at http://flickrj.sourceforge.net/api/. We will start with a try block and the apikey and secret declarations:

    try { 
        String apikey = "Your API key"; 
        String secret = "Your secret"; 
 
    } catch (FlickrException | IOException ex) { 
        // Handle exceptions 
    } 

The Flickr instance is created next, where the apikey and secret are supplied as the first two parameters. The last parameter specifies the transfer technique used to access Flickr servers. Currently, the REST transport is supported using the REST class:

    Flickr flickr = new Flickr(apikey, secret, new REST()); 

To search for images, we will use the SearchParameters class. This class supports a number of criteria that will narrow down the number of images returned from a query and includes such criteria as latitude, longitude, media type, and user ID. In the following sequence, the setBBox method specifies the longitude and latitude for the search. The parameters are (in order): minimum longitude, minimum latitude, maximum longitude, and maximum latitude. The setMedia method specifies the type of media. There are three possible arguments — "all", "photos", and "videos":

    SearchParameters searchParameters = new SearchParameters(); 
    searchParameters.setBBox("-180", "-90", "180", "90"); 
    searchParameters.setMedia("photos"); 

The PhotosInterface class possesses a search method that uses the SearchParameters instance to retrieve a list of photos. The getPhotosInterface method returns an instance of the PhotosInterface class, as shown next. The SearchParameters instance is the first parameter. The second parameter determines how many photos are retrieved per page and the third parameter is the offset. A PhotoList class instance is returned:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 
    PhotoList<Photo> list = pi.search(searchParameters, 10, 0); 

The next sequence illustrates the use of several methods to get information about the images retrieved. Each Photo instance is accessed using the get method. The title, image format, public flag, and photo URL are displayed:

    out.println("Image List"); 
    for (int i = 0; i < list.size(); i++) { 
        Photo photo = list.get(i); 
        out.println("Image: " + i + 
            `"\nTitle: " + photo.getTitle() +  
            "\nMedia: " + photo.getOriginalFormat() + 
            "\nPublic: " + photo.isPublicFlag() + 
            "\nUrl: " + photo.getUrl() + 
            "\n"); 
    } 
    out.println(); 

A partial listing is shown here where many of the specific values have been modified to protect the original data:

Image List
Image: 0
Title: XYZ Image
Media: jpg
Public: true
Url: https://flickr.com/photos/7723...@N02/269...
Image: 1
Title: IMG_5555.jpg
Media: jpg
Public: true
Url: https://flickr.com/photos/2665...@N07/264...
Image: 2
Title: DSC05555
Media: jpg
Public: true
Url: https://flickr.com/photos/1179...@N04/264...

The list of images returned by this example will vary since we used a fairly wide search range and images are being added all of the time.

There are two approaches that we can use to download an image. The first uses the image's URL and the second uses a Photo object. The image's URL can be obtained from a number of sources. We use the Photo class getUrl method for this example.

In the following sequence, we obtain an instance of PhotosInterface using its constructor to illustrate an alternate approach:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 

We get the first Photo instance from the previous list and then its getUrl to get the image's URL. The PhotosInterface class's getImage method returns a BufferedImage object representing the image as shown here:

    Photo currentPhoto = list.get(0);  
    BufferedImage bufferedImage =  
        pi.getImage(currentPhoto.getUrl()); 

The image is then saved to a file using the ImageIO class:

    File outputfile = new File("image.jpg"); 
    ImageIO.write(bufferedImage, "jpg", outputfile); 

The getImage method is overloaded. Here, the Photo instance and the size of the image desired are used as arguments to get the BufferedImage instance:

    bufferedImage = pi.getImage(currentPhoto, Size.SMALL); 

The image can be saved to a file using the previous technique.

The Flickr4Java API supports a number of other techniques for working with Flickr images.

YouTube is a popular video site where users can upload and share videos (https://www.youtube.com/). It has been used to share humorous videos, provide instructions on how to do any number of things, and share information among its viewers. It is a useful source of information as it captures the thoughts and ideas of a diverse group of people. This provides an interesting opportunity to analysis and gain insight into human behavior.

YouTube can serve as a useful source of videos and video metadata. A Java API is available to access its contents (https://developers.google.com/youtube/v3/). Detailed documentation of the API is found at https://developers.google.com/youtube/v3/docs/.

In this section, we will demonstrate how to search for videos by keyword and retrieve information of interest. We will also show how to download a video. To use the YouTube API, you will need a Google account, which can be obtained at https://www.google.com/accounts/NewAccount. Next, create an account in the Google Developer's Console (https://console.developers.google.com/). API access is supported using either API keys or OAuth 2.0 credentials. The project creation process and keys are discussed at https://developers.google.com/youtube/registering_an_application#create_project.

    try { 
        YouTube youtube = new YouTube.Builder( 
            Auth.HTTP_TRANSPORT, 
            Auth.JSON_FACTORY, 
            new HttpRequestInitializer() { 
                public void initialize(HttpRequest request)  
                        throws IOException { 
                } 
            }) 
                .setApplicationName("application_name") 
        ... 
    } catch (GoogleJSONResponseException ex) { 
        // Handle exceptions 
    } catch (IOException ex) { 
        // Handle exceptions 
    } 

    String queryTerm = "cats"; 

    YouTube.Search.List search = youtube 
        .search() 
        .list("id,snippet"); 

{ 
  "kind": "youtube#searchResult", 
  "etag": etag, 
  "id": { 
    "kind": string, 
    "videoId": string, 
    "channelId": string, 
    "playlistId": string 
  }, 
  "snippet": { 
    "publishedAt": datetime, 
    "channelId": string, 
    "title": string, 
    "description": string, 
    "thumbnails": { 
      (key): { 
        "url": string, 
        "width": unsigned integer, 
        "height": unsigned integer 
      } 
    }, 
    "channelTitle": string, 
    "liveBroadcastContent": string 
  } 
} 

    String apiKey = "Your API key"; 
    search.setKey(apiKey); 
    search.setQ(queryTerm); 
    search.setType("video"); 

    search.setFields("items(id/kind,id/videoId,snippet/title," +  
        "snippet/description,snippet/thumbnails/default/url)"); 

    search.setMaxResults(10L); 

    SearchListResponse searchResponse = search.execute(); 
    List<SearchResult> searchResultList =  
        searchResponse.getItems(); 

    SearchResult video = searchResultList.iterator().next(); 
    Thumbnail thumbnail = video 
        .getSnippet().getThumbnails().getDefault(); 
 
    out.println("Kind: " + video.getKind()); 
    out.println("Video Id: " + video.getId().getVideoId()); 
    out.println("Title: " + video.getSnippet().getTitle()); 
    out.println("Description: " +  
        video.getSnippet().getDescription()); 
    out.println("Thumbnail: " + thumbnail.getUrl()); 

Kind: null
Video Id: tntO...
Title: Funny Cats ...
Description: Check out the ...
Thumbnail: https://i.ytimg.com/vi/tntO.../default.jpg

    String url = "http://www.youtube.com/watch?v=videoID"; 
    String path = "."; 
    VGet vget = new VGet(new URL(url), new File(path)); 
    vget.download(); 

OAuth is an open standard used to authenticate users to many different websites. A resource owner effectively delegates access to a server resource without having to share their credentials. It works over HTTPS. OAuth 2.0 succeeded OAuth and is not backwards compatible. It provides client developers a simple way of providing authentication. Several companies use OAuth 2.0 including PayPal, Comcast, and Blizzard Entertainment.

A list of OAuth 2.0 providers is found at https://en.wikipedia.org/wiki/List_of_OAuth_providers. We will use several of these in our discussions.

The sheer volume of data and the popularity of the site, among celebrities and the general public alike, make Twitter a valuable resource for mining social media data. Twitter is a popular social media platform allowing users to read and post short messages called tweets. Twitter provides API support for posting and pulling tweets, including streaming data from all public users. While there are services available for pulling the entire set of public tweet data, we are going to examine other options that, while limiting in the amount of data retrieved at one time, are available at no cost.

We are going to focus on the Twitter API for retrieving streaming data. There are other options for retrieving tweets from a specific user as well as posting data to a specific account but we will not be addressing those in this chapter. The public stream API, at the default access level, allows the user to pull a sample of public tweets currently streaming on Twitter. It is possible to refine the data by specifying parameters to track keywords, specific users, and location.

We are going to use HBC, a Java HTTP client, for this example. You can download a sample HBC application at https://github.com/twitter/hbc. If you prefer to use a different HTTP client, ensure it will return incremental response data. The Apache HTTP client is one option. Before you can create the HTTP connection, you must first create a Twitter account and an application within that account. To get started with the app, visit apps.twitter.com. Once your app is created, you will be assigned a consumer key, consumer secret, access token, and access secret token. We will also use OAuth, as discussed previously in this chapter.

First, we will write a method to perform the authentication and request data from Twitter. The parameters for our method are the authentication information given to us by Twitter when we created our app. We will create a BlockingQueue object to hold our streaming data. For this example, we will set a default capacity of 10,000. We will also specify our endpoint and turn off stall warnings:

    public static void streamTwitter( 
        String consumerKey, String consumerSecret,  
        String accessToken, String accessSecret)  
            throws InterruptedException { 
 
        BlockingQueue<String> statusQueue =  
            new LinkedBlockingQueue<String>(10000); 
        StatusesSampleEndpoint ending =  
            new StatusesSampleEndpoint(); 
        ending.stallWarnings(false); 
        ... 
    } 

Next, we create an Authentication object using OAuth1, a variation of the OAuth class. We can then build our connection client and complete the HTTP connection:

    Authentication twitterAuth = new OAuth1(consumerKey,  
        consumerSecret, accessToken, accessSecret); 
    BasicClient twitterClient = new ClientBuilder() 
            .name("Twitter client") 
            .hosts(Constants.STREAM_HOST) 
            .endpoint(ending) 
            .authentication(twitterAuth) 
            .processor(new StringDelimitedProcessor(statusQueue)) 
            .build(); 
    twitterClient.connect(); 

For the purposes of this example, we will simply read the messages received from the stream and print them to the screen. The messages are returned in JSON format and the decision of how to process them in a real application will depend upon the purpose and limitations of that application:

    for (int msgRead = 0; msgRead < 1000; msgRead++) { 
      if (twitterClient.isDone()) { 
        out.println(twitterClient.getExitEvent().getMessage()); 
        break; 
      } 
 
      String msg = statusQueue.poll(10, TimeUnit.SECONDS); 
      if (msg == null) { 
        out.println("Waited 10 seconds - no message received"); 
      } else { 
        out.println(msg); 
      } 
    } 
    twitterClient.stop(); 

To execute our method, we simply pass our authentication information to the streamTwitter method. For security purposes, we have replaced our personal keys here. Authentication information should always be protected:

    public static void main(String[] args) { 
   
      try { 
        SampleStreamExample.streamTwitter( 
            myKey, mySecret, myToken, myAccess);  
      } catch (InterruptedException e) { 
        out.println(e); 
      } 
    } 

Here is truncated sample data retrieved using the methods listed above. Your data will vary based upon Twitter's live stream, but it should resemble this example:

{"created_at":"Fri May 20 15:47:21 +0000 2016","id":733685552789098496,"id_str":"733685552789098496","text":"bwisit si em bahala sya","source":"\u003ca href="http:\/\/twitter.com" rel="nofollow"\u003eTwitter Web 
...
ntions":[],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"tl","timestamp_ms":"1463759241660"}

Twitter also provides support for pulling all data for one specific user account, as well as posting data directly to an account. A REST API is also available and provides support for specific queries via the search API. These also use the OAuth standard and return data in JSON files.

Wikipedia (https://www.wikipedia.org/) is a useful source of text and image type information. It is an Internet encyclopedia that hosts 38 million articles written in over 250 languages (https://en.wikipedia.org/wiki/Wikipedia). As such, it is useful to know how to programmatically access its contents.

MediaWiki is an open source wiki application that supports wiki type sites. It is used to support Wikipedia and many other sites. The MediaWiki API (http://www.mediawiki.org/wiki/API) provides access to a wiki's data and metadata over HTTP. An application, using this API, can log in, read data, and post changes to a site.

There are several Java APIs that support programmatic access to a wiki site as listed at https://www.mediawiki.org/wiki/API:Client_code#Java. To demonstrate Java access to a wiki we will use Bliki found at https://bitbucket.org/axelclk/info.bliki.wiki/wiki/Home. It provides good access and is easy to use for most basic operations.

The MediaWiki API is complex and has many features. The intent of this section is to illustrate the basic process of obtaining text from a Wikipedia article using this API. It is not possible to cover the API completely here.

We will use the following classes from the info.bliki.api and info.bliki.wiki.model packages:

Javadocs for Bliki are found at http://www.javadoc.io/doc/info.bliki.wiki/bliki-core/3.1.0.

The following example has been adapted from http://www.integratingstuff.com/2012/04/06/hook-into-wikipedia-using-java-and-the-mediawiki-api/. This example will access the English Wikipedia page for the subject, data science. We start by creating an instance of the User class. The first two arguments of the three-argument constructor are the user ID and password, respectively. In this case, they are empty strings. This combination allows us to read a page without having to set up an account. The third argument is the URL for the MediaWiki API page:

    User user = new User("", "",  
        "http://en.wikipedia.org/w/api.php"); 
    user.login(); 

An account will enable us to modify the document. The queryContent method returns a list of Page objects for the subjects found in a string array. Each string should be the title of a page. In this example, we access a single page:

    String[] titles = {"Data science"}; 
    List<Page> pageList = user.queryContent(titles); 

Each Page object contains the content of a page. There are several methods that will return the contents of the page. For each page, a WikiModel instance is created using the two-argument constructor. The first argument is the image base URL and the second argument is the link base URL. These URLs use Wiki variables called image and title, which will be replaced when creating links:

    for (Page page : pageList) { 
        WikiModel wikiModel = new WikiModel("${image}",  
            "${title}"); 
        ... 
    } 

The render method will take the wiki page and render it to HTML. There is also a method to render the page to a PDF document:

    String htmlText = wikiModel.render(page.toString()); 

The HTML text is then displayed:

    out.println(htmlText); 

A partial listing of the output follows:

<p>PageID: 35458904; NS: 0; Title: Data science; 
Image url: 
Content:
{{distinguish}}
{{Use dmy dates}}
{{Data Visualization}}</p>
<p><b>Data science</b> is an interdisciplinary field about processes and systems to extract <a href="Knowledge" >knowledge</a> 
...

We can also obtain basic information about the article using one of several methods as shown here:

    out.println("Title: " + page.getTitle() + "\n" + 
        "Page ID: " + page.getPageid() + "\n" + 
        "Timestamp: " + page.getCurrentRevision().getTimestamp()); 

It is also possible to obtain a list of references in the article and a list of the headers. Here, a list of the references is displayed:

    List <Reference> referenceList = wikiModel.getReferences(); 
    out.println(referenceList.size()); 
    for(Reference reference : referenceList) { 
        out.println(reference.getRefString()); 
    } 

The following illustrates the process of getting the section headers:

    ITableOfContent toc = wikiModel.getTableOfContent(); 
    List<SectionHeader> sections = toc.getSectionHeaders(); 
    for(SectionHeader sh : sections) { 
        out.println(sh.getFirst()); 
    } 

The entire content of Wikipedia can be downloaded. This process is discussed at https://en.wikipedia.org/wiki/Wikipedia:Database_download.

It may be desirable to set up your own Wikipedia server to handle your request.

Flickr (https://www.flickr.com/) is an online photo management and sharing application. It is a possible source for images and videos. The Flickr Developer Guide (https://www.flickr.com/services/developer/) is a good starting point to learn more about Flickr's API.

One of the first steps to using the Flickr API is to request an API key. This key is used to sign your API requests. The process to obtain a key starts at https://www.flickr.com/services/apps/create/. Both commercial and noncommercial keys are available. When you obtain a key you will also get a "secret." Both of these are required to use the API.

We will illustrate the process of locating and downloading images from Flickr. The process involves:

A FlickrException or IOException may be thrown during this process. There are several APIs that support Flickr access. We will be using Flickr4Java, found at https://github.com/callmeal/Flickr4Java. The Flickr4Java Javadocs is found at http://flickrj.sourceforge.net/api/. We will start with a try block and the apikey and secret declarations:

    try { 
        String apikey = "Your API key"; 
        String secret = "Your secret"; 
 
    } catch (FlickrException | IOException ex) { 
        // Handle exceptions 
    } 

The Flickr instance is created next, where the apikey and secret are supplied as the first two parameters. The last parameter specifies the transfer technique used to access Flickr servers. Currently, the REST transport is supported using the REST class:

    Flickr flickr = new Flickr(apikey, secret, new REST()); 

To search for images, we will use the SearchParameters class. This class supports a number of criteria that will narrow down the number of images returned from a query and includes such criteria as latitude, longitude, media type, and user ID. In the following sequence, the setBBox method specifies the longitude and latitude for the search. The parameters are (in order): minimum longitude, minimum latitude, maximum longitude, and maximum latitude. The setMedia method specifies the type of media. There are three possible arguments — "all", "photos", and "videos":

    SearchParameters searchParameters = new SearchParameters(); 
    searchParameters.setBBox("-180", "-90", "180", "90"); 
    searchParameters.setMedia("photos"); 

The PhotosInterface class possesses a search method that uses the SearchParameters instance to retrieve a list of photos. The getPhotosInterface method returns an instance of the PhotosInterface class, as shown next. The SearchParameters instance is the first parameter. The second parameter determines how many photos are retrieved per page and the third parameter is the offset. A PhotoList class instance is returned:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 
    PhotoList<Photo> list = pi.search(searchParameters, 10, 0); 

The next sequence illustrates the use of several methods to get information about the images retrieved. Each Photo instance is accessed using the get method. The title, image format, public flag, and photo URL are displayed:

    out.println("Image List"); 
    for (int i = 0; i < list.size(); i++) { 
        Photo photo = list.get(i); 
        out.println("Image: " + i + 
            `"\nTitle: " + photo.getTitle() +  
            "\nMedia: " + photo.getOriginalFormat() + 
            "\nPublic: " + photo.isPublicFlag() + 
            "\nUrl: " + photo.getUrl() + 
            "\n"); 
    } 
    out.println(); 

A partial listing is shown here where many of the specific values have been modified to protect the original data:

Image List
Image: 0
Title: XYZ Image
Media: jpg
Public: true
Url: https://flickr.com/photos/7723...@N02/269...
Image: 1
Title: IMG_5555.jpg
Media: jpg
Public: true
Url: https://flickr.com/photos/2665...@N07/264...
Image: 2
Title: DSC05555
Media: jpg
Public: true
Url: https://flickr.com/photos/1179...@N04/264...

The list of images returned by this example will vary since we used a fairly wide search range and images are being added all of the time.

There are two approaches that we can use to download an image. The first uses the image's URL and the second uses a Photo object. The image's URL can be obtained from a number of sources. We use the Photo class getUrl method for this example.

In the following sequence, we obtain an instance of PhotosInterface using its constructor to illustrate an alternate approach:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 

We get the first Photo instance from the previous list and then its getUrl to get the image's URL. The PhotosInterface class's getImage method returns a BufferedImage object representing the image as shown here:

    Photo currentPhoto = list.get(0);  
    BufferedImage bufferedImage =  
        pi.getImage(currentPhoto.getUrl()); 

The image is then saved to a file using the ImageIO class:

    File outputfile = new File("image.jpg"); 
    ImageIO.write(bufferedImage, "jpg", outputfile); 

The getImage method is overloaded. Here, the Photo instance and the size of the image desired are used as arguments to get the BufferedImage instance:

    bufferedImage = pi.getImage(currentPhoto, Size.SMALL); 

The image can be saved to a file using the previous technique.

The Flickr4Java API supports a number of other techniques for working with Flickr images.

YouTube is a popular video site where users can upload and share videos (https://www.youtube.com/). It has been used to share humorous videos, provide instructions on how to do any number of things, and share information among its viewers. It is a useful source of information as it captures the thoughts and ideas of a diverse group of people. This provides an interesting opportunity to analysis and gain insight into human behavior.

YouTube can serve as a useful source of videos and video metadata. A Java API is available to access its contents (https://developers.google.com/youtube/v3/). Detailed documentation of the API is found at https://developers.google.com/youtube/v3/docs/.

In this section, we will demonstrate how to search for videos by keyword and retrieve information of interest. We will also show how to download a video. To use the YouTube API, you will need a Google account, which can be obtained at https://www.google.com/accounts/NewAccount. Next, create an account in the Google Developer's Console (https://console.developers.google.com/). API access is supported using either API keys or OAuth 2.0 credentials. The project creation process and keys are discussed at https://developers.google.com/youtube/registering_an_application#create_project.

    try { 
        YouTube youtube = new YouTube.Builder( 
            Auth.HTTP_TRANSPORT, 
            Auth.JSON_FACTORY, 
            new HttpRequestInitializer() { 
                public void initialize(HttpRequest request)  
                        throws IOException { 
                } 
            }) 
                .setApplicationName("application_name") 
        ... 
    } catch (GoogleJSONResponseException ex) { 
        // Handle exceptions 
    } catch (IOException ex) { 
        // Handle exceptions 
    } 

    String queryTerm = "cats"; 

    YouTube.Search.List search = youtube 
        .search() 
        .list("id,snippet"); 

{ 
  "kind": "youtube#searchResult", 
  "etag": etag, 
  "id": { 
    "kind": string, 
    "videoId": string, 
    "channelId": string, 
    "playlistId": string 
  }, 
  "snippet": { 
    "publishedAt": datetime, 
    "channelId": string, 
    "title": string, 
    "description": string, 
    "thumbnails": { 
      (key): { 
        "url": string, 
        "width": unsigned integer, 
        "height": unsigned integer 
      } 
    }, 
    "channelTitle": string, 
    "liveBroadcastContent": string 
  } 
} 

    String apiKey = "Your API key"; 
    search.setKey(apiKey); 
    search.setQ(queryTerm); 
    search.setType("video"); 

    search.setFields("items(id/kind,id/videoId,snippet/title," +  
        "snippet/description,snippet/thumbnails/default/url)"); 

    search.setMaxResults(10L); 

    SearchListResponse searchResponse = search.execute(); 
    List<SearchResult> searchResultList =  
        searchResponse.getItems(); 

    SearchResult video = searchResultList.iterator().next(); 
    Thumbnail thumbnail = video 
        .getSnippet().getThumbnails().getDefault(); 
 
    out.println("Kind: " + video.getKind()); 
    out.println("Video Id: " + video.getId().getVideoId()); 
    out.println("Title: " + video.getSnippet().getTitle()); 
    out.println("Description: " +  
        video.getSnippet().getDescription()); 
    out.println("Thumbnail: " + thumbnail.getUrl()); 

Kind: null
Video Id: tntO...
Title: Funny Cats ...
Description: Check out the ...
Thumbnail: https://i.ytimg.com/vi/tntO.../default.jpg

    String url = "http://www.youtube.com/watch?v=videoID"; 
    String path = "."; 
    VGet vget = new VGet(new URL(url), new File(path)); 
    vget.download(); 

The sheer volume of data and the popularity of the site, among celebrities and the general public alike, make Twitter a valuable resource for mining social media data. Twitter is a popular social media platform allowing users to read and post short messages called tweets. Twitter provides API support for posting and pulling tweets, including streaming data from all public users. While there are services available for pulling the entire set of public tweet data, we are going to examine other options that, while limiting in the amount of data retrieved at one time, are available at no cost.

We are going to focus on the Twitter API for retrieving streaming data. There are other options for retrieving tweets from a specific user as well as posting data to a specific account but we will not be addressing those in this chapter. The public stream API, at the default access level, allows the user to pull a sample of public tweets currently streaming on Twitter. It is possible to refine the data by specifying parameters to track keywords, specific users, and location.

We are going to use HBC, a Java HTTP client, for this example. You can download a sample HBC application at https://github.com/twitter/hbc. If you prefer to use a different HTTP client, ensure it will return incremental response data. The Apache HTTP client is one option. Before you can create the HTTP connection, you must first create a Twitter account and an application within that account. To get started with the app, visit apps.twitter.com. Once your app is created, you will be assigned a consumer key, consumer secret, access token, and access secret token. We will also use OAuth, as discussed previously in this chapter.

First, we will write a method to perform the authentication and request data from Twitter. The parameters for our method are the authentication information given to us by Twitter when we created our app. We will create a BlockingQueue object to hold our streaming data. For this example, we will set a default capacity of 10,000. We will also specify our endpoint and turn off stall warnings:

    public static void streamTwitter( 
        String consumerKey, String consumerSecret,  
        String accessToken, String accessSecret)  
            throws InterruptedException { 
 
        BlockingQueue<String> statusQueue =  
            new LinkedBlockingQueue<String>(10000); 
        StatusesSampleEndpoint ending =  
            new StatusesSampleEndpoint(); 
        ending.stallWarnings(false); 
        ... 
    } 

Next, we create an Authentication object using OAuth1, a variation of the OAuth class. We can then build our connection client and complete the HTTP connection:

    Authentication twitterAuth = new OAuth1(consumerKey,  
        consumerSecret, accessToken, accessSecret); 
    BasicClient twitterClient = new ClientBuilder() 
            .name("Twitter client") 
            .hosts(Constants.STREAM_HOST) 
            .endpoint(ending) 
            .authentication(twitterAuth) 
            .processor(new StringDelimitedProcessor(statusQueue)) 
            .build(); 
    twitterClient.connect(); 

For the purposes of this example, we will simply read the messages received from the stream and print them to the screen. The messages are returned in JSON format and the decision of how to process them in a real application will depend upon the purpose and limitations of that application:

    for (int msgRead = 0; msgRead < 1000; msgRead++) { 
      if (twitterClient.isDone()) { 
        out.println(twitterClient.getExitEvent().getMessage()); 
        break; 
      } 
 
      String msg = statusQueue.poll(10, TimeUnit.SECONDS); 
      if (msg == null) { 
        out.println("Waited 10 seconds - no message received"); 
      } else { 
        out.println(msg); 
      } 
    } 
    twitterClient.stop(); 

To execute our method, we simply pass our authentication information to the streamTwitter method. For security purposes, we have replaced our personal keys here. Authentication information should always be protected:

    public static void main(String[] args) { 
   
      try { 
        SampleStreamExample.streamTwitter( 
            myKey, mySecret, myToken, myAccess);  
      } catch (InterruptedException e) { 
        out.println(e); 
      } 
    } 

Here is truncated sample data retrieved using the methods listed above. Your data will vary based upon Twitter's live stream, but it should resemble this example:

{"created_at":"Fri May 20 15:47:21 +0000 2016","id":733685552789098496,"id_str":"733685552789098496","text":"bwisit si em bahala sya","source":"\u003ca href="http:\/\/twitter.com" rel="nofollow"\u003eTwitter Web 
...
ntions":[],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"tl","timestamp_ms":"1463759241660"}

Twitter also provides support for pulling all data for one specific user account, as well as posting data directly to an account. A REST API is also available and provides support for specific queries via the search API. These also use the OAuth standard and return data in JSON files.

Wikipedia (https://www.wikipedia.org/) is a useful source of text and image type information. It is an Internet encyclopedia that hosts 38 million articles written in over 250 languages (https://en.wikipedia.org/wiki/Wikipedia). As such, it is useful to know how to programmatically access its contents.

MediaWiki is an open source wiki application that supports wiki type sites. It is used to support Wikipedia and many other sites. The MediaWiki API (http://www.mediawiki.org/wiki/API) provides access to a wiki's data and metadata over HTTP. An application, using this API, can log in, read data, and post changes to a site.

There are several Java APIs that support programmatic access to a wiki site as listed at https://www.mediawiki.org/wiki/API:Client_code#Java. To demonstrate Java access to a wiki we will use Bliki found at https://bitbucket.org/axelclk/info.bliki.wiki/wiki/Home. It provides good access and is easy to use for most basic operations.

The MediaWiki API is complex and has many features. The intent of this section is to illustrate the basic process of obtaining text from a Wikipedia article using this API. It is not possible to cover the API completely here.

We will use the following classes from the info.bliki.api and info.bliki.wiki.model packages:

Javadocs for Bliki are found at http://www.javadoc.io/doc/info.bliki.wiki/bliki-core/3.1.0.

The following example has been adapted from http://www.integratingstuff.com/2012/04/06/hook-into-wikipedia-using-java-and-the-mediawiki-api/. This example will access the English Wikipedia page for the subject, data science. We start by creating an instance of the User class. The first two arguments of the three-argument constructor are the user ID and password, respectively. In this case, they are empty strings. This combination allows us to read a page without having to set up an account. The third argument is the URL for the MediaWiki API page:

    User user = new User("", "",  
        "http://en.wikipedia.org/w/api.php"); 
    user.login(); 

An account will enable us to modify the document. The queryContent method returns a list of Page objects for the subjects found in a string array. Each string should be the title of a page. In this example, we access a single page:

    String[] titles = {"Data science"}; 
    List<Page> pageList = user.queryContent(titles); 

Each Page object contains the content of a page. There are several methods that will return the contents of the page. For each page, a WikiModel instance is created using the two-argument constructor. The first argument is the image base URL and the second argument is the link base URL. These URLs use Wiki variables called image and title, which will be replaced when creating links:

    for (Page page : pageList) { 
        WikiModel wikiModel = new WikiModel("${image}",  
            "${title}"); 
        ... 
    } 

The render method will take the wiki page and render it to HTML. There is also a method to render the page to a PDF document:

    String htmlText = wikiModel.render(page.toString()); 

The HTML text is then displayed:

    out.println(htmlText); 

A partial listing of the output follows:

<p>PageID: 35458904; NS: 0; Title: Data science; 
Image url: 
Content:
{{distinguish}}
{{Use dmy dates}}
{{Data Visualization}}</p>
<p><b>Data science</b> is an interdisciplinary field about processes and systems to extract <a href="Knowledge" >knowledge</a> 
...

We can also obtain basic information about the article using one of several methods as shown here:

    out.println("Title: " + page.getTitle() + "\n" + 
        "Page ID: " + page.getPageid() + "\n" + 
        "Timestamp: " + page.getCurrentRevision().getTimestamp()); 

It is also possible to obtain a list of references in the article and a list of the headers. Here, a list of the references is displayed:

    List <Reference> referenceList = wikiModel.getReferences(); 
    out.println(referenceList.size()); 
    for(Reference reference : referenceList) { 
        out.println(reference.getRefString()); 
    } 

The following illustrates the process of getting the section headers:

    ITableOfContent toc = wikiModel.getTableOfContent(); 
    List<SectionHeader> sections = toc.getSectionHeaders(); 
    for(SectionHeader sh : sections) { 
        out.println(sh.getFirst()); 
    } 

The entire content of Wikipedia can be downloaded. This process is discussed at https://en.wikipedia.org/wiki/Wikipedia:Database_download.

It may be desirable to set up your own Wikipedia server to handle your request.

Flickr (https://www.flickr.com/) is an online photo management and sharing application. It is a possible source for images and videos. The Flickr Developer Guide (https://www.flickr.com/services/developer/) is a good starting point to learn more about Flickr's API.

One of the first steps to using the Flickr API is to request an API key. This key is used to sign your API requests. The process to obtain a key starts at https://www.flickr.com/services/apps/create/. Both commercial and noncommercial keys are available. When you obtain a key you will also get a "secret." Both of these are required to use the API.

We will illustrate the process of locating and downloading images from Flickr. The process involves:

A FlickrException or IOException may be thrown during this process. There are several APIs that support Flickr access. We will be using Flickr4Java, found at https://github.com/callmeal/Flickr4Java. The Flickr4Java Javadocs is found at http://flickrj.sourceforge.net/api/. We will start with a try block and the apikey and secret declarations:

    try { 
        String apikey = "Your API key"; 
        String secret = "Your secret"; 
 
    } catch (FlickrException | IOException ex) { 
        // Handle exceptions 
    } 

The Flickr instance is created next, where the apikey and secret are supplied as the first two parameters. The last parameter specifies the transfer technique used to access Flickr servers. Currently, the REST transport is supported using the REST class:

    Flickr flickr = new Flickr(apikey, secret, new REST()); 

To search for images, we will use the SearchParameters class. This class supports a number of criteria that will narrow down the number of images returned from a query and includes such criteria as latitude, longitude, media type, and user ID. In the following sequence, the setBBox method specifies the longitude and latitude for the search. The parameters are (in order): minimum longitude, minimum latitude, maximum longitude, and maximum latitude. The setMedia method specifies the type of media. There are three possible arguments — "all", "photos", and "videos":

    SearchParameters searchParameters = new SearchParameters(); 
    searchParameters.setBBox("-180", "-90", "180", "90"); 
    searchParameters.setMedia("photos"); 

The PhotosInterface class possesses a search method that uses the SearchParameters instance to retrieve a list of photos. The getPhotosInterface method returns an instance of the PhotosInterface class, as shown next. The SearchParameters instance is the first parameter. The second parameter determines how many photos are retrieved per page and the third parameter is the offset. A PhotoList class instance is returned:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 
    PhotoList<Photo> list = pi.search(searchParameters, 10, 0); 

The next sequence illustrates the use of several methods to get information about the images retrieved. Each Photo instance is accessed using the get method. The title, image format, public flag, and photo URL are displayed:

    out.println("Image List"); 
    for (int i = 0; i < list.size(); i++) { 
        Photo photo = list.get(i); 
        out.println("Image: " + i + 
            `"\nTitle: " + photo.getTitle() +  
            "\nMedia: " + photo.getOriginalFormat() + 
            "\nPublic: " + photo.isPublicFlag() + 
            "\nUrl: " + photo.getUrl() + 
            "\n"); 
    } 
    out.println(); 

A partial listing is shown here where many of the specific values have been modified to protect the original data:

Image List
Image: 0
Title: XYZ Image
Media: jpg
Public: true
Url: https://flickr.com/photos/7723...@N02/269...
Image: 1
Title: IMG_5555.jpg
Media: jpg
Public: true
Url: https://flickr.com/photos/2665...@N07/264...
Image: 2
Title: DSC05555
Media: jpg
Public: true
Url: https://flickr.com/photos/1179...@N04/264...

The list of images returned by this example will vary since we used a fairly wide search range and images are being added all of the time.

There are two approaches that we can use to download an image. The first uses the image's URL and the second uses a Photo object. The image's URL can be obtained from a number of sources. We use the Photo class getUrl method for this example.

In the following sequence, we obtain an instance of PhotosInterface using its constructor to illustrate an alternate approach:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 

We get the first Photo instance from the previous list and then its getUrl to get the image's URL. The PhotosInterface class's getImage method returns a BufferedImage object representing the image as shown here:

    Photo currentPhoto = list.get(0);  
    BufferedImage bufferedImage =  
        pi.getImage(currentPhoto.getUrl()); 

The image is then saved to a file using the ImageIO class:

    File outputfile = new File("image.jpg"); 
    ImageIO.write(bufferedImage, "jpg", outputfile); 

The getImage method is overloaded. Here, the Photo instance and the size of the image desired are used as arguments to get the BufferedImage instance:

    bufferedImage = pi.getImage(currentPhoto, Size.SMALL); 

The image can be saved to a file using the previous technique.

The Flickr4Java API supports a number of other techniques for working with Flickr images.

YouTube is a popular video site where users can upload and share videos (https://www.youtube.com/). It has been used to share humorous videos, provide instructions on how to do any number of things, and share information among its viewers. It is a useful source of information as it captures the thoughts and ideas of a diverse group of people. This provides an interesting opportunity to analysis and gain insight into human behavior.

YouTube can serve as a useful source of videos and video metadata. A Java API is available to access its contents (https://developers.google.com/youtube/v3/). Detailed documentation of the API is found at https://developers.google.com/youtube/v3/docs/.

In this section, we will demonstrate how to search for videos by keyword and retrieve information of interest. We will also show how to download a video. To use the YouTube API, you will need a Google account, which can be obtained at https://www.google.com/accounts/NewAccount. Next, create an account in the Google Developer's Console (https://console.developers.google.com/). API access is supported using either API keys or OAuth 2.0 credentials. The project creation process and keys are discussed at https://developers.google.com/youtube/registering_an_application#create_project.

    try { 
        YouTube youtube = new YouTube.Builder( 
            Auth.HTTP_TRANSPORT, 
            Auth.JSON_FACTORY, 
            new HttpRequestInitializer() { 
                public void initialize(HttpRequest request)  
                        throws IOException { 
                } 
            }) 
                .setApplicationName("application_name") 
        ... 
    } catch (GoogleJSONResponseException ex) { 
        // Handle exceptions 
    } catch (IOException ex) { 
        // Handle exceptions 
    } 

    String queryTerm = "cats"; 

    YouTube.Search.List search = youtube 
        .search() 
        .list("id,snippet"); 

{ 
  "kind": "youtube#searchResult", 
  "etag": etag, 
  "id": { 
    "kind": string, 
    "videoId": string, 
    "channelId": string, 
    "playlistId": string 
  }, 
  "snippet": { 
    "publishedAt": datetime, 
    "channelId": string, 
    "title": string, 
    "description": string, 
    "thumbnails": { 
      (key): { 
        "url": string, 
        "width": unsigned integer, 
        "height": unsigned integer 
      } 
    }, 
    "channelTitle": string, 
    "liveBroadcastContent": string 
  } 
} 

    String apiKey = "Your API key"; 
    search.setKey(apiKey); 
    search.setQ(queryTerm); 
    search.setType("video"); 

    search.setFields("items(id/kind,id/videoId,snippet/title," +  
        "snippet/description,snippet/thumbnails/default/url)"); 

    search.setMaxResults(10L); 

    SearchListResponse searchResponse = search.execute(); 
    List<SearchResult> searchResultList =  
        searchResponse.getItems(); 

    SearchResult video = searchResultList.iterator().next(); 
    Thumbnail thumbnail = video 
        .getSnippet().getThumbnails().getDefault(); 
 
    out.println("Kind: " + video.getKind()); 
    out.println("Video Id: " + video.getId().getVideoId()); 
    out.println("Title: " + video.getSnippet().getTitle()); 
    out.println("Description: " +  
        video.getSnippet().getDescription()); 
    out.println("Thumbnail: " + thumbnail.getUrl()); 

Kind: null
Video Id: tntO...
Title: Funny Cats ...
Description: Check out the ...
Thumbnail: https://i.ytimg.com/vi/tntO.../default.jpg

    String url = "http://www.youtube.com/watch?v=videoID"; 
    String path = "."; 
    VGet vget = new VGet(new URL(url), new File(path)); 
    vget.download(); 

Wikipedia (https://www.wikipedia.org/) is a useful source of text and image type information. It is an Internet encyclopedia that hosts 38 million articles written in over 250 languages (https://en.wikipedia.org/wiki/Wikipedia). As such, it is useful to know how to programmatically access its contents.

MediaWiki is an open source wiki application that supports wiki type sites. It is used to support Wikipedia and many other sites. The MediaWiki API (http://www.mediawiki.org/wiki/API) provides access to a wiki's data and metadata over HTTP. An application, using this API, can log in, read data, and post changes to a site.

There are several Java APIs that support programmatic access to a wiki site as listed at https://www.mediawiki.org/wiki/API:Client_code#Java. To demonstrate Java access to a wiki we will use Bliki found at https://bitbucket.org/axelclk/info.bliki.wiki/wiki/Home. It provides good access and is easy to use for most basic operations.

The MediaWiki API is complex and has many features. The intent of this section is to illustrate the basic process of obtaining text from a Wikipedia article using this API. It is not possible to cover the API completely here.

We will use the following classes from the info.bliki.api and info.bliki.wiki.model packages:

Javadocs for Bliki are found at http://www.javadoc.io/doc/info.bliki.wiki/bliki-core/3.1.0.

The following example has been adapted from http://www.integratingstuff.com/2012/04/06/hook-into-wikipedia-using-java-and-the-mediawiki-api/. This example will access the English Wikipedia page for the subject, data science. We start by creating an instance of the User class. The first two arguments of the three-argument constructor are the user ID and password, respectively. In this case, they are empty strings. This combination allows us to read a page without having to set up an account. The third argument is the URL for the MediaWiki API page:

    User user = new User("", "",  
        "http://en.wikipedia.org/w/api.php"); 
    user.login(); 

An account will enable us to modify the document. The queryContent method returns a list of Page objects for the subjects found in a string array. Each string should be the title of a page. In this example, we access a single page:

    String[] titles = {"Data science"}; 
    List<Page> pageList = user.queryContent(titles); 

Each Page object contains the content of a page. There are several methods that will return the contents of the page. For each page, a WikiModel instance is created using the two-argument constructor. The first argument is the image base URL and the second argument is the link base URL. These URLs use Wiki variables called image and title, which will be replaced when creating links:

    for (Page page : pageList) { 
        WikiModel wikiModel = new WikiModel("${image}",  
            "${title}"); 
        ... 
    } 

The render method will take the wiki page and render it to HTML. There is also a method to render the page to a PDF document:

    String htmlText = wikiModel.render(page.toString()); 

The HTML text is then displayed:

    out.println(htmlText); 

A partial listing of the output follows:

<p>PageID: 35458904; NS: 0; Title: Data science; 
Image url: 
Content:
{{distinguish}}
{{Use dmy dates}}
{{Data Visualization}}</p>
<p><b>Data science</b> is an interdisciplinary field about processes and systems to extract <a href="Knowledge" >knowledge</a> 
...

We can also obtain basic information about the article using one of several methods as shown here:

    out.println("Title: " + page.getTitle() + "\n" + 
        "Page ID: " + page.getPageid() + "\n" + 
        "Timestamp: " + page.getCurrentRevision().getTimestamp()); 

It is also possible to obtain a list of references in the article and a list of the headers. Here, a list of the references is displayed:

    List <Reference> referenceList = wikiModel.getReferences(); 
    out.println(referenceList.size()); 
    for(Reference reference : referenceList) { 
        out.println(reference.getRefString()); 
    } 

The following illustrates the process of getting the section headers:

    ITableOfContent toc = wikiModel.getTableOfContent(); 
    List<SectionHeader> sections = toc.getSectionHeaders(); 
    for(SectionHeader sh : sections) { 
        out.println(sh.getFirst()); 
    } 

The entire content of Wikipedia can be downloaded. This process is discussed at https://en.wikipedia.org/wiki/Wikipedia:Database_download.

It may be desirable to set up your own Wikipedia server to handle your request.

Flickr (https://www.flickr.com/) is an online photo management and sharing application. It is a possible source for images and videos. The Flickr Developer Guide (https://www.flickr.com/services/developer/) is a good starting point to learn more about Flickr's API.

One of the first steps to using the Flickr API is to request an API key. This key is used to sign your API requests. The process to obtain a key starts at https://www.flickr.com/services/apps/create/. Both commercial and noncommercial keys are available. When you obtain a key you will also get a "secret." Both of these are required to use the API.

We will illustrate the process of locating and downloading images from Flickr. The process involves:

A FlickrException or IOException may be thrown during this process. There are several APIs that support Flickr access. We will be using Flickr4Java, found at https://github.com/callmeal/Flickr4Java. The Flickr4Java Javadocs is found at http://flickrj.sourceforge.net/api/. We will start with a try block and the apikey and secret declarations:

    try { 
        String apikey = "Your API key"; 
        String secret = "Your secret"; 
 
    } catch (FlickrException | IOException ex) { 
        // Handle exceptions 
    } 

The Flickr instance is created next, where the apikey and secret are supplied as the first two parameters. The last parameter specifies the transfer technique used to access Flickr servers. Currently, the REST transport is supported using the REST class:

    Flickr flickr = new Flickr(apikey, secret, new REST()); 

To search for images, we will use the SearchParameters class. This class supports a number of criteria that will narrow down the number of images returned from a query and includes such criteria as latitude, longitude, media type, and user ID. In the following sequence, the setBBox method specifies the longitude and latitude for the search. The parameters are (in order): minimum longitude, minimum latitude, maximum longitude, and maximum latitude. The setMedia method specifies the type of media. There are three possible arguments — "all", "photos", and "videos":

    SearchParameters searchParameters = new SearchParameters(); 
    searchParameters.setBBox("-180", "-90", "180", "90"); 
    searchParameters.setMedia("photos"); 

The PhotosInterface class possesses a search method that uses the SearchParameters instance to retrieve a list of photos. The getPhotosInterface method returns an instance of the PhotosInterface class, as shown next. The SearchParameters instance is the first parameter. The second parameter determines how many photos are retrieved per page and the third parameter is the offset. A PhotoList class instance is returned:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 
    PhotoList<Photo> list = pi.search(searchParameters, 10, 0); 

The next sequence illustrates the use of several methods to get information about the images retrieved. Each Photo instance is accessed using the get method. The title, image format, public flag, and photo URL are displayed:

    out.println("Image List"); 
    for (int i = 0; i < list.size(); i++) { 
        Photo photo = list.get(i); 
        out.println("Image: " + i + 
            `"\nTitle: " + photo.getTitle() +  
            "\nMedia: " + photo.getOriginalFormat() + 
            "\nPublic: " + photo.isPublicFlag() + 
            "\nUrl: " + photo.getUrl() + 
            "\n"); 
    } 
    out.println(); 

A partial listing is shown here where many of the specific values have been modified to protect the original data:

Image List
Image: 0
Title: XYZ Image
Media: jpg
Public: true
Url: https://flickr.com/photos/7723...@N02/269...
Image: 1
Title: IMG_5555.jpg
Media: jpg
Public: true
Url: https://flickr.com/photos/2665...@N07/264...
Image: 2
Title: DSC05555
Media: jpg
Public: true
Url: https://flickr.com/photos/1179...@N04/264...

The list of images returned by this example will vary since we used a fairly wide search range and images are being added all of the time.

There are two approaches that we can use to download an image. The first uses the image's URL and the second uses a Photo object. The image's URL can be obtained from a number of sources. We use the Photo class getUrl method for this example.

In the following sequence, we obtain an instance of PhotosInterface using its constructor to illustrate an alternate approach:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 

We get the first Photo instance from the previous list and then its getUrl to get the image's URL. The PhotosInterface class's getImage method returns a BufferedImage object representing the image as shown here:

    Photo currentPhoto = list.get(0);  
    BufferedImage bufferedImage =  
        pi.getImage(currentPhoto.getUrl()); 

The image is then saved to a file using the ImageIO class:

    File outputfile = new File("image.jpg"); 
    ImageIO.write(bufferedImage, "jpg", outputfile); 

The getImage method is overloaded. Here, the Photo instance and the size of the image desired are used as arguments to get the BufferedImage instance:

    bufferedImage = pi.getImage(currentPhoto, Size.SMALL); 

The image can be saved to a file using the previous technique.

The Flickr4Java API supports a number of other techniques for working with Flickr images.

YouTube is a popular video site where users can upload and share videos (https://www.youtube.com/). It has been used to share humorous videos, provide instructions on how to do any number of things, and share information among its viewers. It is a useful source of information as it captures the thoughts and ideas of a diverse group of people. This provides an interesting opportunity to analysis and gain insight into human behavior.

YouTube can serve as a useful source of videos and video metadata. A Java API is available to access its contents (https://developers.google.com/youtube/v3/). Detailed documentation of the API is found at https://developers.google.com/youtube/v3/docs/.

In this section, we will demonstrate how to search for videos by keyword and retrieve information of interest. We will also show how to download a video. To use the YouTube API, you will need a Google account, which can be obtained at https://www.google.com/accounts/NewAccount. Next, create an account in the Google Developer's Console (https://console.developers.google.com/). API access is supported using either API keys or OAuth 2.0 credentials. The project creation process and keys are discussed at https://developers.google.com/youtube/registering_an_application#create_project.

    try { 
        YouTube youtube = new YouTube.Builder( 
            Auth.HTTP_TRANSPORT, 
            Auth.JSON_FACTORY, 
            new HttpRequestInitializer() { 
                public void initialize(HttpRequest request)  
                        throws IOException { 
                } 
            }) 
                .setApplicationName("application_name") 
        ... 
    } catch (GoogleJSONResponseException ex) { 
        // Handle exceptions 
    } catch (IOException ex) { 
        // Handle exceptions 
    } 

    String queryTerm = "cats"; 

    YouTube.Search.List search = youtube 
        .search() 
        .list("id,snippet"); 

{ 
  "kind": "youtube#searchResult", 
  "etag": etag, 
  "id": { 
    "kind": string, 
    "videoId": string, 
    "channelId": string, 
    "playlistId": string 
  }, 
  "snippet": { 
    "publishedAt": datetime, 
    "channelId": string, 
    "title": string, 
    "description": string, 
    "thumbnails": { 
      (key): { 
        "url": string, 
        "width": unsigned integer, 
        "height": unsigned integer 
      } 
    }, 
    "channelTitle": string, 
    "liveBroadcastContent": string 
  } 
} 

    String apiKey = "Your API key"; 
    search.setKey(apiKey); 
    search.setQ(queryTerm); 
    search.setType("video"); 

    search.setFields("items(id/kind,id/videoId,snippet/title," +  
        "snippet/description,snippet/thumbnails/default/url)"); 

    search.setMaxResults(10L); 

    SearchListResponse searchResponse = search.execute(); 
    List<SearchResult> searchResultList =  
        searchResponse.getItems(); 

    SearchResult video = searchResultList.iterator().next(); 
    Thumbnail thumbnail = video 
        .getSnippet().getThumbnails().getDefault(); 
 
    out.println("Kind: " + video.getKind()); 
    out.println("Video Id: " + video.getId().getVideoId()); 
    out.println("Title: " + video.getSnippet().getTitle()); 
    out.println("Description: " +  
        video.getSnippet().getDescription()); 
    out.println("Thumbnail: " + thumbnail.getUrl()); 

Kind: null
Video Id: tntO...
Title: Funny Cats ...
Description: Check out the ...
Thumbnail: https://i.ytimg.com/vi/tntO.../default.jpg

    String url = "http://www.youtube.com/watch?v=videoID"; 
    String path = "."; 
    VGet vget = new VGet(new URL(url), new File(path)); 
    vget.download(); 

Flickr (https://www.flickr.com/) is an online photo management and sharing application. It is a possible source for images and videos. The Flickr Developer Guide (https://www.flickr.com/services/developer/) is a good starting point to learn more about Flickr's API.

One of the first steps to using the Flickr API is to request an API key. This key is used to sign your API requests. The process to obtain a key starts at https://www.flickr.com/services/apps/create/. Both commercial and noncommercial keys are available. When you obtain a key you will also get a "secret." Both of these are required to use the API.

We will illustrate the process of locating and downloading images from Flickr. The process involves:

A FlickrException or IOException may be thrown during this process. There are several APIs that support Flickr access. We will be using Flickr4Java, found at https://github.com/callmeal/Flickr4Java. The Flickr4Java Javadocs is found at http://flickrj.sourceforge.net/api/. We will start with a try block and the apikey and secret declarations:

    try { 
        String apikey = "Your API key"; 
        String secret = "Your secret"; 
 
    } catch (FlickrException | IOException ex) { 
        // Handle exceptions 
    } 

The Flickr instance is created next, where the apikey and secret are supplied as the first two parameters. The last parameter specifies the transfer technique used to access Flickr servers. Currently, the REST transport is supported using the REST class:

    Flickr flickr = new Flickr(apikey, secret, new REST()); 

To search for images, we will use the SearchParameters class. This class supports a number of criteria that will narrow down the number of images returned from a query and includes such criteria as latitude, longitude, media type, and user ID. In the following sequence, the setBBox method specifies the longitude and latitude for the search. The parameters are (in order): minimum longitude, minimum latitude, maximum longitude, and maximum latitude. The setMedia method specifies the type of media. There are three possible arguments — "all", "photos", and "videos":

    SearchParameters searchParameters = new SearchParameters(); 
    searchParameters.setBBox("-180", "-90", "180", "90"); 
    searchParameters.setMedia("photos"); 

The PhotosInterface class possesses a search method that uses the SearchParameters instance to retrieve a list of photos. The getPhotosInterface method returns an instance of the PhotosInterface class, as shown next. The SearchParameters instance is the first parameter. The second parameter determines how many photos are retrieved per page and the third parameter is the offset. A PhotoList class instance is returned:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 
    PhotoList<Photo> list = pi.search(searchParameters, 10, 0); 

The next sequence illustrates the use of several methods to get information about the images retrieved. Each Photo instance is accessed using the get method. The title, image format, public flag, and photo URL are displayed:

    out.println("Image List"); 
    for (int i = 0; i < list.size(); i++) { 
        Photo photo = list.get(i); 
        out.println("Image: " + i + 
            `"\nTitle: " + photo.getTitle() +  
            "\nMedia: " + photo.getOriginalFormat() + 
            "\nPublic: " + photo.isPublicFlag() + 
            "\nUrl: " + photo.getUrl() + 
            "\n"); 
    } 
    out.println(); 

A partial listing is shown here where many of the specific values have been modified to protect the original data:

Image List
Image: 0
Title: XYZ Image
Media: jpg
Public: true
Url: https://flickr.com/photos/7723...@N02/269...
Image: 1
Title: IMG_5555.jpg
Media: jpg
Public: true
Url: https://flickr.com/photos/2665...@N07/264...
Image: 2
Title: DSC05555
Media: jpg
Public: true
Url: https://flickr.com/photos/1179...@N04/264...

The list of images returned by this example will vary since we used a fairly wide search range and images are being added all of the time.

There are two approaches that we can use to download an image. The first uses the image's URL and the second uses a Photo object. The image's URL can be obtained from a number of sources. We use the Photo class getUrl method for this example.

In the following sequence, we obtain an instance of PhotosInterface using its constructor to illustrate an alternate approach:

    PhotosInterface pi = new PhotosInterface(apikey, secret,  
        new REST()); 

We get the first Photo instance from the previous list and then its getUrl to get the image's URL. The PhotosInterface class's getImage method returns a BufferedImage object representing the image as shown here:

    Photo currentPhoto = list.get(0);  
    BufferedImage bufferedImage =  
        pi.getImage(currentPhoto.getUrl()); 

The image is then saved to a file using the ImageIO class:

    File outputfile = new File("image.jpg"); 
    ImageIO.write(bufferedImage, "jpg", outputfile); 

The getImage method is overloaded. Here, the Photo instance and the size of the image desired are used as arguments to get the BufferedImage instance:

    bufferedImage = pi.getImage(currentPhoto, Size.SMALL); 

The image can be saved to a file using the previous technique.

The Flickr4Java API supports a number of other techniques for working with Flickr images.

YouTube is a popular video site where users can upload and share videos (https://www.youtube.com/). It has been used to share humorous videos, provide instructions on how to do any number of things, and share information among its viewers. It is a useful source of information as it captures the thoughts and ideas of a diverse group of people. This provides an interesting opportunity to analysis and gain insight into human behavior.

YouTube can serve as a useful source of videos and video metadata. A Java API is available to access its contents (https://developers.google.com/youtube/v3/). Detailed documentation of the API is found at https://developers.google.com/youtube/v3/docs/.

In this section, we will demonstrate how to search for videos by keyword and retrieve information of interest. We will also show how to download a video. To use the YouTube API, you will need a Google account, which can be obtained at https://www.google.com/accounts/NewAccount. Next, create an account in the Google Developer's Console (https://console.developers.google.com/). API access is supported using either API keys or OAuth 2.0 credentials. The project creation process and keys are discussed at https://developers.google.com/youtube/registering_an_application#create_project.

    try { 
        YouTube youtube = new YouTube.Builder( 
            Auth.HTTP_TRANSPORT, 
            Auth.JSON_FACTORY, 
            new HttpRequestInitializer() { 
                public void initialize(HttpRequest request)  
                        throws IOException { 
                } 
            }) 
                .setApplicationName("application_name") 
        ... 
    } catch (GoogleJSONResponseException ex) { 
        // Handle exceptions 
    } catch (IOException ex) { 
        // Handle exceptions 
    } 

    String queryTerm = "cats"; 

    YouTube.Search.List search = youtube 
        .search() 
        .list("id,snippet"); 

{ 
  "kind": "youtube#searchResult", 
  "etag": etag, 
  "id": { 
    "kind": string, 
    "videoId": string, 
    "channelId": string, 
    "playlistId": string 
  }, 
  "snippet": { 
    "publishedAt": datetime, 
    "channelId": string, 
    "title": string, 
    "description": string, 
    "thumbnails": { 
      (key): { 
        "url": string, 
        "width": unsigned integer, 
        "height": unsigned integer 
      } 
    }, 
    "channelTitle": string, 
    "liveBroadcastContent": string 
  } 
} 

    String apiKey = "Your API key"; 
    search.setKey(apiKey); 
    search.setQ(queryTerm); 
    search.setType("video"); 

    search.setFields("items(id/kind,id/videoId,snippet/title," +  
        "snippet/description,snippet/thumbnails/default/url)"); 

    search.setMaxResults(10L); 

    SearchListResponse searchResponse = search.execute(); 
    List<SearchResult> searchResultList =  
        searchResponse.getItems(); 

    SearchResult video = searchResultList.iterator().next(); 
    Thumbnail thumbnail = video 
        .getSnippet().getThumbnails().getDefault(); 
 
    out.println("Kind: " + video.getKind()); 
    out.println("Video Id: " + video.getId().getVideoId()); 
    out.println("Title: " + video.getSnippet().getTitle()); 
    out.println("Description: " +  
        video.getSnippet().getDescription()); 
    out.println("Thumbnail: " + thumbnail.getUrl()); 

Kind: null
Video Id: tntO...
Title: Funny Cats ...
Description: Check out the ...
Thumbnail: https://i.ytimg.com/vi/tntO.../default.jpg

    String url = "http://www.youtube.com/watch?v=videoID"; 
    String path = "."; 
    VGet vget = new VGet(new URL(url), new File(path)); 
    vget.download(); 

YouTube is a popular video site where users can upload and share videos (https://www.youtube.com/). It has been used to share humorous videos, provide instructions on how to do any number of things, and share information among its viewers. It is a useful source of information as it captures the thoughts and ideas of a diverse group of people. This provides an interesting opportunity to analysis and gain insight into human behavior.

YouTube can serve as a useful source of videos and video metadata. A Java API is available to access its contents (https://developers.google.com/youtube/v3/). Detailed documentation of the API is found at https://developers.google.com/youtube/v3/docs/.

In this section, we will demonstrate how to search for videos by keyword and retrieve information of interest. We will also show how to download a video. To use the YouTube API, you will need a Google account, which can be obtained at https://www.google.com/accounts/NewAccount. Next, create an account in the Google Developer's Console (https://console.developers.google.com/). API access is supported using either API keys or OAuth 2.0 credentials. The project creation process and keys are discussed at https://developers.google.com/youtube/registering_an_application#create_project.

    try { 
        YouTube youtube = new YouTube.Builder( 
            Auth.HTTP_TRANSPORT, 
            Auth.JSON_FACTORY, 
            new HttpRequestInitializer() { 
                public void initialize(HttpRequest request)  
                        throws IOException { 
                } 
            }) 
                .setApplicationName("application_name") 
        ... 
    } catch (GoogleJSONResponseException ex) { 
        // Handle exceptions 
    } catch (IOException ex) { 
        // Handle exceptions 
    } 

    String queryTerm = "cats"; 

    YouTube.Search.List search = youtube 
        .search() 
        .list("id,snippet"); 

{ 
  "kind": "youtube#searchResult", 
  "etag": etag, 
  "id": { 
    "kind": string, 
    "videoId": string, 
    "channelId": string, 
    "playlistId": string 
  }, 
  "snippet": { 
    "publishedAt": datetime, 
    "channelId": string, 
    "title": string, 
    "description": string, 
    "thumbnails": { 
      (key): { 
        "url": string, 
        "width": unsigned integer, 
        "height": unsigned integer 
      } 
    }, 
    "channelTitle": string, 
    "liveBroadcastContent": string 
  } 
} 

    String apiKey = "Your API key"; 
    search.setKey(apiKey); 
    search.setQ(queryTerm); 
    search.setType("video"); 

    search.setFields("items(id/kind,id/videoId,snippet/title," +  
        "snippet/description,snippet/thumbnails/default/url)"); 

    search.setMaxResults(10L); 

    SearchListResponse searchResponse = search.execute(); 
    List<SearchResult> searchResultList =  
        searchResponse.getItems(); 

    SearchResult video = searchResultList.iterator().next(); 
    Thumbnail thumbnail = video 
        .getSnippet().getThumbnails().getDefault(); 
 
    out.println("Kind: " + video.getKind()); 
    out.println("Video Id: " + video.getId().getVideoId()); 
    out.println("Title: " + video.getSnippet().getTitle()); 
    out.println("Description: " +  
        video.getSnippet().getDescription()); 
    out.println("Thumbnail: " + thumbnail.getUrl()); 

Kind: null
Video Id: tntO...
Title: Funny Cats ...
Description: Check out the ...
Thumbnail: https://i.ytimg.com/vi/tntO.../default.jpg

    String url = "http://www.youtube.com/watch?v=videoID"; 
    String path = "."; 
    VGet vget = new VGet(new URL(url), new File(path)); 
    vget.download(); 

StringTokenizer was the first and most basic tokenizer and has been available since Java 1. It is not recommended for use in new development as the String class's split method is considered more efficient. While it does provide a speed advantage for files with narrowly defined and set delimiters, it is less flexible than other tokenizer options. The following is a simple implementation of the StringTokenizer class that splits a string on spaces:

StringTokenizer tokenizer = new StringTokenizer(dirtyText," "); 
while(tokenizer.hasMoreTokens()){ 
  out.print(tokenizer.nextToken() + " "); 
} 

When we set the  dirtyText variable to hold our text from Moby Dick, shown previously, we get the following truncated output:

Call me Ishmael. Some years ago- never mind how long precisely...

StreamTokenizer is another core Java tokenizer. StreamTokenizer grants more information about the tokens retrieved, and allows the user to specify data types to parse, but is considered more difficult to use than StreamTokenizer or the split method. The String class split method is the simplest way to split strings up based on a delimiter, but it does not provide a way to parse the split strings and you can only specify one delimiter for the entire string. For these reasons, it is not a true tokenizer, but it can be useful for data cleaning.

The Scanner class is designed to allow you to parse strings into different data types. We used it previously in the Handling CSV data section and we will address it again in the Removing stop words section.

Apache Commons consists of sets of open source Java classes and methods. These provide reusable code that complements the standard Java APIs. One popular class included in the Commons is StrTokenizer. This class provides more advanced support than the standard StringTokenizer class, specifically more control and flexibility. The following is a simple implementation of the StrTokenizer:

StrTokenizer tokenizer = new StrTokenizer(text); 
while (tokenizer.hasNext()) { 
  out.print(tokenizer.next() + " "); 
} 

This operates in a similar fashion to StringTokenizer and by default parses tokens on spaces. The constructor can specify the delimiter as well as how to handle double quotes contained in data.

When we use the string from Moby Dick, shown previously, the first tokenizer implementation produces the following truncated output:

Call me Ishmael. Some years ago- never mind how long precisely - having little or no money in my purse...

We can modify our constructor as follows:

StrTokenizer tokenizer = new StrTokenizer(text,","); 

The output for this implementation is:

Call me Ishmael. Some years ago- never mind how long precisely - having little or no money in my purse
and nothing particular to interest me on shore
I thought I would sail about a little and see the watery part of the world.

Notice how each line is split where commas existed in the original text. This delimiter can be a simple char, as we have shown, or a more complex StrMatcher object.

Google Guava is an open source set of utility Java classes and methods. The primary goal of Guava, as with many APIs, is to relieve the burden of writing basic Java utilities so developers can focus on business processes. We are going to talk about two main tools in Guava in this chapter: the Joiner class and the Splitter class. Tokenization is accomplished in Guava using its Splitter class's split method. The following is a simple example:

Splitter simpleSplit = Splitter.on(',').omitEmptyStrings().trimResults(); 
Iterable<String> words = simpleSplit.split(dirtyText);  
for(String token: words){ 
  out.print(token); 
} 

This splits each token on commas and produces output like our last example. We can modify the parameter of the on method to split on the character of our choosing. Notice the method chaining which allows us to omit empty strings and trim leading and trailing spaces. For these reasons, and other advanced capabilities, Google Guava is considered by some to be the best tokenizer available for Java.

LingPipe is a linguistical toolkit available for language processing in Java. It provides more specialized support for text splitting with its TokenizerFactory interface. We implement a LingPipe IndoEuropeanTokenizerFactory tokenizer in the Simple text cleaning section.

We will use the string shown before from Moby Dick to demonstrate some of the basic String class methods. Notice the use of the toLowerCase and trim methods. Datasets often have non-standard casing and extra leading or trailing spaces. These methods ensure uniformity of our dataset. We also use the replaceAll method twice. In the first instance, we use a regular expression to replace all numbers and anything that is not a word or whitespace character with a single space. The second instance replaces all back-to-back whitespace characters with a single space:

out.println(dirtyText); 
dirtyText =    dirtyText.toLowerCase().replaceAll("[\\d[^\\w\\s]]+", " "); 
dirtyText = dirtyText.trim(); 
while(dirtyText.contains("  ")){ 
  dirtyText = dirtyText.replaceAll("  ", " "); 
} 
out.println(dirtyText);  

When executed, the code produces the following output, truncated:

Call me Ishmael. Some years ago- never mind how long precisely -
call me ishmael some years ago never mind how long precisely

Our next example produces the same result but approaches the problem with regular expressions. In this case, we replace all of the numbers and other special characters first. Then we use method chaining to standardize our casing, remove leading and trailing spaces, and split our words into a String array. The split method allows you to break apart text on a given delimiter. In this case, we chose to use the regular expression \\W, which represents anything that is not a word character:

out.println(dirtyText); 
dirtyText = dirtyText.replaceAll("[\\d[^\\w\\s]]+", ""); 
String[] cleanText = dirtyText.toLowerCase().trim().split("[\\W]+"); 
for(String clean : cleanText){ 
  out.print(clean + " ");
} 

This code produces the same output as shown previously.

Although arrays are useful for many applications, it is often important to recombine text after cleaning. In the next example, we employ the join method to combine our words once we have cleaned them. We use the same chained methods as shown previously to clean and split our text. The join method joins every word in the array words and inserts a space between each word:

out.println(dirtyText); 
String[] words =    dirtyText.toLowerCase().trim().split("[\\W\\d]+"); 
String cleanText = String.join(" ", words); 
out.println(cleanText); 

Again, this code produces the same output as shown previously. An alternate version of the join method is available using Google Guava. Here is a simple implementation of the same process we used before, but using the Guava Joiner class:

out.println(dirtyText);  
String[] words =    dirtyText.toLowerCase().trim().split("[\\W\\d]+"); 
String cleanText = Joiner.on(" ").skipNulls().join(words); 
out.println(cleanText); 

This version provides additional options, including skipping nulls, as shown before. The output remains the same.

Text analysis sometimes requires the omission of common, non-specific words such as the, and, or but. These words are known as stop words are there are several tools available for removing them from text. There are various ways to store a list of stop words, but for the following examples, we will assume they are contained in a file. To begin, we create a new Scanner object to read in our stop words. Then we take the text we wish to transform and store it in an ArrayList using the Arrays class's asList method. We will assume here the text has already been cleaned and normalized. It is essential to consider casing when using String class methods—and is not the same as AND or And, although all three may be stop words you wish to eliminate:

Scanner readStop = new Scanner(new File("C://stopwords.txt")); 
ArrayList<String> words = new    ArrayList<String>(Arrays.asList((dirtyText)); 
out.println("Original clean text: " + words.toString()); 

We also create a new ArrayList to hold a list of stop words actually found in our text. This will allow us to use the ArrayList class removeAll method shortly. Next, we use our Scanner to read through our file of stop words. Notice how we also call the toLowerCase and trim methods against each stop word. This is to ensure that our stop words match the formatting in our text. In this example, we employ the contains method to determine whether our text contains the given stop word. If so, we add it to our foundWords ArrayList. Once we have processed all the stop words, we call removeAll to remove them from our text: