Chapter 9. Analyzing Social Data Participation

Social networks and websites have revolutionized the Internet. Most people online participate in some social network, either it's Facebook, Twitter, Pinterest, GitHub, StackOverflow, or any of the zillion other social networking websites that have sprung up. They're an important way for people to connect and stay in contact, but they're also a major source of data about people's relationships and activities.

Analyzing this data is important for a number of reasons. Of course, advertisers and marketers want to squeeze as much information out of the data as they can. But if you're running the social network, you'll want to analyze the data to figure out what's working and what's falling flat. You want to ask yourself constantly what you can do to engage users better and to make your social network a more compelling, enjoyable, or useful experience for your users.

Over the course of this chapter, we'll get an open data dump from the StackExchange...

Setting up the project

Before we get started, let's set up the project. I've done this using Leiningen 2 (http://leiningen.org/) and Stuart Sierra's reloaded project template (https://github.com/stuartsierra/reloaded). I named the project social-so by running the following code:

$ lein new reloaded social-so

Of course, we'll need more dependencies. The following is the project.clj file for this chapter:

(defproject social-so "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [org.clojure/data.xml "0.0.7"]
                 [org.codehaus.jsr166-mirror/jsr166y "1.7.0"]
                 [org.clojure/data.json "0.2.4"]
                 [cc.mallet/mallet "2.0.7"]
                 [org.jsoup/jsoup "1.7.3"]]
  :profiles {:dev {:dependencies
                   [[org.clojure/tools.namespace "0.2.4"]]
                   :source-paths ["dev"]}}
  :jvm-opts ["-Xmx2048m"])

The highlights here are that we'll use org.clojure/data.xml to read XML files, org.clojure/data.json to...

Summary

However the results of the last experiment, we can see that there's a lot of information embedded in social networks. Depending on the nature of the network, we can have different kinds of interactions and different kinds of data in the network.

In the final chapter, which is next, we'll look at whether analyzing financial data and using machine learning to examine news documents help predict the future of stock prices.