In this chapter, we will cover the following recipes:
Harnessing data from various sources
Accumulating text data from a file path
Catching I/O code faults
Keeping and representing data from a CSV file
Examining a JSON file with the aeson package
Reading an XML file using the HXT package
Capturing table rows from an HTML page
Understanding how to perform HTTP GET requests
Learning how to perform HTTP POST requests
Traversing online directories for data
Using MongoDB queries in Haskell
Reading from a remote MongoDB server
Exploring data from a SQLite database

Data is everywhere, logging is cheap, and analysis is inevitable. One of the most fundamental concepts of this chapter is based on gathering useful data. After building a large collection of usable text, which we call the corpus, we must learn to represent this content in code. The primary focus will be first on obtaining data and later on enumerating ways of representing it.
Gathering data is arguably as important as analyzing it to extrapolate results and form valid generalizable claims. It is a scientific pursuit; therefore, great care must and will be taken to ensure unbiased and representative sampling. We recommend following along closely in this chapter because the remainder of the book depends on having a source of data to work with. Without data, there isn't much to analyze, so we should carefully observe the techniques laid out to build our own formidable corpus.
The first recipe enumerates various sources to start gathering data online. The next few recipes deal with using local data of different file formats. We then learn how to download data from the Internet using our Haskell code. Finally, we finish this chapter with a couple of recipes on using databases in Haskell.
Information can be described as structured, unstructured, or sometimes a mix of the two—semi-structured.
In a very general sense, structured data is anything that can be parsed by an algorithm. Common examples include JSON, CSV, and XML. If given structured data, we can design a piece of code to dissect the underlying format and easily produce useful results. As mining structured data is a deterministic process, it allows us to automate the parsing. This in effect lets us gather more input to feed our data analysis algorithms.
Unstructured data is everything else. It is data not defined in a specified manner. Written languages such as English are often regarded as unstructured because of the difficulty in parsing a data model out of a natural sentence.
In our search for good data, we will often find a mix of structured and unstructured text. This is called semi-structured text.
This recipe will primarily focus on obtaining structured and semi-structured data from the following sources.
Tip
Unlike most recipes in this book, this recipe does not contain any code. The best way to read this book is by skipping around to the recipes that interest you.
We will browse through the links provided in the following sections to build up a list of sources to harness interesting data in usable formats. However, this list is not at all exhaustive.
Some of these sources have an Application Programming Interface (API) that allows more sophisticated access to interesting data. An API specifies the interactions and defines how data is communicated.
The New York Times has one of the most polished API documentation to access anything from real-estate data to article search results. This documentation can be found at http://developer.nytimes.com.
The Guardian also supports a massive datastore with over a million articles at http://www.theguardian.com/data.
USA TODAY provides some interesting resources on books, movies, and music reviews. The technical documentation can be found at http://developer.usatoday.com.
The BBC features some interesting API endpoints including information on BBC programs, and music located at http://www.bbc.co.uk/developer/technology/apis.html.
Facebook, Twitter, Instagram, Foursquare, Tumblr, SoundCloud, Meetup, and many other social networking sites support APIs to access some degree of social information.
For specific APIs such as weather or sports, Mashape is a centralized search engine to narrow down the search to some lesser-known sources. Mashape is located at https://www.mashape.com/
Most data sources can be visualized using the Google Public Data search located at http://www.google.com/publicdata.
For a list of all countries with names in various data formats, refer to the repository located at https://github.com/umpirsky/country-list.
Some data sources are hosted openly by universities around the world for research purposes.
To analyze health care data, the University of Washington has published Institute for Health Metrics and Evaluation (IHME) to collect rigorous and comparable measurement of the world's most important health problems. Navigate to http://www.healthdata.org for more information.
The MNIST database of handwritten digits from NYU, Google Labs, and Microsoft Research is a training set of normalized and centered samples for handwritten digits. Download the data from http://yann.lecun.com/exdb/mnist.
Human Development Reports publishes annual updates ranging from international data about adult literacy to the number of people owning personal computers. It describes itself as having a variety of public international sources and represents the most current statistics available for those indicators. More information is available at http://hdr.undp.org/en/statistics.
The World Bank is the source for poverty and world development data. It regards itself as a free source that enables open access to data about development in countries around the globe. Find more information at http://data.worldbank.org/.
The World Health Organization provides data and analyses for monitoring the global health situation. See more information at http://www.who.int/research/en.
UNICEF also releases interesting statistics, as the quote from their website suggests:
"The UNICEF database contains statistical tables for child mortality, diseases, water sanitation, and more vitals. UNICEF claims to play a central role in monitoring the situation of children and women—assisting countries in collecting and analyzing data, helping them develop methodologies and indicators, maintaining global databases, disseminating and publishing data. Find the resources at http://www.unicef.org/statistics."
The United Nations hosts interesting publicly available political statistics at http://www.un.org/en/databases.
If we crave the urge to discover patterns in the United States (U.S.) government like Nicholas Cage did in the feature film National Treasure (2004), then http://www.data.gov/ is our go-to source. It's the U.S. government's active effort to provide useful data. It is described as a place to increase "public access to high-value, machine-readable datasets generated by the executive branch of the Federal Government". Find more information at http://www.data.gov.
The United States Census Bureau releases population counts, housing statistics, area measurements, and more. These can be found at http://www.census.gov.
One of the easiest ways to get started with processing input is by reading raw text from a local file. In this recipe, we will be extracting all the text from a specific file path. Furthermore, to do something interesting with the data, we will count the number of words per line.
Tip
Haskell is a purely functional programming language, right? Sure, but obtaining input from outside the code introduces impurity. For elegance and reusability, we must carefully separate pure from impure code.
We will first create an input.txt
text file with a couple of lines of text to be read by the program. We keep this file in an easy-to-access directory because it will be referenced later. For example, the text file we're dealing with contains a seven-line quote by Plato. Here's what our terminal prints when we issue the following command:
$ cat input.txt And how will you inquire, Socrates, into that which you know not? What will you put forth as the subject of inquiry? And if you find what you want, how will you ever know that this is what you did not know?
Tip
Downloading the example code
You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you. The code will also be hosted on GitHub at https://github.com/BinRoot/Haskell-Data-Analysis-Cookbook.
Create a new file to start coding. We call our file Main.hs.
As with all executable Haskell programs, start by defining and implementing the
main
function, as follows:main :: IO () main = do
Use Haskell's
readFile :: FilePath -> IO String
function to extract data from aninput.txt
file path. Note that a file path is just a synonym forString
. With the string in memory, pass it into acountWords
function to count the number of words in each line, as shown in the following steps:input <- readFile "input.txt" print $ countWords input
Lastly, define our pure function,
countWords
, as follows:countWords :: String -> [Int] countWords input = map (length.words) (lines input)
The program will print out the number of words per line represented as a list of numbers as follows:
$ runhaskell Main.hs [6,6,10,7,6,7]
Haskell provides useful input and output (I/O) capabilities for reading input and writing output in different ways. In our case, we use readFile
to specify a path of a file to be read. Using the do
keyword in main
suggests that we are joining several IO actions together. The output of readFile
is an I/O string, which means it is an I/O action that returns a String
type.
Now we're about to get a bit technical. Pay close attention. Alternatively, smile and nod. In Haskell, the I/O data type is an instance of something called a Monad. This allows us to use the <-
notation to draw the string out of this I/O action. We then make use of the string by feeding it into our countWords
function that counts the number of words in each line. Notice how we separated the countWords
function apart from the impure main
function.
Finally, we print the output of countWords
. The $
notation means we are using a function application to avoid excessive parenthesis in our code. Without it, the last line of main
would look like print (countWords input)
.
For simplicity's sake, this code is easy to read but very fragile. If an input.txt
file does not exist, then running the code will immediately crash the program. For example, the following command will generate the error message:
$ runhaskell Main.hs Main.hs: input.txt: openFile: does not exist…
To make this code fault tolerant, refer to the Catching I/O code faults recipe.
Making sure our code doesn't crash in the process of data mining or analysis is a substantially genuine concern. Some computations may take hours, if not days. Haskell gifts us with type safety and strong checks to help ensure a program will not fail, but we must also take care to double-check edge cases where faults may occur.
For instance, a program may crash ungracefully if the local file path is not found. In the previous recipe, there was a strong dependency on the existence of input.txt
in our code. If the program is unable to find the file, it will produce the following error:
mycode: input.txt: openFile: does not exist (No such file or directory)
Naturally, we should decouple the file path dependency by enabling the user to specify his/her file path as well as by not crashing in the event that the file is not found.
Consider the following revision of the source code.
Create a new file, name it Main.hs
, and perform the following steps:
First, import a library to catch fatal errors as follows:
import Control.Exception (catch, SomeException)
Next, import a library to get command-line arguments so that the file path is dynamic. We use the following line of code to do this:
import System.Environment (getArgs)
Continuing as before, define and implement
main
as follows:main :: IO () main = do
Define a
fileName
string depending on the user-provided argument, defaulting toinput.txt
if there is no argument. The argument is obtained by retrieving an array of strings from the library function,getArgs :: IO [String]
, as shown in the following steps:args <- getArgs let filename = case args of (a:_) -> a _ -> "input.txt"
Now apply
readFile
on this path, but catch any errors using the library'scatch :: Exception e => IO a -> (e -> IO a) -> IO a
function. The first argument to catch is the computation to run, and the second argument is the handler to invoke if an exception is raised, as shown in the following commands:input <- catch (readFile fileName) $ \err -> print (err::SomeException) >> return ""
The
input
string will be empty if there were any errors reading the file. We can now useinput
for any purpose using the following command:print $ countWords input
Don't forget to define the
countWords
function as follows:countWords input = map (length.words) (lines input)
This recipe demonstrates two ways to catch errors, listed as follows:
Firstly, we use a case expression that pattern matches against any argument passed in. Therefore, if no arguments are passed, the
args
list is empty, and the last pattern,"_"
, is caught, resulting in a default filename ofinput.txt
.Secondly, we use the catch function to handle an error if something goes wrong. When having trouble reading a file, we allow the code to continue running by setting
input
to an empty string.
Conveniently, Haskell also comes with a doesFileExist :: FilePath -> IO Bool
function from the System.Directory
module. We can simplify the preceding code by modifying the input <- …
line. It can be replaced with the following snippet of code:
exists <- doesFileExist filename input <- if exists then readFile filename else return ""
In this case, the code reads the file as an input only if it exists. Do not forget to add the following import
line at the top of the source code:
import System.Directory (doesFileExist)
Comma Separated Value (CSV) is a format to represent a table of values in plain text. It's often used to interact with data from spreadsheets. The specifications for CSV are described in RFC 4180, available at http://tools.ietf.org/html/rfc4180.
In this recipe, we will read a local CSV file called input.csv
consisting of various names and their corresponding ages. Then, to do something useful with the data, we will find the oldest person.
Prepare a simple CSV file with a list of names and their corresponding ages. This can be done using a text editor or by exporting from a spreadsheet, as shown in the following figure:

The raw input.csv
file contains the following text:
$ cat input.csv name,age Alex,22 Anish,22 Becca,23 Jasdev,22 John,21 Jonathon,21 Kelvin,22 Marisa,19 Shiv,22 Vinay,22
The code also depends on the csv
library. We may install the library through Cabal using the following command:
$ cabal install csv
Import the
csv
library using the following line of code:import Text.CSV
Define and implement
main
, where we will read and parse the CSV file, as shown in the following code:main :: IO () main = do let fileName = "input.csv" input <- readFile fileName
Apply
parseCSV
to the filename to obtain a list of rows, representing the tabulated data. The output ofparseCSV
isEither ParseError CSV
, so ensure that we consider both theLeft
andRight
cases:let csv = parseCSV fileName input either handleError doWork csv handleError csv = putStrLn "error parsing" doWork csv = (print.findOldest.tail) csv
Now we can work with the CSV data. In this example, we find and print the row containing the oldest person, as shown in the following code snippet:
findOldest :: [Record] -> Record findOldest [] = [] findOldest xs = foldl1 (\a x -> if age x > age a then x else a) xs age [a,b] = toInt a toInt :: String -> Int toInt = read
After running
main
, the code should produce the following output:$ runhaskell Main.hs ["Becca", "23"]
The CSV data structure in Haskell is represented as a list of records. Record
is merely a list of Fields
, and Field
is a type synonym for String
. In other words, it is a collection of rows representing a table, as shown in the following figure:

The parseCSV
library function returns an Either
type, with the Left
side being a ParseError
and the Right
side being the list of lists. The Either l r
data type is very similar to the Maybe a
type which has the Just a
or Nothing
constructor.
We use the either
function to handle the Left
and Right
cases. The Left
case handles the error, and the Right
case handles the actual work to be done on the data. In this recipe, the Right
side is a Record
. The fields in Record
are accessible through any list operations such as head
, last
, !!
, and so on.
JavaScript Object Notation (JSON) is a way to represent key-value pairs in plain text. The format is described extensively in RFC 4627 (http://www.ietf.org/rfc/rfc4627).
In this recipe, we will parse a JSON description about a person. We often encounter JSON in APIs from web applications.
Install the aeson
library from hackage using Cabal.
Prepare an input.json
file representing data about a mathematician, such as the one in the following code snippet:
$ cat input.json {"name":"Gauss", "nationality":"German", "born":1777, "died":1855}
We will be parsing this JSON and representing it as a usable data type in Haskell.
Use the
OverloadedStrings
language extension to represent strings asByteString
, as shown in the following line of code:{-# LANGUAGE OverloadedStrings #-}
Import
aeson
as well as some helper functions as follows:import Data.Aeson import Control.Applicative import qualified Data.ByteString.Lazy as B
Create the data type corresponding to the JSON structure, as shown in the following code:
data Mathematician = Mathematician { name :: String , nationality :: String , born :: Int , died :: Maybe Int }
Provide an instance for the
parseJSON
function, as shown in the following code snippet:instance FromJSON Mathematician where parseJSON (Object v) = Mathematician <$> (v .: "name") <*> (v .: "nationality") <*> (v .: "born") <*> (v .:? "died")
Define and implement
main
as follows:main :: IO () main = do
Read the input and decode the JSON, as shown in the following code snippet:
input <- B.readFile "input.json" let mm = decode input :: Maybe Mathematician case mm of Nothing -> print "error parsing JSON" Just m -> (putStrLn.greet) m
Now we will do something interesting with the data as follows:
greet m = (show.name) m ++ " was born in the year " ++ (show.born) m
We can run the code to see the following output:
$ runhaskell Main.hs "Gauss" was born in the year 1777
Aeson takes care of the complications in representing JSON. It creates native usable data out of a structured text. In this recipe, we use the .:
and .:?
functions provided by the Data.Aeson
module.
As the Aeson
package uses ByteStrings
instead of Strings
, it is very helpful to tell the compiler that characters between quotation marks should be treated as the proper data type. This is done in the first line of the code which invokes the OverloadedStrings
language extension.
Tip
Language extensions such as OverloadedStrings
are currently supported only by the
Glasgow Haskell Compiler (GHC).
We use the decode
function provided by Aeson to transform a string into a data type. It has the type FromJSON a => B.ByteString -> Maybe a
. Our Mathematician
data type must implement an instance of the FromJSON
typeclass to properly use this function. Fortunately, the only required function for implementing FromJSON
is parseJSON
. The syntax used in this recipe for implementing parseJSON
is a little strange, but this is because we're leveraging applicative functions and lenses, which are more advanced Haskell topics.
The .:
function has two arguments, Object
and Text
, and returns a Parser a
data type. As per the documentation, it retrieves the value associated with the given key of an object. This function is used if the key and the value exist in the JSON document. The :?
function also retrieves the associated value from the given key of an object, but the existence of the key and value are not mandatory. So, we use .:?
for optional key value pairs in a JSON document.
If the implementation of the FromJSON
typeclass is too involved, we can easily let GHC automatically fill it out using the DeriveGeneric
language extension. The following is a simpler rewrite of the code:
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE DeriveGeneric #-}
import Data.Aeson
import qualified Data.ByteString.Lazy as B
import GHC.Generics
data Mathematician = Mathematician { name :: String
, nationality :: String
, born :: Int
, died :: Maybe Int
} deriving Generic
instance FromJSON Mathematician
main = do
input <- B.readFile "input.json"
let mm = decode input :: Maybe Mathematician
case mm of
Nothing -> print "error parsing JSON"
Just m -> (putStrLn.greet) m
greet m = (show.name) m ++" was born in the year "++ (show.born) m
Although Aeson is powerful and generalizable, it may be an overkill for some simple JSON interactions. Alternatively, if we wish to use a very minimal JSON parser and printer, we can use Yocto, which can be downloaded from http://hackage.haskell.org/package/yocto.
Extensible Markup Language (XML) is an encoding of plain text to provide machine-readable annotations on a document. The standard is specified by W3C (http://www.w3.org/TR/2008/REC-xml-20081126/).
In this recipe, we will parse an XML document representing an e-mail conversation and extract all the dates.
We will first set up an XML file called input.xml
with the following values, representing an e-mail thread between Databender and Princess on December 18, 2014 as follows:
$ cat input.xml <thread> <email> <to>Databender</to> <from>Princess</from> <date>Thu Dec 18 15:03:23 EST 2014</date> <subject>Joke</subject> <body>Why did you divide sin by tan?</body> </email> <email> <to>Princess</to> <from>Databender</from> <date>Fri Dec 19 3:12:00 EST 2014</date> <subject>RE: Joke</subject> <body>Just cos.</body> </email> </thread>
Using Cabal, install the HXT library which we use for manipulating XML documents:
$ cabal install hxt
We only need one import, which will be for parsing XML, using the following line of code:
import Text.XML.HXT.Core
Define and implement
main
and specify the XML location. For this recipe, the file is retrieved frominput.xml
. Refer to the following code:main :: IO () main = do input <- readFile "input.xml"
Apply the
readString
function to the input and extract all the date documents. We filter items with a specific name using thehasName :: String -> a XmlTree XmlTree
function. Also, we extract the text using thegetText :: a XmlTree String
function, as shown in the following code snippet:dates <- runX $ readString [withValidate no] input //> hasName "date" //> getText
We can now use the list of extracted dates as follows:
print dates
By running the code, we print the following output:
$ runhaskell Main.hs ["Thu Dec 18 15:03:23 EST 2014", "Fri Dec 19 3:12:00 EST 2014"]
The library function, runX
, takes in an
Arrow. Think of an Arrow as a more powerful version of a Monad. Arrows allow for stateful global XML processing. Specifically, the runX
function in this recipe takes in IOSArrow XmlTree String
and returns an IO
action of the String
type. We generate this IOSArrow
object using the readString
function, which performs a series of operations to the XML data.
For a deep insight into the XML document, //>
should be used whereas />
only looks at the current level. We use the //>
function to look up the date attributes and display all the associated text.
As defined in the documentation, the hasName
function tests whether a node has a specific name, and the getText
function selects the text of a text node. Some other functions include the following:
All the Arrow functions can be found on the Text.XML.HXT.Arrow.XmlArrow
documentation at http://hackage.haskell.org/package/hxt/docs/Text-XML-HXT-Arrow-XmlArrow.html.
Mining Hypertext Markup Language (HTML) is often a feat of identifying and parsing only its structured segments. Not all text in an HTML file may be useful, so we find ourselves only focusing on a specific subset. For instance, HTML tables and lists provide a strong and commonly used structure to extract data whereas a paragraph in an article may be too unstructured and complicated to process.
In this recipe, we will find a table on a web page and gather all rows to be used in the program.
We will be extracting the values from an HTML table, so start by creating an input.html
file containing a table as shown in the following figure:

The HTML behind this table is as follows:
$ cat input.html <!DOCTYPE html> <html> <body> <h1>Course Listing</h1> <table> <tr> <th>Course</th> <th>Time</th> <th>Capacity</th> </tr> <tr> <td>CS 1501</td> <td>17:00</td> <td>60</td> </tr> <tr> <td>MATH 7600</td> <td>14:00</td> <td>25</td> </tr> <tr> <td>PHIL 1000</td> <td>9:30</td> <td>120</td> </tr> </table> </body> </html>
If not already installed, use Cabal to set up the HXT library and the split library, as shown in the following command lines:
$ cabal install hxt $ cabal install split
We will need the
htx
package for XML manipulations and thechunksOf
function from the split package, as presented in the following code snippet:import Text.XML.HXT.Core import Data.List.Split (chunksOf)
Define and implement
main
to read theinput.html
file.main :: IO () main = do input <- readFile "input.html"
Feed the HTML data into
readString
, thereby settingwithParseHTML
toyes
and optionally turning off warnings. Extract all thetd
tags and obtain the remaining text, as shown in the following code:texts <- runX $ readString [withParseHTML yes, withWarnings no] input //> hasName "td" //> getText
The data is now usable as a list of strings. It can be converted into a list of lists similar to how CSV was presented in the previous CSV recipe, as shown in the following code:
let rows = chunksOf 3 texts print $ findBiggest rows
By folding through the data, identify the course with the largest capacity using the following code snippet:
findBiggest :: [[String]] -> [String] findBiggest [] = [] findBiggest items = foldl1 (\a x -> if capacity x > capacity a then x else a) items capacity [a,b,c] = toInt c capacity _ = -1 toInt :: String -> Int toInt = read
Running the code will display the class with the largest capacity as follows:
$ runhaskell Main.hs {"PHIL 1000", "9:30", "120"}
One of the most resourceful places to find good data is online. GET requests are common methods of communicating with an HTTP web server. In this recipe, we will grab all the links from a Wikipedia article and print them to the terminal. To easily grab all the links, we will use a helpful library called HandsomeSoup
, which lets us easily manipulate and traverse a webpage through CSS selectors.
We will be collecting all links from a Wikipedia web page. Make sure to have an Internet connection before running this recipe.
Install the HandsomeSoup
CSS selector package, and also install the HXT library if it is not already installed. To do this, use the following commands:
$ cabal install HandsomeSoup $ cabal install hxt
This recipe requires
hxt
for parsing HTML and requiresHandsomeSoup
for the easy-to-use CSS selectors, as shown in the following code snippet:import Text.XML.HXT.Core import Text.HandsomeSoup
Define and implement
main
as follows:main :: IO () main = do
Pass in the URL as a string to HandsomeSoup's
fromUrl
function:let doc = fromUrl "http://en.wikipedia.org/wiki/Narwhal"
Select all links within the
bodyContent
field of the Wikipedia page as follows:links <- runX $ doc >>> css "#bodyContent a" ! "href" print links
A POST request is another very common HTTP server request used by many APIs. We will be mining the University of Virginia directory search. When sending a POST request for a search query, the Lightweight Directory Access Protocol (LDAP) server replies with a web page of search results.
For this recipe, access to the Internet is necessary.
Install the HandsomeSoup
CSS selector package, and also install the HXT library if it is not already installed:
$ cabal install HandsomeSoup $ cabal install hxt
Import the following libraries:
import Network.HTTP import Network.URI (parseURI) import Text.XML.HXT.Core import Text.HandsomeSoup import Data.Maybe (fromJust)
Define the POST request specified by the directory search website. Depending on the server, the following POST request details would be different. Refer to the following code snippet:
myRequestURL = "http://www.virginia.edu/cgi-local/ldapweb" myRequest :: String -> Request_String myRequest query = Request { rqURI = fromJust $ parseURI myRequestURL , rqMethod = POST , rqHeaders = [ mkHeader HdrContentType "text/html" , mkHeader HdrContentLength $ show $ length body ] , rqBody = body } where body = "whitepages=" ++ query
Define and implement
main
to run the POST request on a query as follows:main :: IO () main = do response <- simpleHTTP $ myRequest "poon"
Gather the HTML and parse it:
html <- getResponseBody response let doc = readString [withParseHTML yes, withWarnings no] html
Find the table rows and print it out using the following:
rows <- runX $ doc >>> css "td" //> getText print rows
Running the code will display all search results relating to "poon"
, such as "Poonam" or "Witherspoon".
A POST request needs the specified URI, headers, and body. By filling out a Request
data type, it can be used to establish a server request.
A directory search typically provides names and contact information per query. By brute forcing many of these search queries, we can obtain all data stored in the directory listing database. This recipe runs thousands of search queries to obtain as much data as possible from a directory search. This recipe is provided only as a learning tool to see the power and simplicity of data gathering in Haskell.
Make sure to have a strong Internet connection.
Install the hxt
and HandsomeSoup
packages using Cabal:
$ cabal install hxt $ cabal install HandsomeSoup
Set up the following dependencies:
import Network.HTTP import Network.URI import Text.XML.HXT.Core import Text.HandsomeSoup
Define a
SearchResult
type, which may either fault in an error or result in a success, as presented in the following code:type SearchResult = Either SearchResultErr [String] data SearchResultErr = NoResultsErr | TooManyResultsErr | UnknownErr deriving (Show, Eq)
Define the POST request specified by the directory search website. Depending on the server, the POST request will be different. Instead of rewriting code, we use the
myRequest
function defined in the previous recipe.Write a helper function to obtain the document from a HTTP POST request, as shown in the following code:
getDoc query = do rsp <- simpleHTTP $ myRequest query html <- getResponseBody rsp return $ readString [withParseHTML yes, withWarnings no] html
Scan the HTML document and return whether there is an error or provide the resulting data. The code in this function is dependent on the error messages produced by the web page. In our case, the error messages are the following:
scanDoc doc = do errMsg <- runX $ doc >>> css "h3" //> getText case errMsg of [] -> do text <- runX $ doc >>> css "td" //> getText return $ Right text "Error: Sizelimit exceeded":_ -> return $ Left TooManyResultsErr "Too many matching entries were found":_ -> return $ Left TooManyResultsErr "No matching entries were found":_ -> return $ Left NoResultsErr _ -> return $ Left UnknownErr
Define and implement
main
. We will use a helper function,main'
, as shown in the following code snippet, to recursively brute force the directory listing:main :: IO () main = main' "a"
Run a search of the query and then recursively again on the next query:
main' query = do print query doc <- getDoc query searchResult <- scanDoc doc print searchResult case searchResult of Left TooManyResultsErr -> main' (nextDeepQuery query) _ -> if (nextQuery query) >= endQuery then print "done!" else main' (nextQuery query)
Write helper functions to define the next logical query as follows:
nextDeepQuery query = query ++ "a" nextQuery "z" = endQuery nextQuery query = if last query == 'z' then nextQuery $ init query else init query ++ [succ $ last query] endQuery = [succ 'z']
The code starts by searching for "a" in the directory lookup. This will most likely fault in an error as there are too many results. So, in the next iteration, the code will refine its search by querying for "aa", then "aaa", until there is no longer TooManyResultsErr :: SearchResultErr
.
Then, it will enumerate to the next logical search query "aab", and if that produces no result, it will search for "aac", and so on. This brute force prefix search will obtain all items in the database. We can gather the mass of data, such as names and department types, to perform interesting clustering or analysis later on. The following figure shows how the program starts:

MongoDB is a nonrelational schemaless database. In this recipe, we will obtain all data from MongoDB into Haskell.
We need to install MongoDB on our local machine and have a database instance running in the background while we run the code in this recipe.
MongoDB installation instructions are located at http://www.mongodb.org. On Debian-based operating systems, we can use apt-get
to install MongoDB, using the following command line:
$ sudo apt-get install mongodb
Run the database daemon by specifying the database file path as follows:
$ mkdir ~/db $ mongod --dbpath ~/db
Fill up a "people"
collection with dummy data as follows:
$ mongo > db.people.insert( {first: "Joe", last: "Shmoe"} )
Install the MongoDB package from Cabal using the following command:
$ cabal install mongoDB
Use the
OverloadedString
andExtendedDefaultRules
language extensions to make the MongoDB library easier to use:{-# LANGUAGE OverloadedStrings, ExtendedDefaultRules #-} import Database.MongoDB
Define and implement
main
to set up a connection to the locally hosted database. Run MongoDB queries defined in therun
function as follows:main :: IO () main = do let db = "test" pipe <- runIOE $ connect (host "127.0.0.1") e <- access pipe master db run close pipe print e
In
run
, we can combine multiple operations. For this recipe,run
will only perform one task, that is, gather data from the"people"
collection:run = getData getData = rest =<< find (select [] "people") {sort=[]}
A pipe is established by the driver between the running program and the database. This allows running MongoDB operations to bridge the program with the database. The find
function takes a query, which we construct by evoking the select :: Selector -> Collection -> aQueryOrSelection
function.
Other functions can be found in the documentation at http://hackage.haskell.org/package/mongoDB/docs/Database-MongoDB-Query.html.
In many cases, it may be more feasible to set up a MongoDB instance on a remote machine. This recipe will cover how to obtain data from a MongoDB hosted remotely.
We should create a remote database. MongoLab (https://mongolab.com) and MongoHQ (http://www.mongohq.com) offer MongoDB as a service and have free options to set up a small development database.
Tip
These services will require us to accept their terms and conditions. For some of us, it may be best to host the database in our own remote server.
Install the MongoDB package from Cabal as follows:
$ cabal install mongoDB
Also, install the helper following helper libraries as follows:
$ cabal install split $ cabal install uri
Use the
OverloadedString
andExtendedDefaultRules
language extensions required by the library. Import helper functions as follows:{-# LANGUAGE OverloadedStrings, ExtendedDefaultRules #-} import Database.MongoDB import Text.URI import Data.Maybe import qualified Data.Text as T import Data.List.Split
Specify the remote URI for the database connection as follows:
mongoURI = "mongodb://user:pass@ds12345.mongolab.com:53788/mydb"
The username, password, hostname, port address number, and database name must be extracted from the URI, as presented in the following code snippet:
uri = fromJust $ parseURI mongoURI getUser = head $ splitOn ":" $ fromJust $ uriUserInfo uri getPass = last $ splitOn ":" $ fromJust $ uriUserInfo uri getHost = fromJust $ uriRegName uri getPort = case uriPort uri of Just port -> show port Nothing -> (last.words.show) defaultPort getDb = T.pack $ tail $ uriPath uri
Create a database connection by reading the host port of the remote URI as follows:
main :: IO () main = do let hostport = getHost ++ ":" ++ getPort pipe <- runIOE $ connect (readHostPort hostport) e <- access pipe master getDb run close pipe print e
Optionally authenticate to the database and obtain data from the
"people"
collection as follows:run = do auth (T.pack getUser) (T.pack getPass) getData getData = rest =<< find (select [] "people") {sort=[]}
SQLite is a relational database that enforces a strict schema. It is simply a file on a machine that we can interact with through Structured Query Language (SQL). There is an easy-to-use Haskell library to send these SQL commands to our database.
In this recipe, we will use such a library to extract all data from a SQLite database.
We need to install the SQLite database if it isn't already set up. It can be obtained from http://www.sqlite.org. On Debian systems, we can get it from apt-get
using the following command:
$ sudo apt-get install sqlite3
Now create a simple database to test our code, using the following commands:
$ sqlite3 test.db "CREATE TABLE test \ (id INTEGER PRIMARY KEY, str text); \ INSERT INTO test (str) VALUES ('test string');"
We must also install the SQLite Haskell package from Cabal as follows:
$ cabal install sqlite-simple
This recipe will dissect the example code presented on the library's documentation page available at http://hackage.haskell.org/package/sqlite-simple/docs/Database-SQLite-Simple.html.
Use the
OverloadedStrings
language extension and import the relevant libraries, as shown in the following code:{-# LANGUAGE OverloadedStrings #-} import Control.Applicative import Database.SQLite.Simple import Database.SQLite.Simple.FromRow
Define a data type for each SQLite table field. Provide it with an instance of the
FromRow
typeclass so that we may easily parse it from the table, as shown in the following code snippet:data TestField = TestField Int String deriving (Show) instance FromRow TestField where fromRow = TestField <$> field <*> field
And lastly, open the database to import everything as follows:
main :: IO () main = do conn <- open "test.db" r <- query_ conn "SELECT * from test" :: IO [TestField] mapM_ print r close conn