Learning Data Mining with Python - Second Edition

4 (1 reviews total)
By Robert Layton
    What do you get with a Packt Subscription?

  • Instant access to this title and 7,500+ eBooks & Videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Free Chapter
    Getting Started with Data Mining
About this book

This book teaches you to design and develop data mining applications using a variety of datasets, starting with basic classification and affinity analysis. This book covers a large number of libraries available in Python, including the Jupyter Notebook, pandas, scikit-learn, and NLTK.

You will gain hands on experience with complex data types including text, images, and graphs. You will also discover object detection using Deep Neural Networks, which is one of the big, difficult areas of machine learning right now.

With restructured examples and code samples updated for the latest edition of Python, each chapter of this book introduces you to new algorithms and techniques. By the end of the book, you will have great insights into using Python for data mining and understanding of the algorithms as well as implementations.

Publication date:
April 2017


Chapter 1. Getting Started with Data Mining

We are collecting information about our world on a scale that has never been seen before in the history of humanity. Along with this trend, we are now placing more day-to-day importance on the use of this information in everyday life. We now expect our computers to translate web pages into other languages, predict the weather with high accuracy, suggest books we would like, and to diagnose our health issues. These expectations will grow into the future, both in application breadth and efficacy. Data Mining is a methodology that we can employ to train computers to make decisions with data and forms the backbone of many high-tech systems of today.

The Python programming language is fast growing in popularity, for a good reason. It gives the programmer flexibility, it has many modules to perform different tasks, and Python code is usually more readable and concise than in any other languages. There is a large and an active community of researchers, practitioners, and beginners using Python for data mining.

In this chapter, we will introduce data mining with Python. We will cover the following topics

  • What is data mining and where can we use it?
  • Setting up a Python-based environment to perform data mining
  • An example of affinity analysis, recommending products based on purchasing habits
  • An example of (a classic) classification problem, predicting the plant species based on its measurement

Introducing data mining

Data mining provides a way for a computer to learn how to make decisions with data. This decision could be predicting tomorrow's weather, blocking a spam email from entering your inbox, detecting the language of a website, or finding a new romance on a dating site. There are many different applications of data mining, with new applications being discovered all the time.


Data mining is part algorithm design, statistics, engineering, optimization, and computer science. However, combined with these base skills in the area, we also need to apply domain knowledge (expert knowledge)of the area we are applying the data mining. Domain knowledge is critical for going from good results to great results. Applying data mining effectively usually requires this domain-specific knowledge to be integrated with the algorithms.

Most data mining applications work with the same high-level view, where a model learns from some data and is applied to other data, although the details often change quite considerably.

Data mining applications involve creating data sets and tuning the algorithm as explained in the following steps

  1. We start our data mining process by creating a dataset, describing an aspect of the real world. Datasets comprise of the following two aspects:
  • Samples: These are objects in the real world, such as a book, photograph, animal, person, or any other object. Samples are also referred to as observations, records or rows, among other naming conventions.
  • Features: These are descriptions or measurements of the samples in our dataset. Features could be the length, frequency of a specific word, the number of legs on an animal, date it was created, and so on. Features are also referred to as variables, columns, attributes or covariant, again among other naming conventions.
    1. The next step is tuning the data mining algorithm. Each data mining algorithm has parameters, either within the algorithm or supplied by the user. This tuning allows the algorithm to learn how to make decisions about the data.

    As a simple example, we may wish the computer to be able to categorize people as short or tall. We start by collecting our dataset, which includes the heights of different people and whether they are considered short or tall:



    Short or tall?













    As explained above, the next step involves tuning the parameters of our algorithm. As a simple algorithm; if the height is more than x, the person is tall. Otherwise, they are short. Our training algorithms will then look at the data and decide on a good value for x. For the preceding data, a reasonable value for this threshold would be 170 cm. A person taller than 170 cm is considered tall by the algorithm. Anyone else is considered short by this measure. This then lets our algorithm classify new data, such as a person with height 167 cm, even though we may have never seen a person with those measurements before.

    In the preceding data, we had an obvious feature type. We wanted to know if people are short or tall, so we collected their heights. This feature engineering is a critical problem in data mining. In later chapters, we will discuss methods for choosing good features to collect in your dataset. Ultimately, this step often requires some expert domain knowledge or at least some trial and error.

    In this book, we will introduce data mining through Python. In some cases, we choose clarity of code and workflows, rather than the most optimized way to perform every task. This clarity sometimes involves skipping some details that can improve the algorithm's speed or effectiveness.


    Using Python and the Jupyter Notebook

    In this section, we will cover installing Python and the environment that we will use for most of the book, the Jupyter Notebook. Furthermore, we will install the NumPy module, which we will use for the first set of examples.


    The Jupyter Notebook was, until very recently, called the IPython Notebook. You'll notice the term in web searches for the project. Jupyter is the new name, representing a broadening of the project beyond using just Python.

    Installing Python

    The Python programming language is a fantastic, versatile, and an easy to use language.

    For this book, we will be using Python 3.5, which is available for your system from the Python Organization's website https://www.python.org/downloads/. However, I recommend that you use Anaconda to install Python, which you can download from the official website at https://www.continuum.io/downloads.


    There will be two major versions to choose from, Python 3.5 and Python 2.7. Remember to download and install Python 3.5, which is the version tested throughout this book. Follow the installation instructions on that website for your system. If you have a strong reason to learn version 2 of Python, then do so by downloading the Python 2.7 version. Keep in mind that some code may not work as in the book, and some workarounds may be needed.

    In this book, I assume that you have some knowledge of programming and Python itself. You do not need to be an expert with Python to complete this book, although a good level of knowledge will help. I will not be explaining general code structures and syntax in this book, except where it is different from what is considered normal python coding practice.

    If you do not have any experience with programming, I recommend that you pick up the Learning Python book from Packt Publishing, or the book Dive Into Python, available online at www.diveintopython3.net

    The Python organization also maintains a list of two online tutorials for those new to Python:

    • For non-programmers who want to learn to program through the Python language:


    • For programmers who already know how to program, but need to learn Python specifically:

                    https://wiki.python.org/moin/BeginnersGuide/ProgrammersWindows users will need to set an environment variable to use Python from the command line, where other systems will usually be immediately executable. We set it in the following steps

    1. First, find where you install Python 3 onto your computer; the default location is C:\Python35.
    2. Next, enter this command into the command line (cmd program): set the environment to PYTHONPATH=%PYTHONPATH%;C:\Python35.


    Remember to change the C:\Python35 if your installation of Python is in a different folder.

    Once you have Python running on your system, you should be able to open a command prompt and can run the following code to be sure it has installed correctly.

        $ python
        Python 3.5.1 (default, Apr 11 2014, 13:05:11)
        [GCC 4.8.2] on Linux
        Type "help", "copyright", "credits" or "license" for more 
        >>> print("Hello, world!")
    Hello, world!
        >>> exit()

    Note that we will be using the dollar sign ($) to denote that a command that you type into the terminal (also called a shell or cmd on Windows). You do not need to type this character (or retype anything that already appears on your screen). Just type in the rest of the line and press Enter.

    After you have the above "Hello, world!" example running, exit the program and move on to installing a more advanced environment to run Python code, the Jupyter Notebook.


    Python 3.5 will include a program called pip, which is a package manager that helps to install new libraries on your system. You can verify that pip is working on your system by running the $ pip freeze command, which tells you which packages you have installed on your system. Anaconda also installs their package manager, conda, that you can use. If unsure, use conda first, use pip only if that fails.

    Installing Jupyter Notebook

    Jupyter is a platform for Python development that contains some tools and environments for running Python and has more features than the standard interpreter. It contains the powerful Jupyter Notebook, which allows you to write programs in a web browser. It also formats your code, shows output, and allows you to annotate your scripts. It is a great tool for exploring datasets and we will be using it as our main environment for the code in this book.

    To install the Jupyter Notebook on your computer, you can type the following into a command line prompt (not into Python):

    $ conda install jupyter notebook

    You will not need administrator privileges to install this, as Anaconda keeps packages in the user's directory.

    With the Jupyter Notebook installed, you can launch it with the following:

    $ jupyter notebook

    Running this command will do two things. First, it will create a Jupyter Notebook instance - the backend - that will run in the command prompt you just used. Second, it will launch your web browser and connect to this instance, allowing you to create a new notebook. It will look something like the following screenshot (where you need to replace /home/bob with your current working directory):

    To stop the Jupyter Notebook from running, open the command prompt that has the instance running (the one you used earlier to run the jupyter notebook   command). Then, press Ctrl + C and you will be prompted Shutdown this notebook server (y/[n])?. Type y and press Enter and the Jupyter Notebook will shut down.

    Installing scikit-learn

    The scikit-learn package is a machine learning library, written in Python (but also containing code in other languages). It contains numerous algorithms, datasets, utilities, and frameworks for performing machine learning. Scikit-learnis built upon the scientific python stack, including libraries such as the NumPy and SciPy for speed. Scikit-learn is fast and scalable in many instances and useful for all skill ranges from beginners to advanced research users. We will cover more details of scikit-learn in Chapter 2, Classifying with scikit-learn Estimators.

    To install scikit-learn, you can use the conda utility that comes with Python 3, which will also install the NumPy and SciPy libraries if you do not already have them. Open a terminal with administrator/root privileges and enter the following command:

    $ conda install scikit-learn

    Users of major Linux distributions such as Ubuntu or Red Hat may wish to install the official package from their package manager.


    Not all distributions have the latest versions of scikit-learn, so check the version before installing it. The minimum version needed for this book is 0.14. My recommendation for this book is to use Anaconda to manage this for you, rather than installing using your system's package manager.

    Those wishing to install the latest version by compiling the source, or view more detailed installation instructions, can go to http://scikit-learn.org/stable/install.html and refer the official documentation on installing scikit-learn.



    A simple affinity analysis example

    In this section, we jump into our first example. A common use case for data mining is to improve sales, by asking a customer who is buying a product if he/she would like another similar product as well. You can perform this analysis through affinity analysis, which is the study of when things exist together, namely. correlate to each other.

    To repeat the now-infamous phrase taught in statistics classes, correlation is not causation. This phrase means that the results from affinity analysis cannot give a cause. In our next example, we perform affinity analysis on product purchases. The results indicate that the products are purchased together, but not that buying one product causes the purchase of the other. The distinction is important, critically so when determining how to use the results to affect a business process, for instance.

    What is affinity analysis?

    Affinity analysis is a type of data mining that gives similarity between samples (objects). This could be the similarity between the following:

    • Users on a website, to provide varied services or targeted advertising
    • Items to sell to those users, to provide recommended movies or products
    • Human genes, to find people that share the same ancestors

    We can measure affinity in several ways. For instance, we can record how frequently two products are purchased together. We can also record the accuracy of the statement when a person buys object 1 and when they buy object 2. Other ways to measure affinity include computing the similarity between samples, which we will cover in later chapters.


    Product recommendations

    One of the issues with moving a traditional business online, such as commerce, is that tasks that used to be done by humans need to be automated for the online business to scale and compete with existing automated businesses. One example of this is up-selling, or selling an extra item to a customer who is already buying. Automated product recommendations through data mining are one of the driving forces behind the e-commerce revolution that is turning billions of dollars per year into revenue.

    In this example, we are going to focus on a basic product recommendation service. We design this based on the following idea: when two items are historically purchased together, they are more likely to be purchased together in the future. This sort of thinking is behind many product recommendation services, in both online and offline businesses.

    A very simple algorithm for this type of product recommendation algorithm is to simply find any historical case where a user has brought an item and to recommend other items that the historical user brought. In practice, simple algorithms such as this can do well, at least better than choosing random items to recommend. However, they can be improved upon significantly, which is where data mining comes in.

    To simplify the coding, we will consider only two items at a time. As an example, people may buy bread and milk at the same time at the supermarket. In this early example, we wish to find simple rules of the form:

    If a person buys product X, then they are likely to purchase product Y

    More complex rules involving multiple items will not be covered such as people buying sausages and burgers being more likely to buy tomato sauce.

    Loading the dataset with NumPy

    The dataset can be downloaded from the code package supplied with the book, or from the official GitHub repository at:  https://github.com/dataPipelineAU/LearningDataMiningWithPython2 Download this file and save it on your computer, noting the path to the dataset. It is easiest to put it in the directory you'll run your code from, but we can load the dataset from anywhere on your computer.

    For this example, I recommend that you create a new folder on your computer to store your dataset and code. From here, open your Jupyter Notebook, navigate to this folder, and create a new notebook.

    The dataset we are going to use for this example is a NumPy two-dimensional array, which is a format that underlies most of the examples in the rest of the book. The array looks like a table, with rows representing different samples and columns representing different features.

    The cells represent the value of a specific feature of a specific sample. To illustrate, we can load the dataset with the following code:

    import numpy as np 
    dataset_filename = "affinity_dataset.txt" 
    X = np.loadtxt(dataset_filename)

    Enter the previous code into the first cell of your (Jupyter) Notebook. You can then run the code by pressing Shift + Enter (which will also add a new cell for the next section of code). After the code is run, the square brackets to the left-hand side of the first cell will be assigned an incrementing number, letting you know that this cell has completed. The first cell should look like the following:


    For code that will take more time to run, an asterisk will be placed here to denote that this code is either running or scheduled to run. This asterisk will be replaced by a number when the code has completed running (including if the code completes because it failed).

    This dataset has 100 samples and five features, which we will need to know for the later code. Let's extract those values using the following code:

    n_samples, n_features = X.shape

    If you choose to store the dataset somewhere other than the directory your Jupyter Notebooks are in, you will need to change the dataset_filename value to the new location.

    Next, we can show some of the rows of the dataset to get an understanding of the data. Enter the following line of code into the next cell and run it, to print the first five lines of the dataset:


    The result will show you which items were bought in the first five transactions listed:

    [[ 0.  1.  0.  0.  0.] 
     [ 1.  1.  0.  0.  0.] 
     [ 0.  0.  1.  0.  1.] 
     [ 1.  1.  0.  0.  0.] 
     [ 0.  0.  1.  1.  1.]]

    Downloading the example code

    You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you could visit http://www.packtpub.com/support and register to have the files e-mailed directly to you. I've also setup a GitHub repository that contains a live version of the code, along with new fixes, updates and so on. You can retrieve the code and datasets at the repository here: https://github.com/dataPipelineAU/LearningDataMiningWithPython2

    You can read the dataset can by looking at each row (horizontal line) at a time. The first row (0, 1, 0, 0, 0) shows the items purchased in the first transaction. Each column (vertical row) represents each of the items. They are bread, milk, cheese, apples, and bananas, respectively. Therefore, in the first transaction, the person bought cheese, apples, and bananas, but not bread or milk. Add the following line in a new cell to allow us to turn these feature numbers into actual words:

    features = ["bread", "milk", "cheese", "apples", "bananas"]

    Each of these features contains binary values, stating only whether the items were purchased and not how many of them were purchased. A1 indicates that at least 1 item was bought of this type, while a 0 indicates that absolutely none of that item was purchased. For a real world dataset, using exact figures or a larger threshold would be required.

    Implementing a simple ranking of rules

    We wish to find rules of the type If a person buys product X, then they are likely to purchase product Y. We can quite easily create a list of all the rules in our dataset by simply finding all occasions when two products are purchased together. However, we then need a way to determine good rules from bad ones allowing us to choose specific products to recommend.

    We can evaluate rules of this type in many ways, on which we will focus on two: support and confidence.

    Support is the number of times that a rule occurs in a dataset, which is computed by simply counting the number of samples for which the rule is valid. It can sometimes be normalized by dividing by the total number of times the premise of the rule is valid, but we will simply count the total for this implementation.


    The premise is the requirements for a rule to be considered active. The conclusion is the output of the rule. For the example if a person buys an apple, they also buy a banana, the rule is only valid if the premise happens - a person buys an apple. The rule's conclusion then states that the person will buy a banana.

    While the support measures how often a rule exists, confidence measures how accurate they are when they can be used. You can compute this by determining the percentage of times the rule applies when the premise applies. We first count how many times a rule applies to our data and divide it by the number of samples where the premise (the if statement) occurs.

    As an example, we will compute the support and confidence for the rule if a person buys apples, they also buy bananas.

    As the following example shows, we can tell whether someone bought apples in a transaction, by checking the value of sample[3], where we assign a sample to a row of our matrix:

    sample = X[2]

    Similarly, we can check if bananas were bought in a transaction by seeing if the value of sample[4] is equal to 1 (and so on). We can now compute the number of times our rule exists in our dataset and, from that, the confidence and support.

    Now we need to compute these statistics for all rules in our database. We will do this by creating a dictionary for both valid rules and invalid rules. The key to this dictionary will be a tuple (premise and conclusion). We will store the indices, rather than the actual feature names. Therefore, we would store (3 and 4) to signify the previous rule If a person buys apples, they will also buy bananas. If the premise and conclusion are given, the rule is considered valid. While if the premise is given but the conclusion is not, the rule is considered invalid for that sample.

    The following steps will help us to compute the confidence and support for all possible rules:

    1. We first set up some dictionaries to store the results. We will use defaultdict for this, which sets a default value if a key is accessed that doesn't yet exist. We record the number of valid rules, invalid rules, and occurrences of each premise:
    from collections import defaultdict 
    valid_rules = defaultdict(int) 
    invalid_rules = defaultdict(int) 
    num_occurences = defaultdict(int)
    1. Next, we compute these values in a large loop. We iterate over each sample in the dataset and then loop over each feature as a premise. When again loop over each feature as a possible conclusion, mapping the relationship premise to conclusion. If the sample contains a person who bought the premise and the conclusion, we record this in valid_rules. If they did not purchase the conclusion product, we record this in invalid_rules.
    2. For sample in X:
    for sample in X:
        for premise in range(n_features):
        if sample[premise] == 0: continue
    # Record that the premise was bought in another transaction
        num_occurences[premise] += 1
        for conclusion in range(n_features):
        if premise == conclusion: 
    # It makes little sense to
        measure if X -> X.
        if sample[conclusion] == 1:
    # This person also bought the conclusion item
        valid_rules[(premise, conclusion)] += 1

    If the premise is valid for this sample (it has a value of 1), then we record this and check each conclusion of our rule. We skip over any conclusion that is the same as the premise-this would give us rules such as: if a person buys Apples, then they buy Apples, which obviously doesn't help us much.

    We have now completed computing the necessary statistics and can now compute the support and confidence for each rule. As before, the support is simply our valid_rules value:

    support = valid_rules

    We can compute the confidence in the same way, but we must loop over each rule to compute this:

    confidence = defaultdict(float)
    for premise, conclusion in valid_rules.keys():
        rule = (premise, conclusion)
        confidence[rule] = valid_rules[rule] / num_occurences [premise]

    We now have a dictionary with the support and confidence for each rule. We can create a function that will print out the rules in a readable format. The signature of the rule takes the premise and conclusion indices, the support and confidence dictionaries we just computed, and the features array that tells us what the features mean. Then we print out the Support and Confidence of this rule:

    for premise, conclusion in confidence:
        premise_name = features[premise]
        conclusion_name = features[conclusion]
        print("Rule: If a person buys {0} they will also 
              buy{1}".format(premise_name, conclusion_name))
        print(" - Confidence: {0:.3f}".format
        print(" - Support: {0}".format(support

    We can test the code by calling it in the following way-feel free to experiment with different premises and conclusions:

    for premise, conclusion in confidence:
        premise_name = features[premise]
        conclusion_name = features[conclusion]
        print("Rule: If a person buys {0} they will also 
              buy{1}".format(premise_name, conclusion_name))
        print(" - Confidence: {0:.3f}".format
        print(" - Support: {0}".format(support

    Ranking to find the best rules

    Now that we can compute the support and confidence of all rules, we want to be able to find the best rules. To do this, we perform a ranking and print the ones with the highest values. We can do this for both the support and confidence values.

    To find the rules with the highest support, we first sort the support dictionary. Dictionaries do not support ordering by default; the items() function gives us a list containing the data in the dictionary. We can sort this list using the itemgetter class as our key, which allows for the sorting of nested lists such as this one. Using itemgetter(1) allows us to sort based on the values. Setting reverse=True gives us the highest values first:

    from operator import itemgetter 
    sorted_support = sorted(support.items(), key=itemgetter(1), reverse=True)

    We can then print out the top five rules:

    sorted_confidence = sorted(confidence.items(), key=itemgetter(1),
    for index in range(5):
        print("Rule #{0}".format(index + 1))
        premise, conclusion = sorted_confidence[index][0]
        print_rule(premise, conclusion, support, confidence, features)

    The result will look like the following:

    Rule #1 
    Rule: If a person buys bananas they will also buy milk 
     - Support: 27 
     - Confidence: 0.474 
    Rule #2 
    Rule: If a person buys milk they will also buy bananas 
     - Support: 27 
     - Confidence: 0.519 
    Rule #3 
    Rule: If a person buys bananas they will also buy apples 
     - Support: 27 
     - Confidence: 0.474 
    Rule #4 
    Rule: If a person buys apples they will also buy bananas 
     - Support: 27 
     - Confidence: 0.628 
    Rule #5 
    Rule: If a person buys apples they will also buy cheese 
     - Support: 22 
     - Confidence: 0.512

    Similarly, we can print the top rules based on confidence. First, compute the sorted confidence list and then print them out using the same method as before.

    sorted_confidence = sorted(confidence.items(), key=itemgetter(1),
    for index in range(5):
        print("Rule #{0}".format(index + 1))
        premise, conclusion = sorted_confidence[index][0]
        print_rule(premise, conclusion, support, confidence, features)

    Two rules are near the top of both lists. The first is If a person buys apples, they will also buy cheese, and the second is If a person buys cheese, they will also buy bananas. A store manager can use rules like these to organize their store. For example, if apples are on sale this week, put a display of cheeses nearby. Similarly, it would make little sense to put both bananas on sale at the same time as cheese, as nearly 66 percent of people buying cheese will probably buy bananas -our sale won't increase banana purchases all that much.


    Jupyter Notebook will display graphs inline, right in the notebook. Sometimes, however, this is not always configured by default. To configure Jupyter Notebook to display graphs inline, use the following line of code: %matplotlib inline

    We can visualize the results using a library called matplotlib.

    We are going to start with a simple line plot showing the confidence values of the rules, in order of confidence. matplotlib makes this easy - we just pass in the numbers, and it will draw up a simple but effective plot:

    from matplotlib import pyplot as plt 
    plt.plot([confidence[rule[0]] for rule in sorted_confidence])

    Using the previous graph, we can see that the first five rules have decent confidence, but the efficacy drops quite quickly after that. Using this information, we might decide to use just the first five rules to drive business decisions. Ultimately with exploration techniques like this, the result is up to the user.

    Data mining has great exploratory power in examples like this. A person can use data mining techniques to explore relationships within their datasets to find new insights. In the next section, we will use data mining for a different purpose: prediction and classification.


    A simple classification example

    In the affinity analysis example, we looked for correlations between different variables in our dataset. In classification, we have a single variable that we are interested in and that we call the class (also called the target). In the earlier example, if we were interested in how to make people buy more apples, we would explore the rules related to apples and use those to inform our decisions.



    What is classification?

    Classification is one of the largest uses of data mining, both in practical use and in research. As before, we have a set of samples that represents objects or things we are interested in classifying. We also have a new array, the class values. These class values give us a categorization of the samples. Some examples are as follows:

    • Determining the species of a plant by looking at its measurements. The class value here would be: Which species is this?
    • Determining if an image contains a dog. The class would be: Is there a dog in this image?
    • Determining if a patient has cancer, based on the results of a specific test. The class would be: Does this patient have cancer?

    While many of the examples previous are binary (yes/no) questions, they do not have to be, as in the case of plant species classification in this section.


    The goal of classification applications is to train a model on a set of samples with known classes and then apply that model to new unseen samples with unknown classes. For example, we want to train a spam classifier on my past e-mails, which I have labeled as spam or not spam. I then want to use that classifier to determine whether my next email is spam, without me needing to classify it myself.

    Loading and preparing the dataset

    The dataset we are going to use for this example is the famous Iris database of plant classification. In this dataset, we have 150 plant samples and four measurements of each: sepal length, sepal width, petal length, and petal width (all in centimeters). This classic dataset (first used in 1936!) is one of the classic datasets for data mining. There are three classes: Iris Setosa, Iris Versicolour, and Iris Virginica. The aim is to determine which type of plant a sample is, by examining its measurements.

    The scikit-learn library contains this dataset built-in, making the loading of the dataset straightforward:

    from sklearn.datasets import load_iris 
    dataset = load_iris() 
    X = dataset.data 
    y = dataset.target

    You can also print(dataset.DESCR) to see an outline of the dataset, including some details about the features.

    The features in this dataset are continuous values, meaning they can take any range of values. Measurements are a good example of this type of feature, where a measurement can take the value of 1, 1.2, or 1.25 and so on. Another aspect of continuous features is that feature values that are close to each other indicate similarity. A plant with a sepal length of 1.2 cm is like a plant with a Sepal width of 1.25 cm.

    In contrast are categorical features. These features, while often represented as numbers, cannot be compared in the same way. In the Iris dataset, the class values are an example of a categorical feature. The class 0 represents Iris Setosa; class 1 represents Iris Versicolour, and class 2 represents Iris Virginica. The numbering doesn't mean that Iris Setosa is more similar to Iris Versicolour than it is to Iris Virginica-despite the class value being more similar. The numbers here represent categories. All we can say is whether categories are the same or different.

    There are other types of features too, which we will cover in later chapters. These include pixel intensity, word frequency and n-gram analysis.

    While the features in this dataset are continuous, the algorithm we will use in this example requires categorical features. Turning a continuous feature into a categorical feature is a process called discretization.

    A simple discretization algorithm is to choose some threshold, and any values below this threshold are given a value 0. Meanwhile, any above this are given the value 1. For our threshold, we will compute the mean (average) value for that feature. To start with, we compute the mean for each feature:

    attribute_means = X.mean(axis=0)

    The result from this code will be an array of length 4, which is the number of features we have. The first value is the mean of the values for the first feature and so on. Next, we use this to transform our dataset from one with continuous features to one with discrete categorical features:

    assert attribute_means.shape == (n_features,)
    X_d = np.array(X >= attribute_means, dtype='int')

    We will use this new X_d dataset (for X discretized) for our training and testing, rather than the original dataset (X).


    Implementing the OneR algorithm

    OneR is a simple algorithm that simply predicts the class of a sample by finding the most frequent class for the feature values. OneR is shorthand for One Rule, indicating we only use a single rule for this classification by choosing the feature with the best performance. While some of the later algorithms are significantly more complex, this simple algorithm has been shown to have good performance in some real-world datasets.

    The algorithm starts by iterating over every value of every feature. For that value, count the number of samples from each class that has that feature value. Record the most frequent class of the feature value, and the error of that prediction.

    For example, if a feature has two values, 0 and 1, we first check all samples that have the value 0. For that value, we may have 20 in Class A, 60 in Class B, and a further 20 in Class C. The most frequent class for this value is B, and there are 40 instances that have different classes. The prediction for this feature value is B with an error of 40, as there are 40 samples that have a different class from the prediction. We then do the same procedure for the value 1 for this feature, and then for all other feature value combinations.

    Once these combinations are computed, we compute the error for each feature by summing up the errors for all values for that feature. The feature with the lowest total error is chosen as the One Rule and then used to classify other instances.

    In code, we will first create a function that computes the class prediction and error for a specific feature value. We have two necessary imports, defaultdict and itemgetter, that we used in earlier code:

    from collections import defaultdict 
    from operator import itemgetter

    Next, we create the function definition, which needs the dataset, classes, the index of the feature we are interested in, and the value we are computing. It loops over each sample, and counts the number of time each feature value corresponds to a specific class. We then choose the most frequent class for the current feature/value pair:

    def train_feature_value(X, y_true, feature, value):
    # Create a simple dictionary to count how frequency they give certain
     class_counts = defaultdict(int)
    # Iterate through each sample and count the frequency of each
    class/value pair
     for sample, y in zip(X, y_true):
        if sample[feature] == value: 
            class_counts[y] += 1
    # Now get the best one by sorting (highest first) and choosing the
    first item
    sorted_class_counts = sorted(class_counts.items(), key=itemgetter(1),
    most_frequent_class = sorted_class_counts[0][0]
     # The error is the number of samples that do not classify as the most
    frequent class
     # *and* have the feature value.
        n_samples = X.shape[1]
        error = sum([class_count for class_value, class_count in
     if class_value != most_frequent_class])
        return most_frequent_class, error

    As a final step, we also compute the error of this rule. In the OneR algorithm, any sample with this feature value would be predicted as being the most frequent class. Therefore, we compute the error by summing up the counts for the other classes (not the most frequent). These represent training samples that result in error or an incorrect classification.

    With this function, we can now compute the error for an entire feature by looping over all the values for that feature, summing the errors, and recording the predicted classes for each value.

    The function needs the dataset, classes, and feature index we are interested in. It then iterates through the different values and finds the most accurate feature value to use for this specific feature, as the rule in OneR:

    def train(X, y_true, feature): 
        # Check that variable is a valid number 
        n_samples, n_features = X.shape 
        assert 0 <= feature < n_features 
        # Get all of the unique values that this variable has 
        values = set(X[:,feature]) 
        # Stores the predictors array that is returned 
        predictors = dict() 
        errors = [] 
        for current_value in values: 
            most_frequent_class, error = train_feature_value
            (X, y_true, feature, current_value) 
            predictors[current_value] = most_frequent_class 
        # Compute the total error of using this feature to classify on 
        total_error = sum(errors) 
        return predictors, total_error

    Let's have a look at this function in a little more detail.

    After some initial tests, we then find all the unique values that the given feature takes. The indexing in the next line looks at the whole column for the given feature and returns it as an array. We then use the set function to find only the unique values:

        values = set(X[:,feature_index])

    Next, we create our dictionary that will store the predictors. This dictionary will have feature values as the keys and classification as the value. An entry with key 1.5 and value 2 would mean that, when the feature has a value set to 1.5, classify it as belonging to class 2. We also create a list storing the errors for each feature value:

    predictors = {} 
        errors = []

    As the main section of this function, we iterate over all the unique values for this feature and use our previously defined train_feature_value function to find the most frequent class and the error for a given feature value. We store the results as outlined earlier:

    Finally, we compute the total errors of this rule and return the predictors along with this value:

    total_error = sum(errors)
    return predictors, total_error

    Testing the algorithm

    When we evaluated the affinity analysis algorithm of the earlier section, our aim was to explore the current dataset. With this classification, our problem is different. We want to build a model that will allow us to classify previously unseen samples by comparing them to what we know about the problem.

    For this reason, we split our machine-learning workflow into two stages: training and testing. In training, we take a portion of the dataset and create our model. In testing, we apply that model and evaluate how effectively it worked on the dataset. As our goal is to create a model that can classify previously unseen samples, we cannot use our testing data for training the model. If we do, we run the risk of overfitting.

    Overfitting is the problem of creating a model that classifies our training dataset very well but performs poorly on new samples. The solution is quite simple: never use training data to test your algorithm. This simple rule has some complex variants, which we will cover in later chapters; but, for now, we can evaluate our OneR implementation by simply splitting our dataset into two small datasets: a training one and a testing one. This workflow is given in this section.

    The scikit-learn library contains a function to split data into training and testing components:

    from sklearn.cross_validation import train_test_split

    This function will split the dataset into two sub-datasets, per a given ratio (which by default uses 25 percent of the dataset for testing). It does this randomly, which improves the confidence that the algorithm will perform as expected in real world environments (where we expect data to come in from a random distribution):

    Xd_train, Xd_test, y_train, y_test = train_test_split(X_d, y, 

    We now have two smaller datasets: Xd_train contains our data for training and Xd_test contains our data for testing. y_train and y_test give the corresponding class values for these datasets.

    We also specify a random_state. Setting the random state will give the same split every time the same value is entered. It will look random, but the algorithm used is deterministic, and the output will be consistent. For this book, I recommend setting the random state to the same value that I do, as it will give you the same results that I get, allowing you to verify your results. To get truly random results that change every time you run it, set random_state to None.

    Next, we compute the predictors for all the features for our dataset. Remember to only use the training data for this process. We iterate over all the features in the dataset and use our previously defined functions to train the predictors and compute the errors:

    all_predictors = {} 
    errors = {} 
    for feature_index in range(Xd_train.shape[1]): 
        predictors, total_error = train(Xd_train,
        all_predictors[feature_index] = predictors 
        errors[feature_index] = total_error

    Next, we find the best feature to use as our One Rule, by finding the feature with the lowest error:

    best_feature, best_error = sorted(errors.items(), key=itemgetter(1))[0]

    We then create our model by storing the predictors for the best feature:

    model = {'feature': best_feature,
             'predictor': all_predictors[best_feature]}

    Our model is a dictionary that tells us which feature to use for our One Rule and the predictions that are made based on the values it has. Given this model, we can predict the class of a previously unseen sample by finding the value of the specific feature and using the appropriate predictor. The following code does this for a given sample:

    variable = model['feature'] 
    predictor = model['predictor'] 
    prediction = predictor[int(sample[variable])]

    Often we want to predict several new samples at one time, which we can do using the following function. It simply uses the above code, but iterate over all the samples in a dataset, obtaining the prediction for each sample:

    def predict(X_test, model):
    variable = model['feature']
    predictor = model['predictor']
    y_predicted = np.array([predictor
                            [int(sample[variable])] for sample
                            in X_test])
    return y_predicted

    For our testing dataset, we get the predictions by calling the following function:

    y_predicted = predict(Xd_test, model)

    We can then compute the accuracy of this by comparing it to the known classes:

    accuracy = np.mean(y_predicted == y_test) * 100 
    print("The test accuracy is {:.1f}%".format(accuracy))

    This algorithm gives an accuracy of 65.8 percent, which is not bad for a single rule!




    In this chapter, we introduced data mining using Python. If you could run the code in this section (note that the full code is available in the supplied code package), then your computer is set up for much of the rest of the book. Other Python libraries will be introduced in later chapters to perform more specialized tasks.

    We used the Jupyter Notebook to run our code, which allows us to immediately view the results of a small section of the code. Jupyter Notebook is a useful tool that will be used throughout the book.

    We introduced a simple affinity analysis, finding products that are purchased together. This type of exploratory analysis gives an insight into a business process, an environment, or a scenario. The information from these types of analysis can assist in business processes, find the next big medical breakthrough, or create the next artificial intelligence.

    Also, in this chapter, there was a simple classification example using the OneR algorithm. This simple algorithm simply finds the best feature and predicts the class that most frequently had this value in the training dataset.

    To expand on the outcomes of this chapter, think about how you would implement a variant of OneR that can take multiple feature/value pairs into consideration. Take a shot at implementing your new algorithm and evaluating it. Remember to test your algorithm on a separate dataset to the training data. Otherwise, you run the risk of over fitting your data.

    Over the next few chapters, we will expand on the concepts of classification and affinity analysis. We will also introduce classifiers in the scikit-learn package and use them to do our machine learning, rather than writing the algorithms ourselves.

    About the Author
    • Robert Layton

      Robert Layton is a data scientist investigating data-driven applications to businesses across a number of sectors. He received a PhD investigating cybercrime analytics from the Internet Commerce Security Laboratory at Federation University Australia, before moving into industry, starting his own data analytics company dataPipeline. Next, he created Eureaktive, which works with tech-based startups on developing their proof-of-concepts and early-stage prototypes. Robert also runs the LearningTensorFlow website, which is one of the world's premier tutorial websites for Google's TensorFlow library.

      Robert is an active member of the Python community, having used Python for more than 8 years. He has presented at PyConAU for the last four years and works with Python Charmers to provide Python-based training for businesses and professionals from a wide range of organisations.

      Robert can be best reached via Twitter @robertlayton

      Browse publications by this author
    Latest Reviews (1 reviews total)
    El contenido es de un nivel medio-alto, tal y como me esperaba. Los materiales de apoyo son de agradecer.
    Learning Data Mining with Python - Second Edition
    Unlock this book and the full library FREE for 7 days
    Start now