Machine Learning with R

Due to the relative youth of machine learning as a discipline and the speed at which it is progressing, the associated legal issues and social norms are often quite uncertain and constantly in flux. Caution should be exercised when obtaining or analyzing data in order to avoid breaking laws, violating terms of service or data use agreements, abusing the trust, or violating privacy of the customers or the public.

Certain jurisdictions may prevent you from using racial, ethnic, religious, or other protected class data for business reasons, but keep in mind that excluding this data from your analysis may not be enough—machine learning algorithms might inadvertently learn this information independently. For instance, if a certain segment of people generally live in a certain region, buy a certain product, or otherwise behave in a way that uniquely identifies them as a group, some machine learning algorithms can infer the protected information from seemingly innocuous data. In such cases, you may need to fully "de-identify" these people by excluding any potentially identifying data in addition to the protected information.

Apart from the legal consequences, using data inappropriately may hurt your bottom line. Customers may feel uncomfortable or become spooked if aspects of their lives they consider private are made public. Recently, several high-profile web applications have experienced a mass exodus of users who felt exploited when the applications' terms of service agreements changed and their data was used for purposes beyond what the users had originally agreed upon. The fact that privacy expectations differ by context, by age cohort, and by locale, adds complexity to deciding the appropriate use of personal data. It would be wise to consider the cultural implications of your work before you begin on your project.

Representing raw input data in a structured format is the quintessential task for a learning algorithm. Prior to this point, the data is merely ones and zeros on a disk or in memory; they have no meaning. The work of assigning a meaning to data occurs during the abstraction process.

The connection between ideas and reality is exemplified by the famous René Magritte painting The Treachery of Images shown as follows:

Source: http://collections.lacma.org/node/239578

The painting depicts a tobacco pipe with the caption Ceci n'est pas une pipe ("this is not a pipe"). The point Magritte was illustrating is that a representation of a pipe is not truly a pipe. In spite of the fact that the pipe is not real, anybody viewing the painting easily recognizes that the picture is a pipe, suggesting that observers' minds are able to connect the picture of a pipe to the idea of a pipe, which can then be connected to an actual pipe that could be held in the hand. Abstracted connections like this are the basis of knowledge representation, the formation of logical structures that assist with turning raw sensory information into a meaningful insight.

During the process of knowledge representation, the computer summarizes raw inputs in a model, an explicit description of the structured patterns among data. There are many different types of models. You may already be familiar with some. Examples include:

The choice of model is typically not left up to the machine. Instead, the model is dictated by the learning task and the type of data being analyzed. Later in this chapter, we will discuss methods for choosing the type of model in more detail.

The process of fitting a particular model to a dataset is known as training. Why is this not called learning? First, note that the learning process does not end with the step of data abstraction. Learning requires an additional step to generalize the knowledge to future data. Second, the term training more accurately describes the actual process undertaken when the model is fitted to the data. Learning implies a sort of inductive, bottom-up reasoning. Training better connotes the fact that the machine learning model is imposed by the human teacher onto the machine student, providing the computer with a structure it attempts to model after.

When the model has been trained, the data has been transformed into an abstract form that summarizes the original information. It is important to note that the model does not itself provide additional data, yet it is sometimes interesting on its own. How can this be? The reason is that by imposing an assumed structure on the underlying data, it gives insight into the unseen and provides a theory about how the data is related. Take for instance the discovery of gravity. By fitting equations to observational data, Sir Isaac Newton deduced the concept of gravity. But gravity was always present. It simply wasn't recognized as a concept until the model noted it in abstract terms—specifically, by becoming the g term in a model that explains observations of falling objects.

Most models will not result in the development of theories that shake up scientific thought for centuries. Still, your model might result in the discovery of previously unseen relationships among data. A model trained on genomic data might find several genes that when combined are responsible for the onset of diabetes; banks might discover a seemingly innocuous type of transaction that systematically appears prior to fraudulent activity; psychologists might identify a combination of characteristics indicating a new disorder. The underlying relationships were always present; but in conceptualizing the information in a different format, a model presents the connections in a new light.

Recall that the learning process is not complete until the learner is able to use its abstracted knowledge for future action. Yet an issue remains before the learner can proceed—there are countless underlying relationships that might be identified during the abstraction process and myriad ways to model these relationships. Unless the number of potential theories is limited, the learner will be unable to utilize the information. It would be stuck where it started, with a large pool of information but no actionable insight.

The term generalization describes the process of turning abstracted knowledge into a form that can be utilized for action. Generalization is a somewhat vague process that is a bit difficult to describe. Traditionally, it has been imagined as a search through the entire set of models (that is, theories) that could have been abstracted during training. Specifically, if you imagine a hypothetical set containing every possible theory that could be established from the data, generalization involves the reduction of this set into a manageable number of important findings.

Generally, it is not feasible to reduce the number of potential concepts by examining them one-by-one and determining which are the most useful. Instead, machine learning algorithms generally employ shortcuts that more quickly divide the set of concepts. Toward this end, the algorithm will employ heuristics, or educated guesses about the where to find the most important concepts.

Heuristics are routinely used by human beings to quickly generalize experience to new scenarios. If you have ever utilized gut instinct to make a snap decision prior to fully evaluating your circumstances, you were intuitively using mental heuristics.

For example, the availability heuristic is the tendency for people to estimate the likelihood of an event by how easily examples can be recalled. The availability heuristic might help explain the prevalence of the fear of airline travel relative to automobile travel, despite automobiles being statistically more dangerous. Accidents involving air travel are highly publicized and traumatic events, and are likely to be very easily recalled, whereas car accidents barely warrant a mention in the newspaper.

The preceding example illustrates the potential for heuristics to result in illogical conclusions. Browsing a list of common logical fallacies, one is likely to note many that seem rooted in heuristic-based thinking. For instance, the gambler's fallacy, or the belief that a run of bad luck implies that a stretch of better luck is due, may be resultant from the application of the representativeness heuristic, which erroneously led the gambler to believe that all random sequences are balanced since most random sequences are balanced.

The folly of misapplied heuristics is not limited to human beings. The heuristics employed by machine learning algorithms also sometimes result in erroneous conclusions. If the conclusions are systematically imprecise, the algorithm is said to have a bias. For example, suppose that a machine learning algorithm learned to identify faces by finding two circles, or eyes, positioned side-by-side above a line for a mouth. The algorithm might then have trouble with, or be biased against faces that do not conform to its model. This may include faces with glasses, turned at an angle, looking sideways, or with darker skin tones. Similarly, it could be biased toward faces with lighter eye colors or other characteristics that do not conform to its understanding of the world.

In modern usage, the word bias has come to carry quite negative connotations. Various forms of media frequently claim to be free from bias, and claim to report the facts objectively, untainted by emotion. Still, consider for a moment the possibility that a little bias might be useful. Without a bit of arbitrariness, might it be a bit difficult to decide among several competing choices, each with distinct strengths and weaknesses? Indeed, some recent studies in the field of psychology have suggested that individuals born with damage to portions of the brain responsible for emotion are ineffectual at decision making, and might spend hours debating simple decisions such as what color shirt to wear or where to eat lunch. Paradoxically, bias is what blinds us from some information while also allowing us to utilize other information for action.

Bias is a necessary evil associated with the abstraction and generalization process inherent in any machine learning task. Every learner has its weaknesses and is biased in a particular way; there is no single model to rule them all. Therefore, the final step in the generalization process is to determine the model's success in spite of its biases.

After a model has been trained on an initial dataset, the model is tested on a new dataset, and judged on how well its characterization of the training data generalizes to the new data. It's worth noting that it is exceedingly rare for a model to perfectly generalize to every unforeseen case.

In part, the failure for models to perfectly generalize is due to the problem of noise, or unexplained variations in data. Noisy data is caused by seemingly random events, such as:

Trying to model the noise in data is the basis of a problem called overfitting. Because noise is unexplainable by definition, attempting to explain the noise will result in erroneous conclusions that do not generalize well to new cases. Attempting to generate theories to explain the noise also results in more complex models that are more likely to ignore the true pattern the learner is trying to identify. A model that seems to perform well during training but does poorly during testing is said to be overfitted to the training dataset as it does not generalize well.

Solutions to the problem of overfitting are specific to particular machine learning approaches. For now, the important point is to be aware of the issue. How well models are able to handle noisy data is an important source of distinction among them.

All machine learning algorithms require input training data. The exact format may differ, but in its most basic form, input data takes the form of examples and features.

An example is literally a single exemplary instance of the underlying concept to be learned; it is one set of data describing the atomic unit of interest for the analysis. If you were building a learning algorithm to identify spam e-mail, the examples would be data from many individual electronic messages. To detect cancerous tumors, the examples might comprise biopsies from a number of patients.

The phrase unit of observation is used to describe the units that the examples are measured in. Commonly, the unit of observation is in the form of transactions, persons, time points, geographic regions, or measurements. Other possibilities include combinations of these such as person years, which would denote cases where the same person is tracked over multiple time points.

A feature is a characteristic or attribute of an example, which might be useful for learning the desired concept. In the previous examples, attributes in the spam detection dataset might consist of the words used in the e-mail messages. For the cancer dataset, the attributes might be genomic data from the biopsied cells, or measured characteristics of the patient such as weight, height, or blood pressure.

The following spreadsheet shows a dataset in matrix format, which means that each example has the same number of features. In matrix data, each row in the spreadsheet is an example and each column is a feature. Here, the rows indicate examples of automobiles while the columns record various features of the cars such as the price, mileage, color, and transmission. Matrix format data is by far the most common form used in machine learning, though as you will see in later chapters, other forms are used occasionally in specialized cases.

Features come in various forms as well. If a feature represents a characteristic measured in numbers, it is unsurprisingly called numeric. Alternatively, if it measures an attribute that is represented by a set of categories, the feature is called categorical or nominal. A special case of categorical variables is called ordinal, which designates a nominal variable with categories falling in an ordered list. Some examples of ordinal variables include clothing sizes such as small, medium, and large, or a measurement of customer satisfaction on a scale from 1 to 5. It is important to consider what the features represent because the type and number of features in your dataset will assist with determining an appropriate machine learning algorithm for your task.

Machine learning algorithms can be divided into two main groups: supervised learners that are used to construct predictive models, and unsupervised learners that are used to build descriptive models. Which type you will need to use depends on the learning task you hope to accomplish.

A predictive model is used for tasks that involve, as the name implies, the prediction of one value using other values in the dataset. The learning algorithm attempts to discover and model the relationship among the target feature (the feature being predicted) and the other features. Despite the common use of the word "prediction" to imply forecasting predictive models need not necessarily foresee future events. For instance, a predictive model could be used to predict past events such as the date of a baby's conception using the mother's hormone levels; or, predictive models could be used in real time to control traffic lights during rush hours.

Because predictive models are given clear instruction on what they need to learn and how they are intended to learn it, the process of training a predictive model is known as supervised learning. The supervision does not refer to human involvement, but rather the fact that the target values provide a supervisory role, which indicates to the learner the task it needs to learn. Specifically, given a set of data, the learning algorithm attempts to optimize a function (the model) to find the combination of feature values that result in the target output.

The often used supervised machine learning task of predicting which category an example belongs to is known as classification. It is easy to think of potential uses for a classifier. For instance, you could predict whether:

The target feature to be predicted is a categorical feature known as the class and is divided into categories called levels. A class can have two or more levels, and the levels need not necessarily be ordinal. Because classification is so widely used in machine learning, there are many types of classification algorithms.

Supervised learners can also be used to predict numeric data such as income, laboratory values, test scores, or counts of items. To predict such numeric values, a common form of numeric prediction fits linear regression models to the input data. Although regression models are not the only type of numeric models, they are by far the most widely used. Regression methods are widely used for forecasting, as they quantify in exact terms the association between the inputs and the target, including both the magnitude and uncertainty of the relationship.

A descriptive model is used for tasks that would benefit from the insight gained from summarizing data in new and interesting ways. As opposed to predictive models that predict a target of interest; in a descriptive model, no single feature is more important than any other. In fact, because there is no target to learn, the process of training a descriptive model is called unsupervised learning. Although it can be more difficult to think of applications for descriptive models—after all, what good is a learner that isn't learning anything in particular—they are used quite regularly for data mining.

For example, the descriptive modeling task called pattern discovery is used to identify frequent associations within data. Pattern discovery is often used for market basket analysis on transactional purchase data. Here, the goal is to identify items that are frequently purchased together, such that the learned information can be used to refine the marketing tactics. For instance, if a retailer learns that swimming trunks are commonly purchased at the same time as sunscreen, the retailer might reposition the items more closely in the store, or run a promotion to "up-sell" customers on associated items.

The descriptive modeling task of dividing a dataset into homogeneous groups is called clustering. This is sometimes used for segmentation analysis that identifies groups of individuals with similar purchasing, donating, or demographic information so that advertising campaigns can be tailored to particular audiences. Although the machine is capable of identifying the groups, human intervention is required to interpret them. For example, given five different clusters of shoppers at a grocery store, the marketing team will need to understand the differences among the groups in order to create a promotion that best suits each group. However, this is almost certainly easier than trying to create a unique appeal for each customer.

The following table lists the general types of machine learning algorithms covered in this book, each of which may be implemented in several ways. Although this covers only some of the entire set of all machine learning algorithms, learning these methods will provide a sufficient foundation for making sense of other methods as you encounter them.

To match a learning task to a machine learning approach, you will need to begin with one of the four types of tasks: classification, numeric prediction, pattern detection, or clustering. Certain tasks make the choice of algorithm simpler. For instance, if you are undertaking pattern detection, you will likely employ association rules. Similarly, a clustering problem will likely utilize the k-means algorithm while numeric prediction will utilize regression analysis or regression trees.

For classification, more thought is needed to match a learning problem to an appropriate classifier. In these cases, it is helpful to consider the various distinctions among the algorithms. For instance, within classification problems, decision trees result in models that are readily understood, while the models of neural networks are notoriously difficult to interpret. If you were designing a credit-scoring model, this could be an important distinction because law often requires that the applicant must be notified about the reasons he or she was rejected for the loan. Even if the neural network was better at predicting loan defaults if the predictions cannot be explained, then it is useless.

In each chapter, the key strengths and weaknesses of each approach will be listed. Although you will sometimes find that these characteristics exclude certain models from consideration in most cases, the choice of model is arbitrary. In this case, feel free to use whichever algorithm you are most comfortable with. Other times, when predictive accuracy is primary, you may need to test several and choose the one that fits best. In later chapters, we will even look at methods of combining models that utilize the best properties of each.

Despite the vast set of available R add-ons, the package format makes installation and use a virtually effortless process. To demonstrate the use of packages, we will install and load the RWeka package, which was developed by Kurt Hornik, Christian Buchta, and Achim Zeileis (see Open-Source Machine Learning: R Meets Weka in Computational Statistics 24: 225-232 for more information). The RWeka package provides a collection of functions that give R access to the machine learning algorithms in the Java-based Weka software package by Ian H. Witten and Eibe Frank. For more information on Weka, see:

http://www.cs.waikato.ac.nz/~ml/weka/.

> install.packages("RWeka")

> install.packages("RWeka", lib="/path/to/library")

> ?install.packages

Installing a package using the point-and-click interface

As an alternative to typing the install.packages() command, R provides a graphical user interface (GUI) for package installation. On a Microsoft Windows system, this can be accessed from the Install package(s) command item under the Packages menu, as shown in the following screenshot. On Mac OS X, the command is labeled Package Installer and is located under the Packages & Data menu.

On Windows, after launching the package installer (and choosing a CRAN mirror location if you haven't already), a large list of packages will appear. Simply scroll to the RWeka package and click on the OK button to install the package and all dependencies to the default location.

On Mac OS X, the package installer menu provides additional options. To load the list of packages, click on the Get List button. Scroll to the RWeka package (or use the Package Search feature) and click on Install Selected. Note that by default, the Mac OS X Package Installer does not install dependencies unless the Install Dependencies checkbox is selected, as shown in the following screenshot:

> library(RWeka)