Chapter 5. How to Address a Data Mining Problem – Data Cleaning and Validation
This chapter is where our real journey begins (finally!, I can hear you exclaiming). We are now familiar enough with R and the data mining process and architecture to get involved with a real problem.
I say real problem I actually mean real, since we are going to face something that actually happened and that actually puzzled a non-trivial number of people in a real company. Of course, we are going to use randomized dataF here and fictitious names, nevertheless, this will not remove any pathos to the problem. We are shortly going to get immersed into some kind of mystery that actually came up, and we will need to solve it, employing data mining techniques.
I know you may be thinking: OK, don't make it too serious, is it something which actually already got solved?You would be right, but what if something similar pops up for you some day in the future? What would you do? The mystery we are going to face will not...
It's a nice day today, you get into your 6th-floor office at Hippalus, Wholesaler Inc, grab a coffee, and sit down at your desk. Suddenly, an email pops in your inbox: Urgent - profits drop. As soon as the meaning of those three words is clear to you, you realize that everybody in the office has received the same email and they are starting to chat about it, there will be a meeting with the head of the office in 15 minutes. When you get into the meeting room, still missing the coffee you left at your desk, a big chart is projected on the wall:
OK, you have just read Chapter 2, A First Primer on Data Mining - Analysing Your Banking Account Data so you know it is a line chart, and that we are looking at a time series, but no title is included so we can just observe the incredible drop at the end of the series. Whoa, those are our quarterly cash flows, says the astonished colleague sitting next to you.
How did we get that low?, replies back the one sitting on your other side. This...
First of all, we need to actually import the data to our R environment (oh yeah, I was taking for granted that we are going to use R for this, hope you do not mind).
We can leverage our old friend the rio package, running it on all of the three files we were provided, once we have unzipped them. Take a minute to figure out if you can remember the function needed to perform the task.
Done? OK, find the solution as follows:
cash_flow_report <- import("cash_flow.csv")
customer_list <- import("customer_list.txt")
stored_data <- import("stored_data.rds")Before actually looking at our data, we should define how we want it to be arranged in order to allow for future manipulation and analyses. Currently, one of the most adopted frameworks for data arrangement and handling is the so called tidy data framework. The concepts behind this framework were originally defined by Hadley Wickham, and nowadays come paired with a couple of R packages that help to apply it...
Our journey has begun within this chapter. Leveraging the knowledge gained within previous chapters, we have started facing a challenge that suddenly appeared: discover the origin of a heavy loss our company is suffering.
We received some dirty data to be cleaned, and this was the occasion to learn about data cleaning and tidy data. This was the first set of activities to make our data fit the analyses' needs, and the second a conceptual framework that can be employed to define which structure our data should have to fit those needs. We also learned how to evaluate the respect of the three main rules of tidy data (every row has a record, every column has an attribute, and every table is an entity).
We also learned about data quality and data validation, discovering which metrics define the level of quality of our data and a set of checks that can be employed to assess this quality and spot any needed improvements.
We applied all these concepts to our data, making our data through the...
