Data Science Process

Over the past decade, organizations have seen a rapid growth in data. Harnessing insight from that data is crucial to the growth and sustenance of these organizations. Yet, groups chartered with extracting value from data fail for various reasons. In this chapter, we will cover how organizations can avoid the potential pitfalls of data science.

There is a larger discussion about the quality and governance of data, which we will not be covering here. Experienced data scientists recognize the challenges with data and account for them in their processes. In general, some of these challenges include the following:

• Poor data quality and consistency
• Silos of data driven by individual business teams
• Technologies that are hard to integrate with other data sources
• The inability to deal with the Vs of big data: volume, velocity, variety, and veracity

In some cases...

TDSP stages

The Team Data Science Process (TDSP) is a methodology created by Microsoft to guide the full life cycle of data science projects in organizations. It is not meant to be a complete solution, but simply a framework by which teams can add structure to their processes and achieve the full business value of their analytics.

Besides TDSP, the other prevalent methodology that organizations have been adopting is called CRISP-DM (short for Cross-Industry Standard Process for Data Mining). This methodology has been around since the mid-1990s. There were several attempts to update it in the 2000s, but they were abandoned. The primary focus of CRISP-DM was data mining, but its principles can be extended to data science as well. The major steps listed in CRISP-DM are as follows: business understanding, data understanding, data preparation, modeling, evaluation, and deployment....

Tools for TDSP

Microsoft has released a set of tools that make it easier for organizations to follow the TDSP process. One of those tools is the IDEAR utility released for CRAN-R, Microsoft R, and Python. Another tool is the Automated Modeling and Reporting (AMAR) utility. In this section, we will look into how we can leverage these tools in the TDSP process.

IDEAR tool for R

To install this tool, navigate to https://github.com/Azure/Azure-TDSP-Utilities/blob/master/DataScienceUtilities/DataReport-Utils/R/Run-IDEAR.R in order to retrieve the code required for this tool. Download the GitHub repository to make it easier to navigate the different files required for this exercise.

Open R Studio on your computer, navigate to the...

Summary

In conclusion, we have introduced you to the TDSP in this chapter and covered each of the different steps that are involved in detail. This process is meant to augment other existing processes rather than replace them. We also looked at various TDSP utilities that Microsoft has provided that make it easier to build some structure into the data science life cycle. In the next few chapters, we will look at each of the options available within Azure to build AI solutions for your business needs.