Shell Workflows, and Data Acquisition and Massaging

In this chapter, we're going to work on an actual dataset and do some basic analysis. We'll learn how to download files straight from the command line, determine what type of file it is, and parse the data using a number of commands. We'll also cover how to perform non-interactive detached processing and review some common terminal multiplexers that enable us to prettify the command line as well as organize detached processing.

In this chapter, we'll cover the following topics:

• How to download a dataset using the command line
• Using built-in tools to inspect the data and its type
• How to perform a word count in bash
• Analyzing a dataset with some simple commands
• Detached processing
• Terminal multiplexers

Download the data

Now that we have an understanding of the command line, let's do something cool with it! Say we had a couple datasets full of book reviews from Amazon, and we wanted to only view the reviews about Packt Publishing. First, let's go ahead and grab the data (if you are using the Docker container, the data is located in /data):

curl -O https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_us_Digital_Ebook_Purchase_v1_00.tsv.gz && curl -O https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_us_Digital_Ebook_Purchase_v1_01.tsv.gz

You should see the following:

We are introducing a couple of new commands and features here to download the files. First, we call the curl command to download the file. You can run curl --help to view all of the options available, or man curl, but we wanted to download a remote file and save it as the original...

Using the file command

Once the data is done downloading, let's take a look and see what we've got. Go ahead and run ls -al amazon* to make sure the files actually downloaded:

If you have anything else in this directory named amazon, that will show up as well. Now that the files are downloaded, let's introduce a new command, called file. Go ahead and run the following file amazon* command:

Wow, without any parameters set, the file command was able to figure out that this is a compressed archive. You'll use the file command a lot to determine the type of files you're working with. Let's decompress the files so we can work with them. This might take a little bit, depending on the speed of your system.

To do so, run the following:

zcat amazon_reviews_us_Digital_Ebook_Purchase_v1_00.tsv.gz >> amazon_reviews_us_Digital_Ebook_Purchase_v1_00.tsv...

Performing a word count

Now that we have some data to work with, let's combine the two files together into a single file. To do so, perform the following:

cat *.tsv > reviews.tsv

This is what you should see once you run the preceding command:

Excellent. Let's say we wanted to count how many words or lines are in this file. Let's introduce the wc command. wc is short for (you guessed it) word count. Let's quickly man wc to see the options available:

Looks like wc can count the lines and also the words of a file. Let's see how many lines our file actually has:

wc -l reviews.tsv

The following is what you should see once you run the preceding command:

That's a lot of lines! What about words? Run the following:

wc -w reviews.tsv

This looks like a great dataset to use. It's not big data by any means, but there's a lot of cool stuff we...

Introduction to cut

Let's break the command down before you run it. The cut command removes sections from each line of a file. The -d parameter tells cut we are working with a tsv (tab separated values), and the -f parameter tells cut what fields we are interested in. Since product_title is the sixth field in our file, we started with that:

cut -d$'\t' -f 6,8,13,14 reviews.tsv | more

Unlike most programs, cut starts at 1 instead of 0.

Let’s see the results:

Much better! Let's go ahead and save this as a new file:

cut -d$'\t' -f 6,8,13,14 reviews.tsv > stripped_reviews.tsv

The following is what you should see once you run the preceding command:

Let's see how many times the word Packt shows up in this dataset:

grep -i Packt stripped_reviews.tsv | wc -w

The following is what you should see once you run the preceding command:

Let&apos...

Detached processing

Detached processing runs a command in the background. This means that terminal control is immediately returned to the shell process while the detached process runs in the background. With job control, these back grounded processes can be resumed in the foreground or killed directly.

How to background a process

Remember when we used the double ampersand to conditionally execute two commands that run one after another? By using a single ampersand, you can fork a process in the background and let it run. Let's use the command to save to a new file and run in the background:

cat all_reviews.csv | awk -F ","  '{print $4}' | grep -i Packt > background_words.txt &

This will take...

Summary

In this chapter, we only scratched the surface on what we can do with the command line. We were able to download a dataset, save it, inspect the file type, and perform some simple analytics. The word count example is considered the "Hello, World" of data science and we saw just how easy it is to perform in bash.

We then took your shell customization to the next level by using terminal multiplexers and background processes. Think of it like using an IDE, but for the command line. It will make working with bash a lot easier.

Being able to control processes and workflows will improve productivity. Detached processing ensures programs can complete without interruption. The terminal multiplexer provides a means of maximizing the use of screen real-estate, while also providing a detached processing environment, which is a double win for all.

In the next chapter, we...