Overview
This chapter describes Jupyter Notebooks and their use in data analysis. It also explains the features of Jupyter Notebooks, which allow for additional functionality beyond running Python code. You will learn and implement the fundamental features of Jupyter Notebooks by completing several hands-on exercises. By the end of this chapter, you will be able to use some important features of Jupyter Notebooks and some key libraries available in Python.
Our approach to learning in this book is highly applied since hands-on learning is the quickest way to understand abstract concepts. With this in mind, the focus of this chapter is to introduce Jupyter Notebooks—the data science tool that we will be using throughout this book.
Since Jupyter Notebooks have gained mainstream popularity, they have been one of the most important tools for data scientists who use Python. This is because they offer a great environment for a variety of tasks, such as performing quick and dirty analysis, researching model selection, and creating reproducible pipelines. They allow for data to be loaded, transformed, and modeled inside a single file, where it's quick and easy to test out code and explore ideas along the way. Furthermore, all of this can be documented inline using formatted text, which means you can make notes or even produce a structured report.
Other comparable platforms—for example, RStudio or Spyder—offer multiple panels to work between. Frequently, one of these panels will be a Read Eval Prompt Loop (REPL), where code is run on a Terminal session that has saved memory. Code written here may end up being copied and pasted into a different panel within the main codebase, and there may also be additional panels to see visualizations or other files. Such development environments are prone to efficiency issues and can promote bad practices for reproducibility if you're not careful.
Jupyter Notebooks work differently. Instead of having multiple panels for different components of your project, they offer the same functionality in a single component (that is, the Notebook), where the text is displayed along with code snippets, and code outputs are displayed inline. This lets you code efficiently and allows you to look back at previous work for reference, or even make alterations.
We'll start this chapter by explaining exactly what Jupyter Notebooks are and why they are so popular among data scientists. Then, we'll access a Notebook together and go through some exercises to learn how the platform is used.
In this section, we will briefly demonstrate the usefulness of Jupyter Notebooks with examples. Then, we'll walk through the basics of how they work and how to run them within the Jupyter platform. For those who have used Jupyter Notebooks before, this will be a good refresher, and you are likely to uncover new things as well.
Jupyter Notebooks are locally run on web applications that contain live code, equations, figures, interactive apps, and Markdown text in which the default programming language is Python. In other words, a Notebook will assume you are writing Python unless you tell it otherwise. We'll see examples of this when we work through our first workbook, later in this chapter.
Note
Jupyter Notebooks support many programming languages through the use of kernels, which act as bridges between the Notebook and the language. These include R, C++, and JavaScript, among many others. A list of available kernels can be found here: https://packt.live/2Y0jKJ0.
The following is an example of a Jupyter Notebook:
Figure 1.1: Jupyter Notebook sample workbook
Besides executing Python code, you can write in Markdown to quickly render formatted text, such as titles, lists, or bold font. This can be done in combination with code using the concept of independent cells in the Notebook, as seen in Figure 1.2. Markdown is not specific to Jupyter; it is also a simple language used for styling text and creating basic documents. For example, most GitHub repositories have a README.md file that is written in Markdown format. It's comparable to HTML but offers much less customization in exchange for simplicity.
Commonly used symbols in markdown include hashes (#) to make text into a heading, square ([]) and round brackets (()) to insert hyperlinks, and asterisks (*) to create italicized or bold text:
Figure 1.2: Sample Markdown document
In addition, Markdown can be used to render images and add hyperlinks in your document, both of which are supported in Jupyter Notebooks.
Jupyter Notebooks was not the first tool to use Markdown alongside code. This was the design of R Markdown, a hybrid language where R code can be written and executed inline with Markdown text. Jupyter Notebooks essentially offer the equivalent functionality for Python code. However, as we will see, they function quite differently from R Markdown documents. For example, R Markdown assumes you are writing Markdown unless otherwise specified, whereas Jupyter Notebooks assume you are inputting code. This and other features (as we will explore throughout) make it more appealing to use Jupyter Notebooks for rapid development in data science research.
While Jupyter Notebooks offer a blank canvas for a general range of applications, the types of Notebooks commonly seen in real-world data science can be categorized as either lab-style or deliverable.
Lab-style Notebooks serve as the programming analog of research journals. These should contain all the work you've done to load, process, analyze, and model the data. The idea here is to document everything you've done for future reference. For this reason, it's usually not advisable to delete or alter previous lab-style Notebooks. It's also a good idea to accumulate multiple date-stamped versions of the Notebook as you progress through the analysis, in case you want to look back at previous states.
Deliverable Notebooks are intended to be presentable and should contain only select parts of the lab-style Notebooks. For example, this could be an interesting discovery to share with your colleagues, an in-depth report of your analysis for a manager, or a summary of the key findings for stakeholders.
In either case, an important concept is reproducibility. As long as all the relevant software versions were documented at runtime, anybody receiving a Notebook can rerun it and compute the same results as before. The process of actually running code in a Notebook (as opposed to reading a pre-computed version) brings you much closer to the actual data. For example, you can add cells and ask your own questions regarding the datasets or tweak existing code. You can also experiment with Python to break down and learn about sections of code that you are struggling to understand.
It's finally time for our first exercise. We'll start by exploring the interface of the Jupyter Notebook and the JupyterLab platforms. These are very similar applications for running Jupyter Notebook (.ipynb) files, and you can use whatever platform you prefer for the remainder of this book, or swap back and forth, once you've finished the following exercises.
Note
The .ipynb file extension is standard for Jupyter Notebooks, which was introduced back when they were called IPython Notebooks. These files are human-readable JSON documents that can be opened and modified with any text editor. However, there is usually no reason to open them with any software other than Juptyer Notebook or JupyterLab, as described in this section. Perhaps the one exception to this rule is when doing version control with Git, if you may want to see the changes in plain text.
At this stage, you'll need to make sure that you have the companion material downloaded. This can be downloaded from the open source repository on GitHub at https://packt.live/2zwhfom.
In order to run the code, you should download and install the Anaconda Python distribution for Python 3.7 (or a more recent version). If you already have Python installed and don't want to use Anaconda, you may choose to install the dependencies manually instead (see requirements.txt in the GitHub repository).
Note
Virtual environments are a great tool for managing multiple projects on the same machine. Each virtual environment may contain a different version of Python and external libraries. In addition to Python's built-in virtual environments, conda also offers virtual environments, which tend to integrate better with Jupyter Notebooks.
For the purposes of this book, you do not need to worry about virtual environments. This is because they add complexity that will likely lead to more issues than they aim to solve. Beginners are advised to run global system installs of Python libraries (that is, using the pip commands shown here). However, more experienced Python programmers might wish to create and activate a virtual environment for this project.
We will install additional Python libraries throughout this book, but it's recommended to install some of these (such as mlxtend, watermark, and graphviz) ahead of time if you have access to an internet connection now. This can be done by opening a new Terminal window and running the pip or conda commands, as follows:
mlxtend (https://packt.live/3ftcN98): This is a useful tool for particular data science tasks. We'll use it to visualize the decision boundaries of models in Chapter 5, Model Validation and Optimization, and Chapter 6, Web Scraping with Jupyter Notebooks:pip install mlxtend
watermark (https://packt.live/2N1qjok): This IPython magic extension is used for printing version information. We'll use it later in this chapter:pip install watermark
graphviz (https://packt.live/3hqqCHz): This is for rendering graph visualizations. We'll use this for visualizing decision trees in Chapter 5, Model Validation and Optimization:conda install -c anaconda graphviz python-graphviz
graphviz will only be used once, so don't worry too much if you have issues installing it. However, hopefully, you were able to get mlxtend installed since we'll need to rely on it in later chapters to compare models and visualize how they learn patterns in the data.
In this exercise, we'll launch the Jupyter Notebook platform from the Terminal and learn how the visual user interface works. Follow these steps to complete this exercise:
git command-line tool:git clone https://github.com/PacktWorkshops/The-Applied-Data-Science-Workshop.git cd The-Applied-Data-Science-Workshop
Note
With Unix machines such as Mac or Linux, command-line navigation can be done using ls to display directory contents and cd to change directories. On Windows machines, use dir to display directory contents and use cd to change directories. If, for example, you want to change the drive from C: to D:, you should execute D: to change drives. This is an important step if you wish to enable all commands based on folder structure and ensure they run smoothly.
Note
The # symbol in the code snippet below denotes a code comment. Comments are added into code to help explain specific bits of logic.
jupyter notebook –-version # should return 6.0.2 or a similar / more recent version
jupyter notebook
A new window or tab of your default browser will open the Notebook Dashboard to the working directory. Here, you will see a list of folders and files contained therein.
[I 20:03:01.045 NotebookApp] The Jupyter Notebook is running at: http:// localhost:8888/?token=e915bb06866f19ce462d959a9193a94c7 c088e81765f9d8a
Going to the highlighted HTTP address will load the app in your browser window, as was done automatically when starting the app.
chapter-06), and then clicking on an .ipynb file (such as chapter_6_workbook.ipynb) to open it. This will cause the Notebook to open in a new tab on your browser.… button (above the folder content listing) or the folder icon above that (in the current directory breadcrumb).requirements.txt text file. Similar to the Notebook file, it will open in a new tab of your browser.y and pressing Enter. After doing this, close the web browser window as well.jupyter notebook --help
9000 by running the following command:jupyter notebook --port 9000
New in the upper right-hand corner of the Jupyter Dashboard and select a kernel from the drop-down menu (that is, select something in the Notebooks section):
Figure 1.3: Selecting a kernel from the drop-down menu
This is the primary method of creating a new Jupyter Notebook.
Kernels provide programming language support for the Notebook. If you have installed Python with Anaconda, that version should be the default kernel. Virtual environments that have been properly configured will also be available here.
print('hello world'), or any other code snippet that writes to the screen.Run Cell from the Cell menu.Any stdout or stderr output from the code will be displayed beneath as the cell runs. Furthermore, the string representation of the object written in the final line will be displayed as well. This is very handy, especially for displaying tables, but sometimes, we don't want the final object to be displayed. In such cases, a semicolon (;) can be added to the end of the line to suppress the display. New cells expect and run code input by default; however, they can be changed to render markdown instead.
Markdown from the Cell menu. Write some text in here (any text will do), making sure to utilize Markdown formatting symbols such as #, and then run the cell using Shift + Enter:
Figure 1.4: Menu options for converting cells into code/Markdown
Run button in the toolbar:
Figure 1.5: Toolbar icon to start cell execution
Stop icon, which can be used to stop cells from running. This is useful, for example, if a cell is taking too long to run:
Figure 1.6: Toolbar icon to stop cell execution
Insert menu:
Figure 1.7: Menu options for adding new cells
Edit menu:
Figure 1.8: Toolbar icons to cut, copy, and paste cells
The drop-down list from the Edit menu is as follows:
Figure 1.9: Menu options to cut, copy, and paste cells
Figure 1.10: Toolbar icons for moving cells up or down
There are useful options in the Cell menu that you can use to run a group of cells or the entire Notebook:
Figure 1.11: Menu options for running cells in bulk
Experiment with the toolbar options to move cells up and down, insert new cells, and delete cells. An important thing to understand about these Notebooks is the shared memory between cells. It's quite simple; every cell that exists on the sheet has access to the global set of variables. So, for example, a function defined in one cell could be called from any other, and the same applies to variables. As you would expect, anything within the scope of a function will not be a global variable and can only be accessed from within that specific function.
Kernel menu to see the selections. The Kernel menu is useful for stopping the execution of the script and restarting the Notebook if the kernel dies: 
Figure 1.12: Menu options for selecting a Notebook kernel
Kernels can also be swapped here at any time, but it is inadvisable to use multiple kernels for a single Notebook due to reproducibility concerns.
File menu to see the selections. The File menu contains options for downloading the Notebook in various formats. It's recommended to save an HTML version of your Notebook, where the content is rendered statically and can be opened and viewed as you would expect in web browsers.Untitled. You can change the name of your .ipynb Notebook file by clicking on the current name in the upper left corner and typing in the new name. Then, save the file.Files section of the Jupyter Dashboard, and it will be listed as Running on the right-hand side next to the last modified date. Notebooks can be shut down from here.Shutdown button.Note
If you plan to spend a lot of time working with Jupyter Notebooks, it's worthwhile learning the keyboard shortcuts. This will speed up your workflow considerably. Particularly useful commands to learn are the shortcuts for manually adding new cells and converting cells from code into Markdown formatting. Click on Keyboard Shortcuts from the Help menu to see how.
y and pressing Enter. This will automatically exit any kernel that is running. Do this now and close the browser window as well.Now that we have learned the basics of Jupyter Notebooks, we will launch and explore the JupyterLab platform.
While the Jupyter Notebook platform is lightweight and simple by design, JupyterLab is closer to R Studio in design. In JupyterLab, you can stack notebooks side by side, along with console environments (REPLs) and data tables, among other things you may want to look at.
Although the new features it provides are nice, the simplicity of the Jupyter Notebook interface means that it's still an appealing choice. Aside from its simplicity, you may find the Jupyter Notebook platform preferable for the following reasons:
.ipynb files (this is an open issue on GitHub, as of early 2020).Please don't let these small issues hold you back from trying out JupyterLab. In fact, it would not be surprising if you decide to use it for running the remainder of the exercises and activities in this book.
The future of open source tooling around Python and data science is going to be very exciting, and there are sure to be plenty of developments regarding Jupyter tools in the years to come. This is all thanks to the open source programmers who build and maintain these projects and the companies that contribute to the community.
In this exercise, we'll launch the JupyterLab platform and see how it compares with the Jupyter Notebook platform.
Follow these steps to complete this exercise:
jupyter lab --version # should return 1.2.3 or a similar / more recent version
jupyter lab
Similar to when we ran the Jupyter Notebook server, a new window or tab on your default browser should open the JupyterLab Dashboard. Here, you will see a list of folders and files in the working directory in a navigation bar to the left:
Figure 1.13: JupyterLab dashboard
8888 by default:[I 18:37:29.369 LabApp] The Jupyter Notebook is running at: [I 18:37:29.369 LabApp] http://localhost:8888/?token=cb55c8f3c03f0d6843ae59e70bedbf3b6ec 4a92288e65fa3
File | New | Notebook:
Figure 1.14: Opening a new notebook
Python 3:
Figure 1.15: Selecting a kernel for our notebook
The Notebook will then load into a new tab inside JupyterLab. Notice how this is different from the Jupyter Notebook platform, where each file is opened in its own browser tab.
Figure 1.16: JupyterLab toolbar and Notebook tab
for i in range(10): print(i, i % 3)
This will look as follows in Jupyter Notebook:
Figure 1.17: Output of the for loop
Figure 1.18: Bars that hide/show cells and output in JupyterLab
File | Save Notebook As and giving it the name test.ipynb:
Figure 1.19: Prompt for saving the name of the file
File | New | Console in order to load up a Python interpreter session:
Figure 1.20: Opening a new console session
test.ipynb under Use Kernel from Other Session. This feature of JupyterLab allows each process to have shared access to variables in memory:
Figure 1.21: Electing the console kernel
Console window tab and drag it down to the bottom half of the screen in order to stack it underneath the Notebook. Now, define something in the console session, such as the following:a = 'apple'
It will look as follows:
Figure 1.22: Split view of the Notebook and console in JupyterLab
Run menu), and then run another cell below to test that your variable returns the value as expected; for example, print(a).test.ipynb Notebook and print the variable there. Test that this works as expected; for example, print(a):
Figure 1.23: Sharing a kernel between processes in JupyterLab
A great feature of JupyterLab is that you can open up and work on multiple views of the same Notebook concurrently—something that cannot be done with the Jupyter Notebook platform. This can be very useful when working in large Notebooks where you want to frequently look at different sections.
test.ipynb by right-clicking on its tab and selecting New View for Notebook: 
Figure 1.24: Opening a new view for an open Notebook
You should see a copy of the Notebook open to the right. Now, start typing something into one of the cells and watch the other view update as well:
Figure 1.25: Two live views of the same Notebook in JupyterLab
There are plenty of other neat features in JupyterLab that you can discover and play with. For now, though, we are going to close down the platform.
SHUT DOWN to close anything that is open:
Figure 1.26: Shutting down Notebook sessions in JupyterLab
Y and pressing Enter. This will automatically exit any kernel that is running. Do this now and close the browser window as well:
Figure 1.27: Shutting down the LabApp
In this exercise, we learned about the JupyterLab platform and how it compares to the older Jupyter Notebook platform for running Notebooks. In addition to learning about the basics of using the app, we explored some of its awesome features, all of which can help your data science workflow.
In the next section, we'll learn about some of the more general features of Jupyter that apply to both platforms.
Having familiarized ourselves with the interface of two platforms for running Notebooks (Jupyter Notebook and JupyterLab), we are ready to start writing and running some more interesting examples.
Note
For the remainder of this book, you are welcome to use either the Jupyter Notebook platform or JupyterLab to follow along with the exercises and activities. The experience is similar, and you will be able to follow along seamlessly either way. Most of the screenshots for the remainder of this book have been taken from JupyterLab.
Jupyter has many appealing core features that make for efficient Python programming. These include an assortment of things, such as tab completion and viewing docstrings—both of which are very handy when writing code in Jupyter. We will explore these and more in the following exercise.
Note
The official IPython documentation can be found here: https://ipython.readthedocs.io/en/stable/. It provides details of the features we will discuss here, as well as others.
In this exercise, we'll relaunch the Jupyter platform and walk through a Notebook to learn about some core features, such as navigating workbooks with keyboard shortcuts and using magic functions. Follow these steps to complete this exercise:
JupyterLab (run jupyter lab)
Jupyter Notebook (run jupyter notebook)
Then, open the platform in your web browser by copying and pasting the URL, as prompted in the Terminal.
Note
Here's the list of basic keyboard shortcuts; these are especially helpful if you wish to avoid having to use the mouse so often, which will greatly speed up your workflow.
Shift + Enter to run a cell
Esc to leave a cell
a to add a cell above
b to add a cell below
dd to delete a cell
m to change a cell to Markdown (after pressing Esc)
y to change a cell to code (after pressing Esc)
Arrow keys to move cells (after pressing Esc)
Enter to enter a cell
You can get help by adding a question mark to the end of any object and running the cell. Jupyter finds the docstring for that object and returns it in a pop-up window at the bottom of the app.
numpy and get the arrange docstring, as follows:import numpy as np np.arange?
The output will be similar to the following:
Figure 1.28: The docstring for np.arange
sorted function docstring as follows:sorted?
The output is as follows:
Figure 1.29: The docstring for sort
a = np.array([1, 2, 3]) a.*?
Here's the output showing the list:
Figure 1.30: The output after running a.*?
Tab Completion section. Type import (including the space after) and then press the Tab key:import <tab>
Tab completion can be used to do the following:
from numpy import <tab>
np.<tab>
np.ar<tab> sor<tab>([2, 3, 1]) myvar_1 = 5 myvar_2 = 6 my<tab>
Test each of these examples for yourself in the following cells:
Figure 1.31: An example of tab completion for variable names
Note
Tab completion is different in the JupyterLab and Jupyter Notebook platforms. The same commands may not work on both.
Tab completion can be especially useful when you need to know the available input arguments for a module, explore a new library, discover new modules, or simply speed up the workflow. They will save time writing out variable names or functions and reduce bugs from typos. Tab completion works so well that you may have difficulty coding Python in other editors after today.
%lsmagic
The output is as follows:
Figure 1.32: Jupyter magic functions
Note
If you're running JupyterLab, you will not see the preceding output. A list of magic functions, along with information about each, can be found here: https://ipython.readthedocs.io/en/stable/interactive/magics.html.
The percent signs, % and %%, are one of the basic features of Jupyter Notebook and are called magic commands. Magic commands starting with %% will apply to the entire cell, while magic commands starting with % will only apply to that line.
plt.show() each time you plot something. You only need to execute it once at the beginning of the session:%matplotlib inline
The timing functions are also very handy magic functions and come in two varieties: a standard timer (%time or %%time) and a timer that measures the average runtime of many iterations (%timeit and %%timeit). We'll see them being used here.
a variable, as follows:a = [1, 2, 3, 4, 5] * int(1e5)
%%time for i in range(len(a)): a[i] += 5
The output is as follows:
CPU times: user 68.8 ms, sys: 2.04 ms, total: 70.8 ms Wall time: 69.6 ms
%time a = [_a + 5 for _a in a]
The output is as follows:
CPU times: user 21.1 ms, sys: 2.6 ms, total: 23.7 ms Wall time: 23.1 ms
%timeit set(a)
The output is as follows:
4.72 ms ± 55.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Note the difference in use between one and two percent signs. Even when using a Python kernel (as you are currently doing), other languages can be invoked using magic commands. The built-in options include JavaScript, R, Perl, Ruby, and Bash. Bash is particularly useful as you can use Unix commands to find out where you are currently (pwd), see what's in the directory (ls), make new folders (mkdir), and write file contents (cat/head/tail).
Note
Notice how list comprehensions are quicker than loops in Python. This can be seen by comparing the wall time for the first and second cell, where the same calculation is done significantly faster with list comprehension. Please note that the step 15-18 are Linux-based commands. If you are working on other operating systems like Windows and MacOS, these steps might not work.
%%bash echo "using bash from inside Jupyter!" > test-file.txt ls echo "" cat test-file.txt rm test-file.txt
The output is as follows:
Figure 1.33: Running a bash command in Jupyter
ls, as follows:%ls
The output is as follows:
chapter_1_workbook.ipynb
pwd. Notice how we needed to use the %%bash magic function for pwd, but not for ls:%%bash pwd
The output is as follows:
/Users/alex/Documents/The-Applied-Data-Science-Workshop/chapter-01
ipython-sql, which allows for SQL code to be executed in cells.Jupyter magic functions can be installed the same way as PyPI Python libraries, using pip or conda. Open a new Terminal window and execute the following code to install ipython-sql:
pip install ipython-sql
%load_ext sql cell to load the external command into the Notebook.This allows connections to be made to remote databases so that queries can be executed (and thereby documented) right inside the Notebook.
%%sql sqlite:// SELECT * FROM ( SELECT 'Hello' as msg1, 'World!' as msg2 );
The output is as follows:
Figure 1.34: Running a SQL query in Jupyter
Here, we connected to the local sqlite source with sqlite://; however, this line could instead point to a specific database on a local or remote server. For example, a .sqlite database file on your desktop could be connected to with the line %sql sqlite:////Users/alex/Desktop/db.sqlite, where the username in this case is alex and the file is db.sqlite.
After connecting, we execute a simple SELECT command to show how the cell has been converted to run SQL code instead of Python.
watermark external library, which helps to document versioning in the Notebook. If you haven't installed it yet, then open a new window and run the following code:pip install watermark
Once installed, it can be imported into any Notebook using %load_ext watermark. Then, it can be used to document library versions and system hardware.
watermark magic function and call its docstring with the following command:%load_ext watermark %watermark?
The output is as follows:
Figure 1.35: The docstring for watermark
Notice the various arguments that can be passed in when calling it, such as -a for author, -v for the Python version, -m for machine information, and -d for date.
watermark library to add version information to the notebook, as follows:Note
The code snippet shown here uses a backslash ( \ ) to split the logic across multiple lines. When the code is executed, Python will ignore the backslash, and treat the code on the next line as a direct continuation of the current line.
%watermark -d -v -m -p \ requests,numpy,pandas,matplotlib,seaborn,sklearn
The output is as follows:
Figure 1.36: watermark output in the Notebook
Note
To access the source code for this specific section, please refer to https://packt.live/30KoAfu.
You can also run this example online at https://packt.live/2Y49zTQ.
In this exercise, we looked at the core features of Jupyter, including tab completion and magic functions. You'll review these features and have a chance to test them out yourself in the activity at the end of this chapter.
In this section, we'll learn how to convert a Jupyter Notebook into a Python script. This is equivalent to copying and pasting the contents of each code cell into a single .py file. The Markdown sections are also included as comments.
It can be beneficial to convert a Notebook into a .py file because the code is then available in plain text format. This can be helpful for version control— to see the difference in code between two versions of a Notebook, for example. It can also be a helpful trick for extracting chunks of code.
This conversion can be done from the Jupyter Dashboard (File -> Download as) or by opening a new Terminal window, navigating to the chapter-02 folder, and executing the following:
jupyter nbconvert --to=python chapter_2_workbook.ipynb
The output is as follows:
Figure 1.37: Converting a Notebook into a script (.py) file
Note that we are using the next chapter's Notebook for this example.
Another benefit of converting Notebooks into .py files is that, when you want to determine the Python library requirements for a Notebook, the pipreqs tool will do this for us, and export them into a requirements.txt file. This tool can be installed by running the following command:
pip install pipreqs
You might require root privileges for this.
This command is called from outside the folder containing your .py files. For example, if the .py files are inside a folder called chapter-02, you could do the following:
pipreqs chapter-02/
The output is as follows:
Figure 1.38: Using pipreqs to generate a requirements.txt file
The resulting requirements.txt file for chapter_2_workbook.ipynb will look similar to the following:
cat chapter-02/requirements.txt matplotlib==3.1.1 seaborn==0.9.0 numpy==1.17.4 pandas==0.25.3 requests==2.22.0 beautifulsoup4==4.8.1 scikit_learn==0.22
Having now seen all the basics of Jupyter Notebooks, and even some more advanced features, we'll shift our attention to the Python libraries we'll be using in this book.
Libraries, in general, extend the default set of Python functions. Some examples of commonly used standard libraries are datetime, time, os, and sys. These are called standard libraries because they are included with every installation of Python.
For data science with Python, the most heavily relied upon libraries are external, which means they do not come as standard with Python.
The external data science libraries we'll be using in this book are numpy, pandas, seaborn, matplotlib, scikit-learn, requests, and bokeh.
Note
It's a good idea to import libraries using industry standards—for example, import numpy as np. This way, your code is more readable. Try to avoid doing things such as from numpy import *, as you may unwittingly overwrite functions. Furthermore, it's often nice to have modules linked to the library via a dot (.) for code readability.
Let's briefly introduce each:
numpy offers multi-dimensional data structures (arrays) that operations can be performed on. This is far quicker than standard Python data structures (such as lists). This is done in part by performing operations in the background using C. NumPy also offers various mathematical and data manipulation functions.pandas is Python's answer to the R DataFrame. It stores data in 2D tabular structures where columns represent different variables and rows correspond to samples. pandas provides many handy tools for data wrangling, such as filling in NaN entries and computing statistical descriptions of the data. Working with pandas DataFrames will be a big focus of this book.matplotlib is a plotting tool inspired by the MATLAB platform. Those familiar with R can think of it as Python's version of ggplot. It's the most popular Python library for plotting figures and allows for a high level of customization.seaborn works as an extension of matplotlib, where various plotting tools that are useful for data science are included. Generally speaking, this allows for analysis to be done much faster than if you were to create the same things manually with libraries such as matplotlib and scikit-learn.scikit-learn is the most commonly used machine learning library. It offers top-of-the-line algorithms and a very elegant API where models are instantiated and then fit with data. It also provides data processing modules and other tools that are useful for predictive analytics.requests is the go-to library for making HTTP requests. It makes it straightforward to get HTML from web pages and interface with APIs. For parsing HTML, many choose BeautifulSoup4, which we'll cover in Chapter 6, Web Scraping with Jupyter Notebooks.We'll start using these libraries in the next chapter.
We are going to be using pandas heavily in this book. In particular, any data that's loaded into our Notebooks will be done using pandas. The data will be contained in a DataFrame object, which can then be transformed and saved back to disk afterward. These DataFrames offer convenient methods for running calculations over the data for exploration, visualization, and modeling.
In this activity, you'll have the opportunity to use pandas DataFrames, along with the Jupyter features that have been discussed in this section. Follow these steps to complete this activity:
Note
While completing this activity, you will need to use many cells in the Notebook. Please insert new cells as required.
pd and np variables, respectively.pd.DataFrame. Scan through the Parameters section and read the Examples section.fruit and score keys, which correspond to list values with at least three items in each. Ensure that you give your dictionary a suitable name (note that in Python, a dictionary is a collection of values); for example, {"fruit": ["apple", ...], "score": [8, ...]}. pd.DataFrame(data=name of dictionary). Assign it to the df variable.df.sort_values DataFrame function and read through the Examples section.timeit magic function to test how long this sorting operation takes.Note
The detailed steps for this activity, along with the solutions, can be found via this link
In this chapter, we've gone over the basics of using Jupyter Notebooks for data science. We started by exploring the platform and finding our way around the interface. Then, we discussed the most useful features, which include tab completion and magic functions. Finally, we introduced the Python libraries we'll be using in this book.
As we'll see in the coming chapters, these libraries offer high-level abstractions that allow data science to be highly accessible with Python. This includes methods for creating statistical visualizations, building data cleaning pipelines, and training models on millions of data points and beyond.
While this chapter focused on the basics of Jupyter platforms, the next chapter is where the real data science begins. The remainder of this book is very interactive, and in Chapter 3, Preparing Data for Predictive Modeling, we'll perform an analysis of housing data using Jupyter Notebook and the Seaborn plotting library.
© 2020 Packt Publishing Limited All Rights Reserved
