Getting Started with Beautiful Soup

3.5 (2 reviews total)
By Vineeth G. Nair
  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies

About this book

Beautiful Soup is a Python library designed for quick turnaround projects like screen-scraping. Beautiful Soup provides a few simple methods and Pythonic idioms for navigating, searching, and modifying a parse tree: a toolkit for dissecting a document and extracting what you need without writing excess code for an application. It doesn't take much code to write an application using Beautiful Soup.

Getting Started with Beautiful Soup is a practical guide to Beautiful Soup using Python. The book starts by walking you through the installation of each and every feature of Beautiful Soup using simple examples which include sample Python codes as well as diagrams and screenshots wherever required for better understanding. The book discusses the problems of how exactly you can get data out of a website and provides an easy solution with the help of a real website and sample code.

Getting Started with Beautiful Soup goes over the different methods to install Beautiful Soup in both Linux and Windows systems. You will then learn about searching, navigating, content modification, encoding support, and output formatting with the help of examples and sample Python codes for each example so that you can try them out to get a better understanding. This book is a practical guide for scraping information from any website. If you want to learn how to efficiently scrape pages from websites, then this book is for you.

Publication date:
January 2014
Publisher
Packt
Pages
130
ISBN
9781783289554

 

Chapter 1. Installing Beautiful Soup

Before we begin using Beautiful Soup, we should ensure that it is properly installed on our machine. The steps required are so simple that any user can install this in no time. In this chapter, we will be covering the following topics:

  • Installing Beautiful Soup

  • Verifying the installation of Beautiful Soup

 

Installing Beautiful Soup


Python supports the installation of third-party modules such as Beautiful Soup. In the best case scenario, we can expect that the module developer might have prepared a platform-specific installer, for example, an executable installer, in the case of Windows; an rpm package, in the case of Red Hat-based Linux operating systems (Red Hat, Open Suse, and so on); and a Debian package, in the case of Debian-based operating systems (Debian, Ubuntu, and so on). But this is not always the case and we should know the alternatives if the platform-specific installer is not available. We will discuss the different installation options available for Beautiful Soup in different operating systems, such as Linux, Windows, and Mac OS X. The Python version that we are going to use in the later examples for installing Beautiful Soup is Python 2.7.5 and the instructions for Python 3 are probably different. You can directly go to the installation section corresponding to the operating system.

Installing Beautiful Soup in Linux

Installing Beautiful Soup is pretty simple and straightforward in Linux machines. For recent versions of Debian or Ubuntu, Beautiful Soup is available as a package and we can install this using the system package manager. For other versions of Debian or Ubuntu, where Beautiful Soup is not available as a package, we can use alternative methods for installation.

Normally, these are the following three ways to install Beautiful Soup in Linux machines:

  • Using package manager

  • Using pip

  • Using easy_install

The choices are ranked depending on the complexity levels and to avoid the trial-and-error method. The easiest method is always using the package manager since it requires less effort from the user, so we will cover this first. If the installation is successful in one step, we don't need to do the next because the three steps mentioned previously do the same thing.

Installing Beautiful Soup using package manager

Linux machines normally come with a package manager to install various packages. In the recent version of Debian or Ubuntu, since Beautiful Soup is available as a package, we will be using the system package manager for installation. In Linux machines such as Ubuntu and Debian, the default package manager is based on apt-get and hence we will use apt-get to do the task.

Just open up a terminal and type in the following command:

sudo apt-get install python-bs4

The preceding command will install Beautiful Soup Version 4 in our Linux operating system. Installing new packages in the system normally requires root user privileges, which is why we append sudo in front of the apt-get command. If we didn't append sudo, we will basically end up with a permission denied error. If the packages are already updated, we will see the following success message in the command line itself:

Since we are using a recent version of Ubuntu or Debian, python-bs4 will be listed in the apt repository. But if the preceding command fails with Package Not Found Error, it means that the package list is not up-to-date. This normally happens if we have just installed our operating system and the package list is not downloaded from the package repository. In this case, we need to first update the package list using the following command:

sudo apt-get update

The preceding command will update the necessary package list from the online package repositories. After this, we need to try the preceding command to install Beautiful Soup.

In the older versions of the Linux operating system, even after running the apt-get update command, we might not be able to install Beautiful Soup because it might not be available in the repositories. In these scenarios, we can rely on the other methods of installation using either pip or easy_install.

Installing Beautiful Soup using pip or easy_install

The pip and easy_install are the tools used for managing and installing Python packages. Either of them can be used to install Beautiful Soup.

Installing Beautiful Soup using pip

From the terminal, type the following command:

sudo pip install beautifulsoup4

The preceding command will install Beautiful Soup Version 4 in the system after downloading the necessary packages from http://pypi.python.org/.

Installing Beautiful Soup using easy_install

The easy_install tool installs the package from Python Package Index (PyPI). So, in the terminal, type the following command:

sudo easy_install beautifulsoup4

All the previous methods to install Beautiful Soup in Linux will not work if you do not have an active network connection. So, in case everything fails, we can still install Beautiful Soup. The last option would be to use the setup.py script that comes with every Python package downloaded from pypi.python.org. This method is also the recommended method to install Beautiful Soup in Windows and in Mac OS X machines. So, we will discuss this method in the Installing Beautiful Soup in Windows section.

Installing Beautiful Soup in Windows

In Windows, we will make use of the recent Python package for Beautiful Soup available from https://pypi.python.org/packages/source/b/beautifulsoup4/ and use the setup.py script to install Beautiful Soup. But before doing this, it will be easier for us if we add the path of Python in the system path. The next section discusses setting up the path to Python on a Windows machine.

Verifying Python path in Windows

Often, the path to python.exe will not be added to an environment variable by default in Windows. So, in order to check this from the Windows command-line prompt, you need to type the following command:

python.

The preceding command will work without any errors if the path to Python is already added in the environment path variable or we are already within the Python installed directory. But, it would be good to check the path variable for the Python directory entry.

If it doesn't exist in the path variable, we have to find out the actual path, which is entirely dependent on where you installed Python. For Python 2.x, it will be by C:\Python2x by default, and for Python 3.x, the path will be C:\Python3x by default.

We have to add this to the Path environment variable in the Windows machine. For this, right-click on My Computer | Properties | Environment Variables | System Variable.

Pick the Path variable and add the following section to the Path variable:

;C:\PythonXY for example C:\Python27

This is shown in the following screenshot:

Adding Python path in Windows (Python 2.7 is used in this example)

After the Python path is ready, we can follow the steps for installing Beautiful Soup on a Windows machine.

Note

The method, which will be explained in the next section, of installing Beautiful Soup using setup.py is the same for Linux, Windows, and Mac OS X operating systems.

Installing Beautiful Soup using setup.py

We can install Python packages using the setup.py script that comes with every Python package downloaded from the Python package index website: https://pypi.python.org/. The following steps are used to install the Beautiful Soup using setup.py:

  1. Download the latest tarball from https://pypi.python.org/packages/source/b/beautifulsoup4/.

  2. Unzip it to a folder (for example, BeautifulSoup).

  3. Open up the command-line prompt and navigate to the folder where you have unzipped the folder as follows:

    cd BeautifulSoup
    python setup.py install.
    
  4. The python setup.py install line will install Beautiful Soup in our system.

Note

We are not done with the list of possible options to use Beautiful Soup. We can use Beautiful Soup in our applications even if all of the options outlined until now fail.

 

Using Beautiful Soup without installation


The installation processes that we have discussed till now normally copy the module contents to a chosen installation directory. This varies from operating system to operating system and the path is normally /usr/local/lib/pythonX.Y/site-packages in Linux operating systems such as Debian and C:\PythonXY\Lib\site-packages in Windows (where X and Y represent the corresponding versions, such as Python 2.7). When we use import statements in the Python interpreter or as a part of a Python script, normally what the Python interpreter does is look in the predefined Python Path variable and look for the module in those directories. So, installing actually means copying the module contents into the predefined directory or copying this to some other location and adding the location into the Python path. The following method of using Beautiful Soup without going through the installation can be used in any operating system, such as Windows, Linux, or Mac OS X:

  1. Download the latest version of Beautiful Soup package from https://pypi.python.org/packages/source/b/beautifulsoup4/.

  2. Unzip the package.

  3. Copy the bs4 directory into the directory where we want to place all our Python Beautiful Soup scripts.

After we perform all the preceding steps, we are good to use Beautiful Soup. In order to import Beautiful Soup in this case, either we need to open the terminal in the directory where the bs4 directory exists or add this directory to the Python Path variable; otherwise, we will get the module not found error. This extra step is required because the method is specific to a project where the bs4 directory is included. But in the case of installing methods, as we have seen previously, Beautiful Soup will be available globally and can be used in any of the projects, and so the additional steps are not required.

 

Verifying the installation


To verify the installation, perform the following steps:

  1. Open up the Python interpreter in a terminal by using the following command:

    python
    
  2. Now, we can issue a simple import statement to see whether we have successfully installed Beautiful Soup or not by using the following command:

    from bs4 import BeautifulSoup
    

If we did not install Beautiful Soup and instead copied the bs4 directory in the workspace, we have to change to the directory where we have placed the bs4 directory before using the preceding commands.

 

Quick reference


The following table is an overview of commands and their implications:

sudo apt-get install python-bs4

This command is used for installing Python using a package manger in Linux.

sudo pip install beautifulsoup4

This command is used for installing Python using pip.

sudo easy_install beautifulsoup4

This command is used for installing Python using easy_install.

python setup.py install

This command is used for installing Python using setup.py.

from bs4 import BeautifulSoup

This command is used for verifying installation.

 

Summary


In this chapter, we covered the various options to install Beautiful Soup in Linux machines. We also discussed a way of installing Beautiful Soup in Windows, Linux, and Mac OS X using the Python setup.py script itself. We also discussed the method to use Beautiful Soup without even installing it. The verification of the Beautiful Soup installation was also covered.

In the next chapter, we are going to have a first look at Beautiful Soup by learning the different methods of converting HTML/XML content to different Beautiful Soup objects and thereby understanding the properties of Beautiful Soup.

About the Author

  • Vineeth G. Nair

    Vineeth G. Nair completed his bachelors in Computer Science and Engineering from Model Engineering College, Cochin, Kerala. He is currently working with Oracle India Pvt. Ltd. as a Senior Applications Engineer.

    He developed an interest in Python during his college days and began working as a freelance programmer. This led him to work on several web scraping projects using Beautiful Soup. It helped him gain a fair level of mastery on the technology and a good reputation in the freelance arena. He can be reached at [email protected]. You can visit his website at www.kochi-coders.com.

    Browse publications by this author

Latest Reviews

(2 reviews total)
To difficult to order hard copy.
well written and clear, thanks

Recommended For You

Book Title
Access this book, plus 7,500 other titles for FREE
Access now