Go Web Scraping Quick Start Guide

4 (1 reviews total)
By Vincent Smith
    What do you get with a Packt Subscription?

  • Instant access to this title and 7,500+ eBooks & Videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Free Chapter
    Introducing Web Scraping and Go
About this book

Web scraping is the process of extracting information from the web using various tools that perform scraping and crawling. Go is emerging as the language of choice for scraping using a variety of libraries. This book will quickly explain to you, how to scrape data data from various websites using Go libraries such as Colly and Goquery.

The book starts with an introduction to the use cases of building a web scraper and the main features of the Go programming language, along with setting up a Go environment. It then moves on to HTTP requests and responses and talks about how Go handles them. You will also learn about a number of basic web scraping etiquettes.

You will be taught how to navigate through a website, using a breadth-first and then a depth-first search, as well as find and follow links. You will get to know about the ways to track history in order to avoid loops and to protect your web scraper using proxies.

Finally the book will cover the Go concurrency model, and how to run scrapers in parallel, along with large-scale distributed web scraping.

Publication date:
January 2019
Publisher
Packt
Pages
132
ISBN
9781789615708

 

Introducing Web Scraping and Go

Collecting, parsing, storing, and processing data are essential tasks that almost everybody will need to do in their software development career. Staying on top of emerging technologies that greatly improve the stability, speed, and efficiency of application development is another challenge. To provide insight into how to accomplish both of these goals, I have written this book. Here, you will find a guide for performing web scraping in Go. This book covers a broad perspective on web scraping, from the basics of the Hypertext Transfer Protocol (HTTP) and Hypertext Markup Language (HTML), to building highly concurrent distributed systems.

In this chapter, you will find explanations on the following topics:

  • What is web scraping?
  • Why do you need a web scraper?
  • What is Go?
  • Why is Go a good fit for web scraping?
  • How can you set up a Go development environment?
 

What is web scraping?

Web scraping at, its core, is collecting publicly available information from the internet for a specific purpose. It has taken form under many different names, such as following:

  • Spiders
  • Crawlers
  • Bots

Although the name may carry a negative connotation, the practice of web scraping has been around since the beginning of the internet and has grown into various technologies and techniques. In fact, some companies have built their entire business on web scraping!

 

Why do you need a web scraper?

There are many different use cases where you might need to build a web scraper. All cases center around the fact that information on the internet is often disparate, but can be very valuable when collected into one single package. Often, in these cases, the person collecting the information does not have a working or business relationship with the producers of the data, meaning they cannot request the information to be packaged and delivered to them. Because of the lack of this relationship, the one who needs the data has to rely on their own means to gather the information.

Search engines

One well-known use case for web scraping is indexing websites for the purpose of building a search engine. In this case, a web scraper would visit different websites and follow references to other websites in order to discover all of the content available on the internet. By collecting some of the content from the pages, you could respond to search queries by matching the terms to the contents of the pages you have collected. You could also suggest similar pages if you track how pages are linked together, and rank the most important pages by the number of connections they have to other sites.

Googlebot is the most famous example of a web scraper used to build a search engine. It is the first step in building the search engine as it downloads, indexes, and ranks each page on a website. It will also follow links to other websites, which is how it is able to index a substantial portion of the internet. According to Googlebot's documentation, the scraper attempts to reach each web page every few seconds, which requires them to reach estimates of well into billions of pages per day!

If your goal is to build a search engine, albeit on a much smaller scale, you will find enough tools in this book to collect the information you need. This book will not, however, cover indexing and ranking pages to provide relevant search results.

Price comparison

Another known use case is to find specific products or services sold through various websites and track their prices. You would be able to see who sells the item, who has the lowest price, or when it is most likely to be in stock. You might even be interested in similar products from different sources. Having a web scraper periodically visit websites to monitor these products and services would be easily solve this problem. This is very similar to tracking prices for flights, hotels, and rental cars as well.

Sites like camelcamelcamel (https://camelcamelcamel.com/) build their business model around such a case. According to their blog post explaining how their system works, they actively collect pricing information from multiple retailers every half hour to every few hours, covering millions of products. This allows users to view pricing differences across multiple platforms, as well as get notified if the price of an item drops.

This type of web scraper requires very careful parsing of the web pages to extract only the content that is relevant. In later chapters, you will learn how to extract information from HTML pages in order to collect this information.

Building datasets

Data scientists often need hundreds of thousands of data points in order to build, train, and test machine learning models. In some cases, this data is already pre-packaged and ready for consumption. Most of the time, the scientist would need to venture out on their own and build a custom dataset. This is often done by building a web scraper to collect raw data from various sources of interest, and refining it so it can be processed later on. These web scrapers also need to periodically collect fresh data to update their predictive models with the most relevant information.

A common use case that data scientists run into is determining how people feel about a specific subject, known as sentiment analysis. Through this process, a company could look for discussions surrounding one of their products, or their overall presence, and gather a general consensus. In order to do this, the model must be trained on what a positive comment and a negative comment are, which could take thousands of individual comments in order to make a well-balanced training set. Building a web scraper to collect comments from relevant forums, reviews, and social media sites would be helpful in constructing such a dataset.

These are just a few examples of web scrapers that drive large business such as Google, Mozenda, and Cheapflights.com. There are also companies that will scrape the web for whatever available data you need, for a fee. In order to run scrapers at such a large scale, you would need to use a language that is fast, scalable, and easy to maintain.

 

What is Go?

Go is a programming language created by Google employees in 2007. At the time of its creation, the goal was to build a language that was fast, safe, and simple. Go first officially released its 1.0 version in 2012 and is one of the fastest growing programming languages today. According to the Stack Overflow 2018 Developer Survey, Go is ranked in the top five of the most-loved languages and the top three in the most-wanted languages.

Go powers many large-scale web infrastructure platforms and tools such as Docker, Kubernetes, and Terraform. These platforms enable companies to build production-scale products supporting Fortune 500 companies. This is mostly the result of the design of the Go language, making it straightforward and clear to work with. Many other companies using Go for their development often tout the performance improvements over other languages.

 

Why is Go a good fit for web scraping?

The architecture of the Go programming language, as well as its standard libraries, make it a great choice for building web scrapers that are fast, scalable, and maintainable. Go is a statically typed, garbage-collected language with a syntax closer to C/C++. The syntax of the language will feel very familiar to developers coming from object-oriented programming languages. Go also has a few functional programming elements as well, such as higher-order functions. With all that being said, there are three main reasons why Go is a great fit for web scraping:

Go is fast

Speed is one of the primary objectives of the Go programming language. Many benchmarks put the speed of Go on par with that of C++, Java, and Rust, and miles ahead of languages such as Python and Ruby. Benchmark tests should always be considered with a bit of skepticism, but Go consistently stands out as a language with extremely high-performance numbers. This speed is typically coupled with a low resource footprint, as the runtime is very lightweight and does not use much RAM. One of the hidden benefits of this is being able to run Go programs on smaller machines, or to run multiple instances on the same machine, without significant overhead. This reduces the cost of operating a web scraper at larger scales.

This speed is inherently important in building web scrapers, and becomes more noticeable at larger scales. Take, for example, a web scraper that requires two minutes to scrape a page; you could theoretically process 720 pages in a day. If you were able to reduce that time to one minute per page, you would double the amount of pages per day to 1,440! Better yet, this would be done at the same cost. The speed and efficiency of Go allow you to do more with less.

Go is safe

One of the contributing factors to its speed is the fact that Go is statically typed. This makes the language ideal for building systems at a large scale and being confident in how your program will run in production. Also, since Go programs are built with a compiler instead of being run with an interpreter, it allows you to catch more bugs at compile time and greatly reduces the dreaded runtime errors.

This safety net is also extended to the Go garbage collector. Garbage collection means that you do not need to manually allocate and deallocate memory. This helps prevent memory leaks that might occur from mishandling objects in your code. Some may argue that garbage collection impedes the performance of your application, however, the Go garbage collector adds very little overhead in terms of interfering with your code execution. Many source report that the pauses caused by Go's garbage collector are less than one millisecond. In most cases, it's a very small price to pay to avoid chasing down memory leaks in the future. This certainly holds true for web scrapers.

As web scrapers grow in both size and complexity, it can be difficult to track all of the errors that may occur during processing. Thinking on the scale of processing thousands of web pages per day, one small bug could cause significantly affect the collection of data. At the end of the day, data missed is money lost, so preventing as many known errors as possible before the system is running is critical to your system.

Go is simple

Beyond the architecture of the Go programming language itself, the standard library offers all the right packages you need to make web scraping easy. Go offers a built-in HTTP client in the net/http package that is fully-featured out of the box, but also allows for a lot of customization. Making an HTTP request is as simple, as follows:

http.Get("http://example.com")

Also a part of the net/http package are utilities to structure HTTP requests, HTTP responses, and all of the HTTP status codes, which we will dive into later in this book. You will rarely need any third-party packages to handle communication with web servers. The Go standard library also has tools to help analyze HTTP requests, quickly consume HTTP response bodies, and debug the requests and responses in your web scraper. The HTTP client in the net/http package is also very configurable, letting you tune special parameters and methods to suit your specific needs. This typically will not need to be done, but the option exists if you encounter such a situation.

This simplicity will help eliminate some of the guesswork of writing code. You will not need to determine the best way to make an HTTP request; Go has already worked it out and provided you with the best tools you need to get the job done. Even when you need more than just the standard library, the Go community has built tools that follow the same culture of simplicity. This certainly makes integrating third-party libraries an easy task.

 

How to set up a Go development environment

Before you get started building a web scraper, you will need the proper tools. Setting up a development environment for writing Go code is relatively simple. There are not a lot of external tools that you will need to install, and there is support for all major computing platforms. For all of the tools listed in this chapter, you will find individual instructions for Windows, Mac, and Linux systems. Also, since all of the tools we will use are open source, you will be able to access the source code and build it for your specific needs if necessary.

Go language and tools

First and foremost, you'll need to install the Go programming language and tools on your machine. The installation process varies for different operating systems so please follow the instructions at https://golang.org/doc/install. On the installation page, you will find instructions for downloading Go for your platform, as well as the minimum operating system requirements.

It would be a good idea for you to spend some extra time browsing the Go programming language website to learn more about the language, read tutorials, and find the standard library documentation.

This is a screenshot from the installation page from the Go website, containing all of the instructions necessary for installing Go on your computer:

You can also build the language from source if you are so inclined. By the end of the installation, you should have the all of the Go libraries, the Go command line, and a simple hello world project built to ensure that everything was installed properly.

It is very important to follow the instructions all the way through testing your installation. Go can be a little tricky sometimes with respect to $GOPATH. Once you set up your $GOPATH, you must ensure that following is done:

  • You have the required src, bin, and pkg directories
  • All source code is contained within the src directory
  • The folder structure inside your src directory mimics what you want your package names to be

By completing the testing section, you will save yourself a lot of frustration in the future.

Since the release of version 1.11, the Go team has announced support for Go modules, which allows you to develop outside of the $GOPATH. Because this feature is still considered experimental, this book will continue with the classic method for Go development.

Git

You will also need to install the Git version control software. This will be used to download third-party libraries onto your machine. The go get command relies on Git being installed on your system to download libraries and install them directly into your $GOPATH. You may also feel free to use Git to download the examples for each chapter. All of the examples in this book will be using open source libraries that are available on GitHub. You can install Git for your system by following the instructions at https://git-scm.com/download.

The Git command-line tool is a vast set of commands used for versioning, storing, and retrieving source code. These commands are the basis that power the GitHub website. It is highly recommended that you learn how to use the tool to interact with the GitHub site, rather than going through the UI.

The following is a screenshot of the Git download page, containing the links for your respective operating system:

Editor

The second tool you will need is a good text editor or Integrated Development Environment (IDE). If you are not familiar with IDEs, they are basically text editors that are custom-built for writing applications for specific programming languages. One well-known IDE for Go is GoLand by JetBrains. This comes with built-in support for syntax highlighting, run and debug modes, built-in version control, and package management.

GoLand is available as a 30-day trial, after which you must buy a license to continue using it.

The following is a screenshot of the GoLand IDE displaying the standard Hello World program:

If you prefer to use a text editor, there are many available and they often have plugins for Go that make developing easier. Two of the best text editors available today are Visual Studio Code by Microsoft and Atom by GitHub. Both of these are general purpose editors that also have plugins for syntax highlighting, building, and running Go code. This way you can add what you need without too much overhead.

This screenshot is the same Hello World program, displayed in Visual Studio Code:

Finally, the Atom Version of the Hello World program looks like the following screenshot:

Both the Visual Studio Code and Atom are excellent choices for building Go applications due to the level of community support for the plugins, which I highly recommend installing. Alternatively, you can write Go programs in any text editor and run the code using your terminal or Command Prompt with the standard Go commands.

You will need a solid internet connection. A proper internet connection will eliminate errors connecting to different websites. If you are building a web scraper that sits behind a network firewall, or if you have a weak network connection, you may encounter difficulties accessing some of the sites used as examples in this book.
 

Summary

In this chapter, you learned a few of the use cases for building a web scraper and examples of businesses related to them. You also learned a few of the strengths of the Go programming language and created a development environment suitable for building your web scraper. These steps should help you get started on that path.

In Chapter 2, The Request/Response Cycle, we look at how to communicate with web servers in Go. We will learn the basics of how your web scraper communicates with web servers.

About the Author
  • Vincent Smith

    Vincent Smith has been a software engineer for 10 years, having worked in various fields from health and IT to machine learning, and large-scale web scrapers. He has worked for both large-scale Fortune 500 companies and start-ups alike and has sharpened his skills from the best of both worlds. While obtaining a degree in electrical engineering, he learned the foundations of writing good code through his Java courses. These basics helped spur his career in software development early in his professional career in order to provide support for his team. He fell in love with the process of teaching computers how to behave and set him on the path he still walks today.

    Browse publications by this author
Latest Reviews (1 reviews total)
I thought the book was a little concise, but if you have done web scraping with other programming languages (like Python), then it's informative enough and it gives some good tips for Go language as well.
Recommended For You
Go Web Scraping Quick Start Guide
Unlock this book and the full library FREE for 7 days
Start now