Home Programming Python Geospatial Development - Third Edition

Python Geospatial Development - Third Edition

By Erik Westra
books-svg-icon Book
eBook $43.99 $29.99
Print $54.99
Subscription $15.99 $10 p/m for three months
$10 p/m for first 3 months. $15.99 p/m after that. Cancel Anytime!
What do you get with a Packt Subscription?
This book & 7000+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook + Subscription?
Download this book in EPUB and PDF formats, plus a monthly download credit
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook?
Download this book in EPUB and PDF formats
Access this title in our online reader
DRM FREE - Read whenever, wherever and however you want
Online reader with customised display settings for better reading experience
What do you get with video?
Download this video in MP4 format
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with video?
Stream this video
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with Audiobook?
Download a zip folder consisting of audio files (in MP3 Format) along with supplementary PDF
What do you get with Exam Trainer?
Flashcards, Mock exams, Exam Tips, Practice Questions
Access these resources with our interactive certification platform
Mobile compatible-Practice whenever, wherever, however you want
BUY NOW $10 p/m for first 3 months. $15.99 p/m after that. Cancel Anytime!
eBook $43.99 $29.99
Print $54.99
Subscription $15.99 $10 p/m for three months
What do you get with a Packt Subscription?
This book & 7000+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook + Subscription?
Download this book in EPUB and PDF formats, plus a monthly download credit
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook?
Download this book in EPUB and PDF formats
Access this title in our online reader
DRM FREE - Read whenever, wherever and however you want
Online reader with customised display settings for better reading experience
What do you get with video?
Download this video in MP4 format
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with video?
Stream this video
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with Audiobook?
Download a zip folder consisting of audio files (in MP3 Format) along with supplementary PDF
What do you get with Exam Trainer?
Flashcards, Mock exams, Exam Tips, Practice Questions
Access these resources with our interactive certification platform
Mobile compatible-Practice whenever, wherever, however you want
  1. Free Chapter
    Geospatial Development Using Python
About this book
Geospatial development links your data to locations on the surface of the Earth. Writing geospatial programs involves tasks such as grouping data by location, storing and analyzing large amounts of spatial information, performing complex geospatial calculations, and drawing colorful interactive maps. In order to do this well, you’ll need appropriate tools and techniques, as well as a thorough understanding of geospatial concepts such as map projections, datums, and coordinate systems. This book provides an overview of the major geospatial concepts, data sources, and toolkits. It starts by showing you how to store and access spatial data using Python, how to perform a range of spatial calculations, and how to store spatial data in a database. Further on, the book teaches you how to build your own slippy map interface within a web application, and finishes with the detailed construction of a geospatial data editor using the GeoDjango framework. By the end of this book, you will be able to confidently use Python to write your own geospatial applications ranging from quick, one-off utilities to sophisticated web-based applications using maps and other geospatial data.
Publication date:
May 2016
Publisher
Packt
Pages
446
ISBN
9781785288937

 

Chapter 1. Geospatial Development Using Python

This chapter provides an overview of the Python programming language and geospatial development. Please note that this is not a tutorial on how to use the Python language; Python is easy to learn, but the details are beyond the scope of this book.

In this chapter, we will see:

  • What the Python programming language is and how it differs from other languages

  • How the Python Standard Library and the Python Package Index make Python even more powerful

  • What the terms geospatial data and geospatial development refer to

  • An overview of the process of accessing, manipulating, and displaying geospatial data. How geospatial data can be accessed, manipulated, and displayed.

  • Some of the major applications of geospatial development

  • Some of the recent trends in the field of geospatial development

 

Python


Python (http://python.org) is a modern, high-level language suitable for a wide variety of programming tasks. It is often used as a scripting language, automating and simplifying tasks at the operating system level, but it is equally suitable for building large and complex programs. Python has been used to write web-based systems, desktop applications, games, scientific programs, and even utilities and other higher-level parts of various operating systems.

Python supports a wide range of programming idioms, from straightforward procedural programming to object-oriented programming and functional programming.

Python is sometimes criticized for being an interpreted language, and can be slow compared to compiled languages such as C. However, the use of bytecode compilation and the fact that much of the heavy lifting is done by library code means that Python's performance is often surprisingly good—and there are many things you can do to improve the performance of your programs if you need to.

Open source versions of the Python interpreter are freely available for all major operating systems. Python is eminently suitable for all sorts of programming, from quick one-off scripts to building huge and complex systems. It can even be run in interactive (command-line) mode, allowing you to type in one-off commands and short programs and immediately see the results. This is ideal for doing quick calculations or figuring out how a particular library works.

One of the first things a developer notices about Python compared with other languages such as Java or C++ is how expressive the language is: what may take 20 or 30 lines of code in Java can often be written in half a dozen lines of code in Python. For example, imagine that you wanted to print a sorted list of the words that occur in a given piece of text. In Python, this is easy:

words = set(text.split())
for word in sorted(words):
    print(word)

Implementing this kind of task in other languages is often surprisingly difficult.

While the Python language itself makes programming quick and easy, allowing you to focus on the task at hand, the Python Standard Library makes programming even more efficient. This library makes it easy to do things such as converting date and time values, manipulating strings, downloading data from web sites, performing complex maths, working with e-mail messages, encoding and decoding data, XML parsing, data encryption, file manipulation, compressing and decompressing files, working with databases—the list goes on. What you can do with the Python Standard Library is truly amazing.

As well as the built-in modules in the Python Standard Library, it is easy to download and install custom modules, which could be written either in Python or C. The Python Package Index (http://pypi.python.org) provides thousands of additional modules that you can download and install. And if this isn't enough, many other systems provide Python bindings to allow you to access them directly from within your programs. We will be making heavy use of Python bindings in this book.

Python is in many ways an ideal programming language. Once you are familiar with the language and have used it a few times, you'll find it incredibly easy to write programs to solve various tasks. Rather than getting buried in a morass of type definitions and low-level string manipulation, you can simply concentrate on what you want to achieve. You almost end up thinking directly in Python code. Programming in Python is straightforward, efficient, and, dare I say it, fun.

Python 3

There are two main flavors of Python in use today: the Python 2.x series has been around for many years and is still widely used today, while Python 3.x isn't backward compatible with Python 2 and is becoming more and more popular as it is seen as the main version of Python going forward.

One of the main things holding back the adoption of Python 3 is the lack of support for third-party libraries. This has been particularly acute for Python libraries used for geospatial development, which are often dependent on individual developers or have requirements that were not compatible with Python 3 for quite a long time. However, all the major libraries used in this book can now be run using Python 3, and so all the code examples in this book have been converted to use Python 3 syntax.

If your computer runs Linux or Mac OS X, then you can use Python 3 with all these libraries directly. If, however, your computer runs MS Windows, then Python 3 compatibility is more problematic. In this case, you have two options: you can attempt to compile the libraries yourself to work with Python 3 or you can revert to using Python 2 and make adjustments to the example code as required. Fortunately, the syntax differences between Python 2 and Python 3 are quite straightforward, so not many changes will be required if you do choose to use Python 2.x rather than Python 3.x.

 

Geospatial development


The term geospatial refers to finding information that is located on the earth's surface. This can include, for example, the position of a cellphone tower, the shape of a road, or the outline of a country:

Geospatial data often associates some piece of information with a particular location. For example, the following map, taken from http://opendata.zeit.de/nuclear-reactors-usa, shows how many people live within 50 miles of a nuclear reactor within the eastern United States:

Geospatial development is the process of writing computer programs that can access, manipulate, and display this type of information.

Internally, geospatial data is represented as a series of coordinates, often in the form of latitude and longitude values. Additional attributes, such as temperature, soil type, height, or the name of a landmark, are also often present. There can be many thousands (or even millions) of data points for a single set of geospatial data. For example, the following outline of New Zealand consists of almost 12,000 individual data points:

Because so much data is involved, it is common to store geospatial information within a database. A large part of this book will be concerned with how to store your geospatial information in a database and access it efficiently.

Geospatial data comes in many different forms. Different Geographical Information Systems vendors have produced their own file formats over the years, and various organizations have also defined their own standards. It is often necessary to use a Python library to read files in the correct format when importing geospatial data into your database.

Unfortunately, not all geospatial data points are compatible. Just like a distance value of 2.8 can have very different meanings depending on whether you are using kilometers or miles, a given coordinate value can represent any number of different points on the curved surface of the earth, depending on which projection has been used.

A projection is a way of representing the earth's surface in two dimensions. We will look at projections in more detail in Chapter 2, GIS, but for now, just keep in mind that every piece of geospatial data has a projection associated with it. To compare or combine two sets of geospatial data, it is often necessary to convert the data from one projection to another.

Note

Latitude and longitude values are sometimes referred to as unprojected coordinates. We'll learn more about this in the next chapter.

In addition to the prosaic tasks of importing geospatial data from various external file formats and translating data from one projection to another, geospatial data can also be manipulated to solve various interesting problems. Obvious examples include the task of calculating the distance between two points, calculating the length of a road, or finding all data points within a given radius of a selected point. We will be using Python libraries to solve all of these problems and more.

Finally, geospatial data by itself is not very interesting. A long list of coordinates tells you almost nothing; it isn't until those numbers are used to draw a picture that you can make sense of it. Drawing maps, placing data points onto a map, and allowing users to interact with maps are all important aspects of geospatial development. We will be looking at all of these in later chapters.

 

Applications of geospatial development


Let's take a brief look at some of the more common geospatial development tasks you might encounter.

Analysing geospatial data

Imagine that you have a database containing a range of geospatial data for San Francisco. This database might include geographical features, roads, the location of prominent buildings, and other man-made features such as bridges, airports, and so on.

Such a database can be a valuable resource for answering various questions such as the following:

  • What's the longest road in Sausalito?

  • How many bridges are there in Oakland?

  • What is the total area of Golden Gate Park?

  • How far is it from Pier 39 to Coit Tower?

Many of these types of problems can be solved using tools such as the PostGIS spatially-enabled database toolkit. For example, to calculate the total area of Golden Gate Park, you might use the following SQL query:

select ST_Area(geometry) from features
  where name = "Golden Gate Park";

To calculate the distance between two locations, you first have to geocode the locations to obtain their latitude and longitude values. There are various ways to do this; one simple approach is to use a free geocoding web service such as the following:

http://nominatim.openstreetmap.org/search?format=json&q=Pier 39,San Francisco, CA

This returns (among other things) a latitude value of 37.8101274 and a longitude value of -122.4104622 for Pier 39 in San Francisco.

Note

These latitude and longitude values are in decimal degrees. If you don't know what these are, don't worry; we'll talk about decimal degrees in Chapter 2, GIS.

Similarly, we can find the location of Coit Tower in San Francisco using this query:

http://nominatim.openstreetmap.org/search?format=json&q=Coit Tower, San Francisco, CA

This returns a latitude value of 37.80237485 and a longitude value of -122.405832766082.

Now that we have the coordinates for the two desired locations, we can calculate the distance between them using the pyproj Python library:

Tip

If you want to run this example, you will need to install the pyproj library. We will look at how to do this in Chapter 3, Python Libraries for Geospatial Development.

import pyproj

lat1,long1 = (37.8101274,-122.4104622)
lat2,long2 = (37.80237485,-122.405832766082)

geod = pyproj.Geod(ellps="WGS84")
angle1,angle2,distance = geod.inv(long1, lat1, long2, lat2)

print("Distance is {:0.2f} meters".format(distance))

This prints the distance between the two points:

Distance is 952.17 meters

Note

Don't worry about the WGS84 reference at this stage; we'll look at what this means in Chapter 2, GIS.

Of course, you wouldn't normally do this sort of analysis on a one-off basis like this—it's much more common to create a Python program that will answer these sorts of questions for any desired set of data. You might, for example, create a web application that displays a menu of available calculations. One of the options in this menu might be to calculate the distance between two points; when this option is selected, the web application would prompt the user to enter the two locations, attempt to geocode them by calling an appropriate web service (and display an error message if a location couldn't be geocoded), then calculate the distance between the two points using pyproj, and finally display the results to the user.

Alternatively, if you have a database containing useful geospatial data, you could let the user select the two locations from the database rather than having them type in arbitrary location names or street addresses.

However you choose to structure it, performing calculations like this will often be a major part of your geospatial application.

Visualizing geospatial data

Imagine you wanted to see which areas of a city are typically covered by a taxi during an average working day. You might place a GPS recorder in a taxi and leave it to record the taxi's position over several days. The result would be a series of timestamps and latitude and longitude values, like this:

2010-03-21 9:15:23  -38.16614499  176.2336626
2010-03-21 9:15:27  -38.16608632  176.2335635
2010-03-21 9:15:34  -38.16604198  176.2334771
2010-03-21 9:15:39  -38.16601507  176.2333958
...

By themselves, these raw numbers tell you almost nothing. But when you display this data visually, the numbers start to make sense:

Tip

Detailed steps to download the code bundle are mentioned in the Preface of this book. Please have a look.

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Python-Geospatial-Development-Third-Edition. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

You can immediately see that the taxi tends to go along the same streets again and again, and if you draw this data as an overlay on top of a street map, you can see exactly where the taxi has been:

Street map courtesy of http://openstreetmap.org

While this is a simple example, visualization is a crucial aspect of working with geospatial data. How data is displayed visually, how different data sets are overlaid, and how the user can manipulate data directly in a visual format are all going to be major topics in this book.

Creating a geospatial mash-up

The concept of a mash-up has become popular in recent years. Mash-ups are applications that combine data and functionality from more than one source. For example, a typical mash-up might collect details of houses for rent in a given city and plot the location of each rental on a map, like this:

Image courtesy of http://housingmaps.com

The Google Maps API has been immensely popular in creating these types of mash-ups. However, Google Maps has some serious licensing and other limitations. It is not the only option, however tools such as Mapnik, OpenLayers, and MapServer, to name a few, also allow you to create mash-ups that overlay your own data onto a map.

Most of these mash-ups run as web applications across the Internet, running on a server that can be accessed by anyone who has a web browser. Sometimes, the mash-ups are private, requiring password access, but usually, they are publicly available and can be used by anyone. Indeed, many businesses (such as the housing maps site shown in the previous screen snapshot) are based on freely-available geospatial mash-ups.

 

Recent developments


A decade ago, geospatial development was vastly more limited than it is today. Professional (and hugely expensive) geographical information systems were the norm for working with and visualizing geospatial data. Open-source tools, where they were available, were obscure and hard to use. What is more, everything ran on the desktop—the concept of working with geospatial data across the Internet was no more than a distant dream.

In 2005, Google released two products that completely changed the face of geospatial development: Google Maps and Google Earth made it possible for anyone with a web browser or desktop computer to view and work with geospatial data. Instead of requiring expert knowledge and years of practice, even a four-year-old could instantly view and manipulate interactive maps of the world.

Google's products are not perfect: the map projections are deliberately simplified, leading to errors and problems with displaying overlays. These products are only free for non-commercial use, and they include almost no ability to perform geospatial analysis. Despite these limitations, they have had a huge effect on the field of geospatial development. People became aware of what is possible, and the use of maps and their underlying geospatial data has become so prevalent that even cellphones now commonly include built-in mapping tools.

The Global Positioning System (GPS) has also had a major influence on geospatial development. Geospatial data for streets and other man-made and natural features used to be an expensive and tightly-controlled resource, often created by scanning aerial photographs and then manually drawing an outline of a street or coastline over the top to digitize the required features. With the advent of cheap and readily-available portable GPS units, as well as phones which have GPS built in, anyone who wishes to can now capture their own geospatial data. Indeed, many people have made a hobby of recording, editing, and improving the accuracy of street and topological data, which is then freely shared across the Internet. All this means that you're not limited to recording your own data or purchasing data from a commercial organization; volunteered information is now often as accurate and useful as commercially-available data, and may well be suitable for your geospatial application.

The open source software movement has also had a major influence on geospatial development. Instead of relying on commercial toolsets, it is now possible to build complex geospatial applications entirely out of freely-available tools and libraries. Because the source code for these tools is often available, developers can improve and extend these toolkits, fixing problems and adding new features for the benefit of everyone. Tools such as PROJ.4, PostGIS, OGR, and GDAL are all excellent geospatial toolkits that are benefactors of the open source movement. We will be making use of all these tools throughout this book.

As well as standalone tools and libraries, a number of geospatial application programming interfaces (APIs) have become available. Google has provided a number of APIs that can be used to include maps and perform limited geospatial analysis within a web site. Other sites, such as the OpenStreetMap geocoder we used earlier, allow you to perform various geospatial tasks that would be difficult to do if you were limited to using your own data and programming resources.

As more and more geospatial data becomes available from an increasing number of sources, and as the number of tools and systems that can work with this data also increases, it has become increasingly important to define standards for geospatial data. The Open Geospatial Consortium (http://www.opengeospatial.org) is an international standards organization that aims to do precisely this: provide a set of standard formats and protocols for sharing and storing geospatial data. These standards, including GML, KML, GeoRSS, WMS, WFS, and WCS, provide a shared language in which geospatial data can be expressed. Tools such as commercial and open source GIS systems, Google Earth, web-based APIs, and specialized geospatial toolkits such as OGR are all able to work with these standards. Indeed, an important aspect of a geospatial toolkit is the ability to understand and translate data between these various formats.

As devices with built-in GPS receivers have become more ubiquitous, it has become possible to record your location data while performing another task. Geolocation, the act of recording your location while you are doing something else, is becoming increasingly common. The Twitter social networking service, for example, now allows you to record and display your current location when you enter a status update. As you approach your office, sophisticated to-do list software can now automatically hide any tasks that can't be done at that location. Your phone can also tell you which of your friends are nearby, and search results can be filtered to only show nearby businesses.

All of this is simply the continuation of a trend that started when GIS systems were housed on mainframe computers and operated by specialists who spent years learning about them. Geospatial data and applications have been "democratized" over the years, making them available in more places, to more people. What was possible only in a large organization can now be done by anyone using a handheld device. As technology continues to improve and tools become more powerful, this trend is sure to continue.

 

Summary


In this chapter, we briefly introduced the Python programming language and the main concepts behind geospatial development. We saw that Python is a very high-level language and that the availability of third-party libraries for working with geospatial data makes it eminently suited to the task of geospatial development. We learned that the term geospatial data refers to finding information that is located on the earth's surface using coordinates, and the term "geospatial development" refers to the process of writing computer programs that can access, manipulate, and display geospatial data.

We then looked at the types of questions that can be answered by analyzing geospatial data, saw how geospatial data can be used for visualization, and learned about geospatial mash-ups, which combine data (often geospatial data) in useful and interesting ways.

Next, we learned how Google Maps, Google Earth, and the development of cheap and portable GPS units have "democratized" geospatial development. We saw how the open source software movement has produced a number of high-quality, freely available tools for geospatial development and looked at how various standards organizations have defined formats and protocols for sharing and storing geospatial data.

Finally, we saw how geolocation is being used to capture and work with geospatial data in surprising and useful ways.

In the next chapter, we will look in more detail at traditional geographic information systems including a number of important concepts that you need to understand in order to work with geospatial data. Different geospatial formats will be examined, and we will finish by using Python to perform various calculations using geospatial data.

About the Author
  • Erik Westra

    Erik Westra has been a professional software developer for over 25 years, and has worked almost exclusively in Python for the past decade. Eriks early interest in Graphical User Interface design led to the development of one of the most advanced urgent courier dispatch systems used by messenger and courier companies worldwide. In recent years, Erik has been involved in the design and implementation of systems matching seekers with providers of goods and services across a range of geographical areas, as well as real-time messaging, payment, and identity systems. This work has included the creation of real-time geocoders and map-based views of constantly changing data. Erik is based in New Zealand, and works for companies worldwide. Erik is the author of the following Packt books: Python Geospatial Development (third edition), Python Geospatial Analysis, Building Mapping Applications with QGIS, and Modular Programming with Python. Erik has also authored the video course entitled Introduction to QGIS Python Programming.

    Browse publications by this author
Latest Reviews (5 reviews total)
Book is nicely written with good information. Tried to download the e-pub version but got the pdf version. Not sure what happened.
New to geospatial apps book went into lots of detail about various sources of info and how to use them.
Python Geospatial Development - Third Edition
Unlock this book and the full library FREE for 7 days
Start now