ArcGIS Blueprints

4.5 (2 reviews total)
By Eric Pimpler
  • Instant online access to over 8,000+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Extracting Real-Time Wildfire Data from ArcGIS Server with the ArcGIS REST API

About this book

This book is an immersive guide to take your ArcGIS Desktop application development skills to the next level

It starts off by providing detailed description and examples of how to create ArcGIS Desktop Python toolboxes that will serve as containers for many of the applications that you will build. We provide several practical projects that involve building a local area/community map and extracting wildfire data. You will then learn how to build tools that can access data from ArcGIS Server using the ArcGIS REST API. Furthermore, we deal with the integration of additional open source Python libraries into your applications, which will help you chart and graph advanced GUI development; read and write JSON, CSV, and XML format data sources; write outputs to Google Earth Pro, and more. Along the way, you will be introduced to advanced ArcPy Mapping and ArcPy Data Access module techniques and use data-driven Pages to automate the creation of map books.

Finally, you will learn advanced techniques to work with video and social media feeds. By the end of the book, you will have your own desktop application without having spent too much time learning sophisticated theory.

Publication date:
December 2015
Publisher
Packt
Pages
378
ISBN
9781785286223

 

Chapter 1. Extracting Real-Time Wildfire Data from ArcGIS Server with the ArcGIS REST API

The ArcGIS platform, which contains a number of different products, including ArcGIS Desktop, ArcGIS Pro, ArcGIS for Server, and ArcGIS Online, provides a robust environment to perform geographic analysis and mapping. The content produced by this platform can be integrated using the ArcGIS REST API and a programming language such as Python. Many of the applications we'll build in this book use the ArcGIS REST API as the bridge to exchange information between software products.

We're going to start by developing a simple ArcGIS Desktop custom script tool in ArcToolbox that connects to an ArcGIS Server map service to retrieve real-time wildfire information. The wildfire information will be retrieved from a USGS map service that provides real-time wildfire data. For this chapter and all other chapters in this book, the reader is expected to have intermediate-level experience of Python and ArcPy. Ideally, you should be running version 10.3 or 10.2 of ArcGIS Desktop. Previous versions of ArcGIS Desktop have some significant differences that may cause problems in the development of some applications in the book.

We'll use the ArcGIS REST API and the Python requests module to connect to the map service and request the data. The response from the map service will contain data that will be written to a feature class stored in a local geodatabase using the ArcPy data access module.

This will all be accomplished with a custom script tool attached to an ArcGIS Python Toolbox. ArcGIS Python toolboxes are relatively new; they were first introduced in version 10.1 of ArcGIS Desktop. They provide a Python-centric method to create custom toolboxes and tools. The older method to create toolboxes in ArcGIS Desktop, while still relevant, requires a combination of Python and a wizard-based approach to create tools.

In this chapter, we will cover the following topics:

  • ArcGIS Desktop Python's toolboxes

  • The ArcGIS Server map and feature services

  • The Python requests module

  • The Python JSON module

  • The ArcGIS REST API

  • The ArcPy data access module that is arcpy.da

A general overview of the Python libraries for ArcGIS is provided in the appendix of this book. It is recommended that you read this chapter before continuing with the appendix and other chapters.

 

Design


Before we start building the application, we'll spend some time planning what we'll build. This is a fairly simple application, but it serves to illustrate how ArcGIS Desktop and ArcGIS Server can be easily integrated using the ArcGIS REST API. In this application, we'll build an ArcGIS Python Toolbox that serves as a container for a single tool called USGSDownload. The USGSDownload tool will use the Python requests, JavaScript Object Notation (JSON), and ArcPy da modules to request real-time wildfire data from a USGS map service. The response from the map service will contain information including the location of the fire, the name of the fire, and some additional information that will then be written to a local geodatabase.

The communication between the ArcGIS Desktop Python Toolbox and the ArcGIS Server map service will be accomplished through the ArcGIS REST API and the Python language.

Let's get started and build the application.

 

Creating the ArcGIS Desktop Python Toolbox


There are two ways to create toolboxes in ArcGIS: script tools in custom toolboxes and script tools in Python toolboxes. Python toolboxes encapsulate everything in one place: parameters, validation code, and source code. This is not the case with custom toolboxes, which are created using a wizard and a separate script that processes the business logic.

A Python Toolbox functions like any other toolbox in ArcToolbox, but it is created entirely in Python and has a file extension of .pyt. It is created programmatically as a class named Toolbox. In this section, you will learn how to create a Python Toolbox and add a tool. You'll only create the basic structure of the toolbox and tool that will ultimately connect to an ArcGIS Server map service containing the wildfire data. In a later section, you'll complete the functionality of the tool by adding code that connects to the map service, downloads the current data, and inserts it into a feature class. Take a look at the following steps:

  1. Open ArcCatalog: You can create a Python Toolbox in a folder by right-clicking on the Folder and navigating to New | Python Toolbox. In ArcCatalog, there is a folder called Toolboxes, and inside it, there is a My Toolboxes folder, as shown in the following screenshot. Right-click on this folder and navigate to New | Python Toolbox.

  2. The name of the toolbox is controlled by the filename. Name the toolbox InsertWildfires.pyt.

  3. The Python Toolbox file (.pyt) can be edited in any text or code editor. By default, the code will open in Notepad. However, you will want to use a more advanced Python development environment, such as PyScripter, IDLE, and so on. You can change this by setting the default editor for your script by navigating to Geoprocessing | Geoprocessing Options and going to the Editor section. In the following screenshot, you'll notice that I have set my editor to PyScripter, which is my preferred environment. You may want to change this to IDLE or whichever development environment you are currently using.

  4. For example, to find the path to the executable for the IDLE development environment, you can navigate to Start | All Programs | ArcGIS | Python 2.7 | IDLE. Right-click on IDLE and select Properties to display the properties window. Inside the Target textbox, you should see a path to the executable, as shown in the following screenshot. You will want to copy and paste only the actual path starting with C:\Python27 and not the quotes that surround the path.

  5. Copy and paste the path into the Editor and Debugger sections inside the Geoprocessing Options dialog box.

  6. Right-click on InsertWildfires.pyt and select Edit. This will open the development environment you defined earlier, as shown in the following screenshot. Your environment will vary depending upon the editor that you have defined:

  7. Remember that you will not be changing the name of the class, which is Toolbox. However, you will have to rename the Tool class to reflect the name of the tool you want to create. Each tool will have various methods, including __init__(), which is the constructor for the tool along with getParameterInfo(), isLicensed(), updateParameters(), updateMessages(), and execute(). You can use the __init__() method to set initialization properties, such as the tool's label and description. Find the class named Tool in your code, and change the name of this tool to USGSDownload, and set the label and description properties:

    class USGSDownload(object):
      def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "USGS Download"
        self.description = "Download from USGS ArcGIS Server instance"
        self.canRunInBackground = False

    Tip

    Downloading the example code.

    You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you

    You can use the Tool class as a template for other tools that you'd like to add to the toolbox by copying and pasting the class and its methods. We're not going to do that in this chapter, but I wanted you to be aware of this.

  8. You will need to add each tool to the tools property (the Python list) in the Toolbox class. Add the USGSDownload tool, as shown in the following code snippet:

      def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of the .pyt file."""
        self.label = "Toolbox"
        self.alias = ""
        
        #List of tool classes associated with this toolbox
        self.tools = [USGSDownload]
    
  9. Save your code.

  10. When you close the code editor, your toolbox should be automatically refreshed. You can also manually refresh a toolbox by right-clicking on the InsertWildfires.pyt and selecting Refresh. If a syntax error occurs in your code, the toolbox icon will change, as shown in the following screenshot. Note the red X next to the toolbox. Your tool may not visible inside the toolbox either. If you've coded everything correctly, you will not see this icon, as shown in the following screenshot:

  11. To see the error, right-click on InsertWildfires.pyt and select Check Syntax.

    Assuming that you don't have any syntax errors, you should see the following screenshot of the Toolbox/Tool structure:

Working with tool parameters

Almost all tools have parameters. The parameter values will be set by the end user with the Tool dialog box, or they will be hardcoded within the script. When the tool is executed, the parameter values are sent to your tool's source code. Your tool reads these values and proceeds with its work. You use the getParameterInfo() method to define the parameters for your tool. Individual parameter objects are created as part of this process. If necessary, open InsertWildfires.pyt in your code editor and find the getParameterInfo() function in the USGSDownload class. Add the following parameters, and then we'll discuss what the code is doing:

def getParameterInfo(self):

        """Define parameter definitions"""
        # First parameter
        param0 = arcpy.Parameter(displayName="ArcGIS Server Wildfire URL",
                        name="url",
                        datatype="GPString",
                        parameterType="Required",
                        direction="Input")
        param0.value = "http://wildfire.cr.usgs.gov/arcgis/rest/services/geomac_dyn/MapServer/0/query"

        # Second parameter
        param1 = arcpy.Parameter(displayName="Output Feature Class",
                        name="out_fc",
                        datatype="DEFeatureClass",
                        parameterType="Required",
                        direction="Input")

Each parameter is created using arcpy.Parameter and is passed a number of arguments that define the object. For the first Parameter object (param0), we are going to capture a URL to an ArcGIS Server map service containing real-time wildfire data. We give it a display name ArcGIS Server Wildfire URL, which will be displayed on the dialog box for the tool. A name for the parameter, a datatype, a parameter type, and a direction. In the case of the first parameter (param0), we also assign an initial value, which is the URL to an existing map service containing the wildfire data. For the second parameter, we're going to define an output feature class where the wildfire data that is read from the map service will be written. An empty feature class to store the data has already been created for you.

Next we'll add both the parameters to a Python list called params and return, to the list to the calling function. Add the following code:

def getParameterInfo(self):

        """Define parameter definitions"""
        # First parameter
        param0 = arcpy.Parameter(displayName="ArcGIS Server Wildfire URL",
                        name="url",
                        datatype="GPString",
                        parameterType="Required",
                        direction="Input")
        param0.value = "http://wildfire.cr.usgs.gov/arcgis/rest/services/geomac_dyn/MapServer/0/query"

        # Second parameter
        param1 = arcpy.Parameter(displayName="Output Feature Class",
                        name="out_fc",
                        datatype="DEFeatureClass",
                        parameterType="Required",
                        direction="Input")

        params = [param0, param1]
        return params

Tool execution

The main work of a tool is done inside the execute() method. This is where the geoprocessing of your tool takes place. The execute() method, as shown in the following code snippet, can accept a number of arguments, including the tools self, parameters, and messages:

def execute(self, parameters, messages):
        """The source code of the tool."""
        return

To access the parameter values that are passed into the tool, you can use the valueAsText() method. The following steps will guide you through how to add and execute the execute() method:

  1. Find the execute() function in the USGSDownload class and add the following code snippet to access the parameter values that will be passed into your tool. Remember from a previous step that the first parameter will contain a URL to a map service containing the wildfire data and the second parameter will be the output feature class where the data will be written:

    def execute(self, parameters, messages):
            inFeatures = parameters[0].valueAsText
            outFeatureClass = parameters[1].valueAsText

    At this point, you have created a Python Toolbox, added a tool, defined the parameters for the tool, and created variables that will hold the parameter values that the end user has defined. Ultimately, this tool will use the URL that is passed to the tool to connect to an ArcGIS Server map service, download the current wildfire data, create a feature class, and write the wildfire data to the feature class. However, we're going to save the geoprocessing tasks for later. For now, I just want you to print out the values of the parameters that have been entered so that we know that the structure of the tool is working correctly. Print this information using the following code:

    def execute(self, parameters, messages):
            inFeatures = parameters[0].valueAsText
            arcpy.AddMessage(inFeatures)
            outFeatureClass = parameters[1].valueAsText
            arcpy.AddMessage(outFeatureClass)
  2. Execute the tool by double-clicking on USGS Download from the InsertWildfires.pyt toolbox that you've created. You should see the following dialog box:

  3. Leave the default URL parameter as it is and select an output feature class. You'll want to click on the Browse button and then navigate to the C:\ArcGIS_Blueprint_Python\Data\WildfireData folder and select the WildlandFires geodatabase. Inside, there is an empty feature class called CurrentFires. Select this feature class.

  4. Now click on OK. The progress dialog box will contain the parameter values that you passed in. Your output feature class will probably be different than mine, as shown in the following screenshot:

We'll complete the actual geoprocessing of this tool in the next section.

 

Populating the feature class


In the previous section, you learned how to create a Python Toolbox and add tools. You created a new toolbox called InsertWildfires and added a tool called USGS Download. However, in that exercise, you didn't complete the geoprocessing operations that connect to an ArcGIS Server map service, query the service for current wildfires, and populate the feature class from the data pulled from the map service query. You'll complete these steps in the following section.

Installing pip and the requests module

This section of the application uses the Python requests module. If you don't already have this module installed on your computer, you will need to do this at this time using pip.

The pip is a package manager that serves as a repository and installation manager for Python modules. It makes finding and installing Python modules much easier. There are several steps that you'll need to follow in order to install pip and the requests module. Instructions to install pip and the requests module are provided in the first few steps:

  1. Open the Environment Variables dialog box in Windows. The easiest way to display this dialog box is to go to Start and then type Environment Variables in the search box. This should display an Edit environment variables for your account entry. Select this item.

  2. If you don't see a variable called PATH, click on the New button to create one. If you already have a PATH variable, you can just add to the existing content. Select the PATH variable, and click on Edit, and then set the value to C:\Python27\ArcGIS10.3; C:\Python27\ArcGIS10.3\Scripts. The first path will provide a reference to the location of the Python executable and the second will reference the location of pip when it is installed. This makes it possible to run Python and pip from the Command Prompt in Windows.

  3. Click on OK and then click on OK again to save the changes.

  4. Next, we'll install pip if you haven't already done so in the past. You need to install pip before you can install requests. In a web browser, go to https://pip.pypa.io/en/latest/installing.html and scroll down to the install pip section. Right-click on get-pip.py and select Save Link As or something similar. This will vary depending upon the browser you are using. Save it to your C:\ArcGIS_Blueprint_Python folder.

  5. Open the Command Prompt in Windows, type python C:\ArcGIS_Blueprint_Python\get-pip.py, and press Enter on your keyboard. This will install pip.

  6. In the Command Prompt, type pip install requests and press Enter on your keyboard. This will install the requests module.

  7. Close the Command Prompt.

Requesting data from ArcGIS Server

In the following steps, we will learn how to request data from ArcGIS Server:

  1. Open ArcCatalog and navigate to the location where you've created your Python Toolbox, it would look like following screenshot:

  2. Right-click on InsertWildfires.pyt and select Edit to display the code for the toolbox.

  3. First, we'll clean up a little by removing the AddMessage() functions. Clean up your execute() method so that it appears as follows:

    def execute(self, parameters, messages):
            inFeatures = parameters[0].valueAsText
            outFeatureClass = parameters[1].valueAsText
  4. Next, add the code that connects to the wildfire map service, to perform a query. In this step, you will also define the QueryString parameters that will be passed into the query of the map service. First, import the requests and json modules:

    import arcpy
    import requests, json
    
    
    class Toolbox(object):
        def __init__(self):
            """Define the toolbox (the name of the toolbox is the name of the
            .pyt file)."""
            self.label = "Toolbox"
            self.alias = ""
    
            # List of tool classes associated with this toolbox
            self.tools = [USGSDownload]
  5. Then, create the agisurl and json_payload variables that will hold the QueryString parameters. Note that, in this case, we have defined a WHERE clause so that only wildfires where the acres are greater than 5 will be returned. The inFeatures variable holds the ArcGIS Server Wildfire URL:

        def execute(self, parameters, messages):
            inFeatures = parameters[0].valueAsText
            outFeatureClass = parameters[1].valueAsText
    
            agisurl = inFeatures
            json_payload = { 'where': 'acres > 5', 'f': 'pjson',
            'outFields': 'latitude,longitude,incidentname,acres' }
    
  6. Submit the request to the ArcGIS Server instance; the response should be stored in a variable called r. Print a message to the dialog box indicating the response as:

    def execute(self, parameters, messages):
            inFeatures = parameters[0].valueAsText
            outFeatureClass = parameters[1].valueAsText
    
            agisurl = inFeatures
            json_payload = { 'where': 'acres > 5', 'f': 'pjson', 'outFields': 'latitude,longitude,incidentname,acres' }
    
            r = requests.get(agisurl, params=json_payload)
            arcpy.AddMessage("The response: " + r.text)
    
  7. Test the code to make sure that we're on the right track. Save the file and refresh InsertWildfires in ArcCatalog. Execute the tool and leave the default URL. If everything is working as expected, you should see a JSON object output to the progress dialog box. Your output will probably vary from the following screenshot:

  8. Return to the execute() method and convert the JSON object to a Python dictionary using the json.loads() method:

    def execute(self, parameters, messages):
            inFeatures = parameters[0].valueAsText
            outFeatureClass = parameters[1].valueAsText
    
            agisurl = inFeatures
            json_payload = { 'where': 'acres > 5', 'f': 'pjson', 'outFields': 'latitude,longitude,incidentname,acres' }
    
            r = requests.get(inFeatures, params=json_payload)
            arcpy.AddMessage("The response: " + r.text)
    
            decoded = json.loads(r.text)
    

Inserting data in a feature class with the ArcPy data access module

The following steps will guide you, to insert data in a feature class with the help of the ArcPy data access module:

  1. Now, we'll use the ArcPy data access module, that is Arcpy.da, to create an InsertCursor object by passing the output feature class defined in the tool dialog box along with the fields that will be populated:

        def execute(self, parameters, messages):
            inFeatures = parameters[0].valueAsText
            outFeatureClass = parameters[1].valueAsText
    
            agisurl = inFeatures
            json_payload = { 'where': 'acres > 5', 'f': 'pjson', 'outFields': 'latitude,longitude,fire_name,acres' }
    
            r = requests.get(inFeatures, params=json_payload)
            arcpy.AddMessage("The response: " + r.text)
    
            decoded = json.loads(r.text)
            cur = arcpy.da.InsertCursor(outFeatureClass, ("[email protected]", "NAME", "ACRES"))
    
  2. Create a For loop that you can see in the following code, and then we'll discuss what this section of code accomplishes:

    def execute(self, parameters, messages):
            inFeatures = parameters[0].valueAsText
            outFeatureClass = parameters[1].valueAsText
    
            agisurl = inFeatures
            json_payload = { 'where': 'acres > 5', 'f': 'pjson', 'outFields': 'latitude,longitude,fire_name,acres' }
    
            r = requests.get(inFeatures, params=json_payload)
            arcpy.AddMessage("The response: " + r.text)
    
            decoded = json.loads(r.text)
            cur = arcpy.da.InsertCursor(outFeatureClass, ("[email protected]", "NAME", "ACRES"))
            cntr = 1
            for rslt in decoded['features']:
                fireName = rslt['attributes']['incidentname']
                latitude = rslt['attributes']['latitude']
                longitude = rslt['attributes']['longitude']
                acres = rslt['attributes']['acres']
                cur.insertRow([(longitude,latitude),fireName, acres])
                arcpy.AddMessage("Record number: " + str(cntr) + " written to feature class")
                cntr = cntr + 1
            del cur
    

    The first line simply creates a counter that will be used to display the progress information in the Progress Dialog box. We then start a For loop that loops through each of the features (wildfires) that have been returned. The decoded variable is a Python dictionary. Inside the For loop, we retrieve the wildfire name, latitude, longitude, and acres from the attributes dictionary. Finally, we call the insertRow() method to insert a new row into the feature class along with the wildfire name and acres as attributes. The progress information is written to the Progress Dialog box and the counter is updated.

  3. Save the file and refresh your Python Toolbox.

  4. Double-click on the USGS Download tool.

  5. Leave the default URL and select the CurrentFires feature class in the WildlandFires geodatabase. The CurrentFires feature class is empty and has fields for NAMES and ACRES:

  6. Click on OK to execute the tool. The number of features written to the feature class will vary depending upon the current wildfire activity. Most of the time, there is at least a little activity, but it is possible that there wouldn't be any wildfires in the U.S. as shown in the following screenshot:

  7. View the feature class in ArcMap. To view the feature class in the following screenshot, I've plotted the points along with a Basemap topography layer. Your data will almost certainly be different than mine as we are pulling real-time data:

 

Summary


Integrating ArcGIS Desktop and ArcGIS Server is easily accomplished using the ArcGIS REST API and the Python programming language. In this chapter, we created an ArcGIS Python Toolbox containing a tool that connects to an ArcGIS Server map service containing real-time wildfire information and hosted by the USGS. The connection was accomplished through the use of the Python request module that we used in order to submit a request and handle the response. Finally, we used the ArcPy data access module to write this information to a local geodatabase.

In the next chapter, we'll continue working with ArcGIS Python toolboxes, and you'll also learn how to read CSV files with the Python CSV module, insert data into a feature class, and use the arcpy.mapping module to work with time-enabled data.

About the Author

  • Eric Pimpler

    Eric Pimpler is the founder and owner of GeoSpatial Training Services and has over 20 years of experience in implementing and teaching GIS solutions using Esri, Google Earth/Maps, and open source technology. Currently, he focuses on ArcGIS scripting with Python and the development of custom ArcGIS Server web and mobile applications using JavaScript. He is the author of Programming ArcGIS 10.1 with Python Cookbook. Eric has a bachelor's degree in geography from Texas A&M University and a master's degree in applied geography with a specification in GIS from Texas State University.

    Browse publications by this author

Latest Reviews

(2 reviews total)
Good
Excellent
Book Title
Access this book, plus 8,000 other titles for FREE
Access now