The ArcGIS platform, which contains a number of different products, including ArcGIS Desktop, ArcGIS Pro, ArcGIS for Server, and ArcGIS Online, provides a robust environment to perform geographic analysis and mapping. The content produced by this platform can be integrated using the ArcGIS REST API and a programming language such as Python. Many of the applications we'll build in this book use the ArcGIS REST API as the bridge to exchange information between software products.
We're going to start by developing a simple ArcGIS Desktop custom script tool in ArcToolbox that connects to an ArcGIS Server map service to retrieve real-time wildfire information. The wildfire information will be retrieved from a USGS map service that provides real-time wildfire data. For this chapter and all other chapters in this book, the reader is expected to have intermediate-level experience of Python and ArcPy. Ideally, you should be running version 10.3 or 10.2 of ArcGIS Desktop. Previous versions of ArcGIS Desktop have some significant differences that may cause problems in the development of some applications in the book.
We'll use the ArcGIS REST API and the Python requests module to connect to the map service and request the data. The response from the map service will contain data that will be written to a feature class stored in a local geodatabase using the ArcPy data access module.
This will all be accomplished with a custom script tool attached to an ArcGIS Python Toolbox. ArcGIS Python toolboxes are relatively new; they were first introduced in version 10.1 of ArcGIS Desktop. They provide a Python-centric method to create custom toolboxes and tools. The older method to create toolboxes in ArcGIS Desktop, while still relevant, requires a combination of Python and a wizard-based approach to create tools.
In this chapter, we will cover the following topics:
ArcGIS Desktop Python's toolboxes
The ArcGIS Server map and feature services
The Python requests module
The Python JSON module
The ArcGIS REST API
The
ArcPydata access module that isarcpy.da
A general overview of the Python libraries for ArcGIS is provided in the appendix of this book. It is recommended that you read this chapter before continuing with the appendix and other chapters.
Before we start building the application, we'll spend some time planning what we'll build. This is a fairly simple application, but it serves to illustrate how ArcGIS Desktop and ArcGIS Server can be easily integrated using the ArcGIS REST API. In this application, we'll build an ArcGIS Python Toolbox that serves as a container for a single tool called USGSDownload. The USGSDownload tool will use the Python requests, JavaScript Object Notation (JSON), and ArcPy da modules to request real-time wildfire data from a USGS map service. The response from the map service will contain information including the location of the fire, the name of the fire, and some additional information that will then be written to a local geodatabase.
The communication between the ArcGIS Desktop Python Toolbox and the ArcGIS Server map service will be accomplished through the ArcGIS REST API and the Python language.
Let's get started and build the application.
There are two ways to create toolboxes in ArcGIS: script tools in custom toolboxes and script tools in Python toolboxes. Python toolboxes encapsulate everything in one place: parameters, validation code, and source code. This is not the case with custom toolboxes, which are created using a wizard and a separate script that processes the business logic.
A Python Toolbox functions like any other toolbox in ArcToolbox, but it is created entirely in Python and has a file extension of .pyt. It is created programmatically as a class named Toolbox. In this section, you will learn how to create a Python Toolbox and add a tool. You'll only create the basic structure of the toolbox and tool that will ultimately connect to an ArcGIS Server map service containing the wildfire data. In a later section, you'll complete the functionality of the tool by adding code that connects to the map service, downloads the current data, and inserts it into a feature class. Take a look at the following steps:
Open ArcCatalog: You can create a Python Toolbox in a folder by right-clicking on the Folder and navigating to New | Python Toolbox. In ArcCatalog, there is a folder called Toolboxes, and inside it, there is a My Toolboxes folder, as shown in the following screenshot. Right-click on this folder and navigate to New | Python Toolbox.
The name of the toolbox is controlled by the filename. Name the toolbox
InsertWildfires.pyt.
The Python Toolbox file (
.pyt) can be edited in any text or code editor. By default, the code will open in Notepad. However, you will want to use a more advanced Python development environment, such as PyScripter, IDLE, and so on. You can change this by setting the default editor for your script by navigating to Geoprocessing | Geoprocessing Options and going to the Editor section. In the following screenshot, you'll notice that I have set my editor to PyScripter, which is my preferred environment. You may want to change this to IDLE or whichever development environment you are currently using.
For example, to find the path to the executable for the IDLE development environment, you can navigate to Start | All Programs | ArcGIS | Python 2.7 | IDLE. Right-click on IDLE and select Properties to display the properties window. Inside the Target textbox, you should see a path to the executable, as shown in the following screenshot. You will want to copy and paste only the actual path starting with
C:\Python27and not the quotes that surround the path.
Copy and paste the path into the Editor and Debugger sections inside the Geoprocessing Options dialog box.
Right-click on
InsertWildfires.pytand select Edit. This will open the development environment you defined earlier, as shown in the following screenshot. Your environment will vary depending upon the editor that you have defined:
Remember that you will not be changing the name of the class, which is
Toolbox.However, you will have to rename theToolclass to reflect the name of the tool you want to create. Each tool will have various methods, including__init__(), which is the constructor for the tool along withgetParameterInfo(),isLicensed(),updateParameters(),updateMessages(), andexecute(). You can use the__init__()method to set initialization properties, such as the tool's label and description. Find the class namedToolin your code, and change the name of this tool toUSGSDownload, and set thelabelanddescriptionproperties:class USGSDownload(object): def __init__(self): """Define the tool (tool name is the name of the class).""" self.label = "USGS Download" self.description = "Download from USGS ArcGIS Server instance" self.canRunInBackground = FalseTip
Downloading the example code.
You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you
You can use the
Toolclass as a template for other tools that you'd like to add to the toolbox by copying and pasting the class and its methods. We're not going to do that in this chapter, but I wanted you to be aware of this.You will need to add each tool to the
toolsproperty (the Python list) in theToolboxclass. Add theUSGSDownloadtool, as shown in the following code snippet:def __init__(self): """Define the toolbox (the name of the toolbox is the name of the .pyt file.""" self.label = "Toolbox" self.alias = "" #List of tool classes associated with this toolbox self.tools = [USGSDownload]Save your code.
When you close the code editor, your toolbox should be automatically refreshed. You can also manually refresh a toolbox by right-clicking on the InsertWildfires.pyt and selecting Refresh. If a syntax error occurs in your code, the toolbox icon will change, as shown in the following screenshot. Note the red X next to the toolbox. Your tool may not visible inside the toolbox either. If you've coded everything correctly, you will not see this icon, as shown in the following screenshot:
To see the error, right-click on InsertWildfires.pyt and select Check Syntax.
Assuming that you don't have any syntax errors, you should see the following screenshot of the Toolbox/Tool structure:
Almost all tools have parameters. The parameter values will be set by the end user with the Tool dialog box, or they will be hardcoded within the script. When the tool is executed, the parameter values are sent to your tool's source code. Your tool reads these values and proceeds with its work. You use the getParameterInfo() method to define the parameters for your tool. Individual parameter objects are created as part of this process. If necessary, open InsertWildfires.pyt in your code editor and find the getParameterInfo() function in the USGSDownload class. Add the following parameters, and then we'll discuss what the code is doing:
def getParameterInfo(self):
"""Define parameter definitions"""
# First parameter
param0 = arcpy.Parameter(displayName="ArcGIS Server Wildfire URL",
name="url",
datatype="GPString",
parameterType="Required",
direction="Input")
param0.value = "http://wildfire.cr.usgs.gov/arcgis/rest/services/geomac_dyn/MapServer/0/query"
# Second parameter
param1 = arcpy.Parameter(displayName="Output Feature Class",
name="out_fc",
datatype="DEFeatureClass",
parameterType="Required",
direction="Input")Each parameter is created using arcpy.Parameter and is passed a number of arguments that define the object. For the first Parameter object (param0), we are going to capture a URL to an ArcGIS Server map service containing real-time wildfire data. We give it a display name ArcGIS Server Wildfire URL, which will be displayed on the dialog box for the tool. A name for the parameter, a datatype, a parameter type, and a direction. In the case of the first parameter (param0), we also assign an initial value, which is the URL to an existing map service containing the wildfire data. For the second parameter, we're going to define an output feature class where the wildfire data that is read from the map service will be written. An empty feature class to store the data has already been created for you.
Next we'll add both the parameters to a Python list called params and return, to the list to the calling function. Add the following code:
def getParameterInfo(self):
"""Define parameter definitions"""
# First parameter
param0 = arcpy.Parameter(displayName="ArcGIS Server Wildfire URL",
name="url",
datatype="GPString",
parameterType="Required",
direction="Input")
param0.value = "http://wildfire.cr.usgs.gov/arcgis/rest/services/geomac_dyn/MapServer/0/query"
# Second parameter
param1 = arcpy.Parameter(displayName="Output Feature Class",
name="out_fc",
datatype="DEFeatureClass",
parameterType="Required",
direction="Input")
params = [param0, param1]
return params
The main work of a tool is done inside the execute() method. This is where the geoprocessing of your tool takes place. The execute() method, as shown in the following code snippet, can accept a number of arguments, including the tools self, parameters, and messages:
def execute(self, parameters, messages):
"""The source code of the tool."""
returnTo access the parameter values that are passed into the tool, you can use the valueAsText() method. The following steps will guide you through how to add and execute the execute() method:
Find the
execute()function in theUSGSDownloadclass and add the following code snippet to access the parameter values that will be passed into your tool. Remember from a previous step that the first parameter will contain a URL to a map service containing the wildfire data and the second parameter will be the output feature class where the data will be written:def execute(self, parameters, messages): inFeatures = parameters[0].valueAsText outFeatureClass = parameters[1].valueAsTextAt this point, you have created a Python Toolbox, added a tool, defined the parameters for the tool, and created variables that will hold the parameter values that the end user has defined. Ultimately, this tool will use the URL that is passed to the tool to connect to an ArcGIS Server map service, download the current wildfire data, create a feature class, and write the wildfire data to the feature class. However, we're going to save the geoprocessing tasks for later. For now, I just want you to print out the values of the parameters that have been entered so that we know that the structure of the tool is working correctly. Print this information using the following code:
def execute(self, parameters, messages): inFeatures = parameters[0].valueAsText arcpy.AddMessage(inFeatures) outFeatureClass = parameters[1].valueAsText arcpy.AddMessage(outFeatureClass)Execute the tool by double-clicking on USGS Download from the InsertWildfires.pyt toolbox that you've created. You should see the following dialog box:
Leave the default URL parameter as it is and select an output feature class. You'll want to click on the Browse button and then navigate to the
C:\ArcGIS_Blueprint_Python\Data\WildfireDatafolder and select theWildlandFiresgeodatabase. Inside, there is an empty feature class calledCurrentFires. Select this feature class.Now click on OK. The progress dialog box will contain the parameter values that you passed in. Your output feature class will probably be different than mine, as shown in the following screenshot:
We'll complete the actual geoprocessing of this tool in the next section.
In the previous section, you learned how to create a Python Toolbox and add tools. You created a new toolbox called InsertWildfires and added a tool called USGS Download. However, in that exercise, you didn't complete the geoprocessing operations that connect to an ArcGIS Server map service, query the service for current wildfires, and populate the feature class from the data pulled from the map service query. You'll complete these steps in the following section.
This section of the application uses the Python requests module. If you don't already have this module installed on your computer, you will need to do this at this time using pip.
The pip is a package manager that serves as a repository and installation manager for Python modules. It makes finding and installing Python modules much easier. There are several steps that you'll need to follow in order to install pip and the requests module. Instructions to install pip and the requests module are provided in the first few steps:
Open the Environment Variables dialog box in Windows. The easiest way to display this dialog box is to go to Start and then type Environment Variables in the search box. This should display an Edit environment variables for your account entry. Select this item.
If you don't see a variable called PATH, click on the New button to create one. If you already have a PATH variable, you can just add to the existing content. Select the PATH variable, and click on Edit, and then set the value to
C:\Python27\ArcGIS10.3;C:\Python27\ArcGIS10.3\Scripts. The first path will provide a reference to the location of the Python executable and the second will reference the location of pip when it is installed. This makes it possible to run Python and pip from the Command Prompt in Windows.Click on OK and then click on OK again to save the changes.
Next, we'll install pip if you haven't already done so in the past. You need to install pip before you can install requests. In a web browser, go to https://pip.pypa.io/en/latest/installing.html and scroll down to the install pip section. Right-click on
get-pip.pyand select Save Link As or something similar. This will vary depending upon the browser you are using. Save it to yourC:\ArcGIS_Blueprint_Pythonfolder.Open the Command Prompt in Windows, type python
C:\ArcGIS_Blueprint_Python\get-pip.py, and press Enter on your keyboard. This will install pip.In the Command Prompt, type
pip install requestsand press Enter on your keyboard. This will install the requests module.Close the Command Prompt.
In the following steps, we will learn how to request data from ArcGIS Server:
Open ArcCatalog and navigate to the location where you've created your Python Toolbox, it would look like following screenshot:
Right-click on InsertWildfires.pyt and select Edit to display the code for the toolbox.
First, we'll clean up a little by removing the AddMessage() functions. Clean up your execute() method so that it appears as follows:
def execute(self, parameters, messages): inFeatures = parameters[0].valueAsText outFeatureClass = parameters[1].valueAsTextNext, add the code that connects to the wildfire map service, to perform a query. In this step, you will also define the
QueryStringparameters that will be passed into the query of the map service. First, import therequestsandjsonmodules:import arcpy import requests, json class Toolbox(object): def __init__(self): """Define the toolbox (the name of the toolbox is the name of the .pyt file).""" self.label = "Toolbox" self.alias = "" # List of tool classes associated with this toolbox self.tools = [USGSDownload]
Then, create the
agisurlandjson_payloadvariables that will hold theQueryStringparameters. Note that, in this case, we have defined aWHEREclause so that only wildfires where the acres are greater than 5 will be returned. TheinFeaturesvariable holds the ArcGIS Server Wildfire URL:def execute(self, parameters, messages): inFeatures = parameters[0].valueAsText outFeatureClass = parameters[1].valueAsText agisurl = inFeatures json_payload = { 'where': 'acres > 5', 'f': 'pjson', 'outFields': 'latitude,longitude,incidentname,acres' }Submit the request to the ArcGIS Server instance; the response should be stored in a variable called
r. Print a message to the dialog box indicating the response as:def execute(self, parameters, messages): inFeatures = parameters[0].valueAsText outFeatureClass = parameters[1].valueAsText agisurl = inFeatures json_payload = { 'where': 'acres > 5', 'f': 'pjson', 'outFields': 'latitude,longitude,incidentname,acres' } r = requests.get(agisurl, params=json_payload) arcpy.AddMessage("The response: " + r.text)Test the code to make sure that we're on the right track. Save the file and refresh
InsertWildfiresin ArcCatalog. Execute the tool and leave the default URL. If everything is working as expected, you should see a JSON object output to the progress dialog box. Your output will probably vary from the following screenshot:
Return to the
execute()method and convert the JSON object to a Python dictionary using thejson.loads()method:def execute(self, parameters, messages): inFeatures = parameters[0].valueAsText outFeatureClass = parameters[1].valueAsText agisurl = inFeatures json_payload = { 'where': 'acres > 5', 'f': 'pjson', 'outFields': 'latitude,longitude,incidentname,acres' } r = requests.get(inFeatures, params=json_payload) arcpy.AddMessage("The response: " + r.text) decoded = json.loads(r.text)
The following steps will guide you, to insert data in a feature class with the help of the ArcPy data access module:
Now, we'll use the
ArcPydata access module, that isArcpy.da, to create anInsertCursorobject by passing the output feature class defined in the tool dialog box along with the fields that will be populated:def execute(self, parameters, messages): inFeatures = parameters[0].valueAsText outFeatureClass = parameters[1].valueAsText agisurl = inFeatures json_payload = { 'where': 'acres > 5', 'f': 'pjson', 'outFields': 'latitude,longitude,fire_name,acres' } r = requests.get(inFeatures, params=json_payload) arcpy.AddMessage("The response: " + r.text) decoded = json.loads(r.text) cur = arcpy.da.InsertCursor(outFeatureClass, ("SHAPE@XY", "NAME", "ACRES"))Create a
Forloop that you can see in the following code, and then we'll discuss what this section of code accomplishes:def execute(self, parameters, messages): inFeatures = parameters[0].valueAsText outFeatureClass = parameters[1].valueAsText agisurl = inFeatures json_payload = { 'where': 'acres > 5', 'f': 'pjson', 'outFields': 'latitude,longitude,fire_name,acres' } r = requests.get(inFeatures, params=json_payload) arcpy.AddMessage("The response: " + r.text) decoded = json.loads(r.text) cur = arcpy.da.InsertCursor(outFeatureClass, ("SHAPE@XY", "NAME", "ACRES")) cntr = 1 for rslt in decoded['features']: fireName = rslt['attributes']['incidentname'] latitude = rslt['attributes']['latitude'] longitude = rslt['attributes']['longitude'] acres = rslt['attributes']['acres'] cur.insertRow([(longitude,latitude),fireName, acres]) arcpy.AddMessage("Record number: " + str(cntr) + " written to feature class") cntr = cntr + 1 del curThe first line simply creates a
counterthat will be used to display the progress information in the Progress Dialog box. We then start aForloop that loops through each of the features (wildfires) that have been returned. The decoded variable is a Python dictionary. Inside theForloop, we retrieve the wildfire name, latitude, longitude, and acres from the attributes dictionary. Finally, we call theinsertRow()method to insert a new row into the feature class along with the wildfire name and acres as attributes. The progress information is written to the Progress Dialog box and the counter is updated.Double-click on the USGS Download tool.
Leave the default URL and select the
CurrentFiresfeature class in theWildlandFiresgeodatabase. TheCurrentFiresfeature class is empty and has fields forNAMESandACRES:
Click on OK to execute the tool. The number of features written to the feature class will vary depending upon the current wildfire activity. Most of the time, there is at least a little activity, but it is possible that there wouldn't be any wildfires in the U.S. as shown in the following screenshot:
View the feature class in ArcMap. To view the feature class in the following screenshot, I've plotted the points along with a Basemap topography layer. Your data will almost certainly be different than mine as we are pulling real-time data:
Integrating ArcGIS Desktop and ArcGIS Server is easily accomplished using the ArcGIS REST API and the Python programming language. In this chapter, we created an ArcGIS Python Toolbox containing a tool that connects to an ArcGIS Server map service containing real-time wildfire information and hosted by the USGS. The connection was accomplished through the use of the Python request module that we used in order to submit a request and handle the response. Finally, we used the ArcPy data access module to write this information to a local geodatabase.
In the next chapter, we'll continue working with ArcGIS Python toolboxes, and you'll also learn how to read CSV files with the Python CSV module, insert data into a feature class, and use the arcpy.mapping module to work with time-enabled data.
