Python for Google App Engine

There are two options if we want to host an application on Google Cloud Platform:

Google Cloud Platform provides several options to store and access users' data:

Google Cloud Storage: This is a highly available and scalable file storage service with versioning and caching. We will learn how to use Cloud Storage in Chapter 3, Storing and Processing User's Data.

Google Cloud SQL: This is a fully managed MySQL relational database; replication, security and availability are Google's responsibilities. Chapter 5, Storing Data in Google Cloud SQL, is entirely dedicated to this service.

Google Cloud Datastore: This is a managed schemaless database that stores nonrelational data objects called entities; it scales automatically, supports transactions, and can be queried with SQL-like syntax. We will start using it in Chapter 2, A More Complex Application, and learn how to get the most out of it in Chapter 4, Improving Application Performance.

BigQuery is a tool provided by Google Cloud Platform that allows to perform queries using an SQL-like syntax against a huge amount of data in a matter of seconds. Before it can be analyzed, data must be streamed into BigQuery through its API or uploaded to Google Cloud Storage.

Instead of writing code from scratch, we can easily add functionalities to our applications using some of Google's services through APIs that are very well integrated within Google Cloud Platform:

All the tools and services provided by Google Cloud Platform are billed with a pay-per-use model so that applications can scale up or down as needed and we only pay for resources we actually use. A handy calculator is provided to have a precise idea of the costs depending on the services and resources we think we will need. Google Cloud Platform offers a certain amount of resources we can use without paying anything; usually, these free quotas are well suited to host web applications with low traffic at no cost.

App Engine runs on computing units that are completely managed called instances. We can (and should) ignore which operating system is running on an instance because we interact solely with the runtime environment, which is an abstraction of the operating system that provides resource allocation, computation management, request handling, scaling, and load balancing.

Every time a client contacts an application that runs on App Engine, a component of the runtime environment called scheduler selects an instance that can provide a fast response, initializes it with application data if needed, and executes the application with a Python interpreter in a safe, sandboxed environment. The application receives the HTTP request, performs its work, and sends an HTTP response back to the environment. Communication between the runtime environment and the application is performed using the Web Server Gateway Interface (WSGI) protocol; this means that developers can use any WSGI-compatible web framework in their application.

The runtime environment is sandboxed to improve security and provide isolation between applications running on the same instance. The interpreter can execute any Python code, import other modules, and access the standard library, provided that it doesn't violate sandbox restrictions. In particular, the interpreter will raise an exception whenever it tries to write to the filesystem, perform network connections, or import extension modules written in the C language. Another isolation mechanism we must be aware of that is provided by sandboxing, prevents an application from overusing an instance by raising an exception whenever the entire request/response cycle lasts more than 60 seconds.

Thanks to sandboxing, the runtime can decide at any given time whether to run an application on one instance or many instances, with requests being spread across all of them depending on the traffic. This capability, together with load balancing and scheduler settings is what makes App Engine really scalable.

Users can easily tune an application's performance by increasing its responsiveness or optimizing costs with a simple and interactive administrative console. We can specify instance performance in terms of memory and CPU limits, the number of idle instances always ready to satisfy a request, and the number of instances dynamically started when the traffic increases. We can also specify the maximum amount of time in milliseconds we tolerate for a pending request and let App Engine adjust the settings automatically.

At first sight, restrictions imposed by the runtime environment might seem too restrictive. In the end, how can developers make something useful without being able to write data on disk, receive incoming network connections, fetch resources from external web applications, or start utility services such as a cache? This is why App Engine provides a set of higher-level APIs/services that can be used by developers to store and cache data or communicate over the Internet.

Some of these services are provided by the Google Cloud Platform as standalone products and are smoothly integrated into App Engine, while some others are only available from within the runtime environment.

The list of available services changes quite often as Google releases new APIs and tools; the following is a subset of tools we will use later in the book in addition to the Datastore, Google Cloud Endpoints, Google Cloud SQL, and Google Cloud Storage services we introduced earlier:

For more information on the tools and services provided by Google that we can use from within the App Engine environment, refer to https://developers.google.com/appengine/features/.

To get started, we need to install the Google App Engine SDK for Python for the platform of our choice. The SDK contains all the libraries needed to develop an application and a set of tools to run and test the application in the local environment and deploy it in the production servers. On some platforms, administrative tasks can be performed through a GUI, the Google App Engine Launcher, on other platforms we can use a comprehensive set of command line tools. We will see Google App Engine Launcher in detail later in this chapter.

Before installing the SDK, we have to check whether a working installation of Python 2.7 (version 2.7.8 is the latest at the time of writing this book) is available on our system; we need this specific version of Python because, with 2.5 deprecated now, it is the only version supported by the App Engine platform. If we are using Linux or Mac OS X, we can check the Python version from the terminal that issues the command (notice the capital letter V):

python -V

The output should look like this:

Python 2.7.8

If we are on Windows, we can just ensure the right version of Python is listed in the Programs section within the Control Panel.

The official App Engine download page contains links for all the available SDKs. The following link points directly to the Python version: https://developers.google.com/appengine/downloads#Google_App_Engine_SDK_for_Python.

We have to choose the right package for our platform, download the installer, and proceed with the installation.

The Windows and OS X versions of the SDK ships with a graphical user interface tool called Launcher that we can use to perform administrative tasks such as creating and managing multiple applications.

The following screenshot shows the Launcher window in OS X:

We can see the Launcher in Windows in the following screenshot:

Before we start using the Launcher it's important to check whether it is using the right Python version. This is very important if we have more than one Python installation in our system. To check the Python version used by Launcher and to change it, we can open the Preferences... dialog by clicking the appropriate menu depending on our platform and set the Python path value. In the same dialog we can specify which text editor Launcher will open by default when we need to edit application files.

To create a new application we can click New Application in the File menu or click the button with a plus sign icon in the bottom-left corner of the Launcher window. Launcher will prompt for the application name and the path to the folder that will contain all the project files; once created, the application will be listed in the main window of Launcher.

We can start the local development server by clicking the Run button on the Launcher toolbar or clicking Run in the Control menu. Once the server is started, we can stop it by clicking on the Stop button or the Stop entry in the Control menu. Clicking the Browse button or the Browse entry in the Control menu opens the default browser at the home page of the selected application. To browse the logs produced by the development server, we can open the Log Console window by clicking the Logs button on the toolbar or the Logs entry in the Control menu. The SDK Console button on the toolbar and the SDK Console action on the Control menu will open the default browser at the URL that serves the Developer Console, a built-in application to interact with the local development server, which we will explore in detail later in this chapter.

The Edit button will open the configuration file for the selected application in an external text editor, maybe the one we specified in the Preferences... dialog; the same happens when we click the Open in External Editor action in the Edit menu.

To deploy and upload the selected application to App Engine we can click the Deploy button on the toolbar or click the Deploy action in the Control menu. The Dashboard button on the toolbar and the Dashboard action in the Control menu will open the default browser at the URL of App Engine Administrative Console.

Using Launcher we can set additional flags for the local development server and customize some parameters such as the TCP port number to which listens. To do so we have to click the Application Settings... entry in the Edit menu and make the desired adjustments in the settings dialog.

Launcher can also handle existing applications created from scratch through the command line or checked out from an external repository. To add an existing application to the Launcher, we can click the Add Existing Application... entry in the File menu and specify the application path.

The first step to create an application is pick a name for it. According to the tradition we're going to write an application that will print "Hello, World!" so we can choose helloword as the application name. We already know how to create an application from Launcher, the alternative is to do it manually from the command line.

At the simplest, a working Python application consists of a folder called application root that contains an app.yaml configuration file and a Python module with the code needed to handle HTTP requests. When we create an application within Launcher, it takes care of generating those files and the root folder for us, but let's see how can we can accomplish the same result from the command line.

mkdir helloworld && cd helloworld

application: helloworld
version: 1
runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: .*
  script: main.app

libraries:
- name: webapp2
  version: "2.5.2"

import webapp2

class MainHandler(webapp2.RequestHandler):
    def get(self):
        self.response.write('Hello world!')

app = webapp2.WSGIApplication([
    ('/', MainHandler)
], debug=True)

The App Engine SDK provides an extremely useful tool called development server that runs on our local system emulating the runtime environment we will find in production. This way, we can test our applications locally as we write them. We already know how to start the development server from Launcher. To launch it from the command line instead, we run the dev_appserver.py command tool passing the root folder of the application we want to execute as an argument. For example, if we're already inside the root folder of our helloworld application, to start the server, we can run this command:

dev_appserver.py .

The development server will print some status information on the shell and will then start listen at the local host to the default TCP ports 8000 and 8080, serving the admin console and the application respectively.

While the server is running, we can open a browser, point it at http://localhost:8080 and see our first web application serving content.

The following screenshot shows the output:

If we are using Launcher, we can simply press the Browse button and the browser will be opened automatically at the right URL.

The development server automatically restarts application instances whenever it detects that some content on the application root folder has changed. For example, while the server is running we can try to change the Python code that alters the string we write in the response body:

import webapp2

class MainHandler(webapp2.RequestHandler):
    def get(self):
        self.response.write('<H1>Hello world!</H1>')
        self.response.write("<p>I'm using App Engine!</p>")

app = webapp2.WSGIApplication([
    ('/', MainHandler)
], debug=True)

After saving the file, we can refresh the browser and immediately see the changes without reloading the server, as shown in the following screenshot:

We can now move our application to a production server on App Engine and make it available through the Internet.

Every application running on App Engine is uniquely identified by its name within the Google Cloud Platform. That is why sometimes we find parts of the documentation and tools referring to that as application ID. When working on a local system, we can safely pick any name we want for an application as the local server does not enforce any control on the application ID; but, if we want to deploy an application in production, the application ID must be validated and registered through App Engine Admin Console.

Admin Console can be accessed at https://appengine.google.com/ and log in with a valid Google user account or a Google apps account for custom domains. If we are using Application Launcher, clicking the Dashboard button will open the browser at the right address for us. Once logged in, we can click the Create Application button to access the application creation page. We have to provide an application ID (the console will tell us whether it is valid and available) and a title for the application and we're done. For now, we can accept the default values for the remaining options; clicking on Create Application again will finally register the application's ID for us.

Now we have to change the dummy application ID we provided for our application with the one registered on App Engine. Open the app.yaml configuration file and change the application property accordingly:

application: the_registered_application_ID
version: 1
runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: .*
  script: main.app

libraries:
- name: webapp2
  version: "2.5.2"

We are now ready to deploy the application on App Engine. If we are using Application Launcher, all we have to do is click on the Deploy button in the toolbar. Launcher will ask for our Google credentials and then the log window will open showing the deployment status. If everything went fine the last line shown should be something like this:

*** appcfg.py has finished with exit code 0 ***

Deploying from the command line is just as easy; from the application root directory, we issue the command:

appcfg.py update .

We will be prompted for our Google account credentials, and then the deployment will proceed automatically.

Every App Engine application running in production can be accessed via http://the_registered_application_ID.appspot.com/, so we can tell whether the application is actually working by accessing this URL from a browser and checking whether the output is the same as that produced by the local development server.

Google App Engine allow us to serve content over HTTPS (HTTP Secure) connections on top of the Secure Sockets Layer (SSL) protocol, which means that data transferred from and to the server is encrypted. When using the appspot.com domain, this option is free of charge. To enable secure connections between clients and the App Engine server, all we have to do is add the secure option to the URLs listed in the app.yaml file:

handlers:
- url: .*
  script: main.app
  secure: always

On the local development server we will still use regular HTTP connections, but in production we will access https://the_registered_application_ID.appspot.com/ in a secure manner over HTTPS connections.

If we want to access the application over HTTPS through a custom domain instead, such as example.com, we have to configure App Engine so that the platform can use our certificates by following the instructions at https://cloud.google.com/appengine/docs/ssl. This service has a fee and we will be charged monthly.

When we are on the local development server we can still access a tool to browse and manage Datastore, task queues, cron jobs, and other App Engine emulated components running locally. This tool is called Development Console and is accessible at http://localhost:8000 when the local server is active.