Google App Engine (GAE) is a platform and SDK for developing and hosting web applications, using Google's servers and infrastructure. Google Web Toolkit (GWT) is a development toolkit for building complex AJAX-based web applications using Java, which is then compiled to optimized JavaScript. Used together, GAE/Java and GWT provide an end-to-end Java solution for AJAX web applications, which can solve many of the problems that arise in developing, maintaining, and scaling web applications.
State-of-the-art AJAX web apps with optimized JavaScript and no worries about browser compatibility: all major browsers are supported automatically
A powerful widget library enabling you to build complex UIs quickly
Interactive debugging and compile-time checking of frontend Java code, prior to its compilation to JavaScript
Almost unlimited scalability: App Engine applications are scaled and load-balanced automatically for you as the demand for your application as well as your data storage needs increase
Freedom from application administration: app server and data storage administration is handled by the platform
"One-click" deployment of an application to App Engine's hosting platform, with all of the GWT compilation and deployment details managed for you
Access to powerful and scalable App Engine services that provide support for asynchronous computation and a range of communications APIs
This book is designed to give developers the tools they need to build their own GAE+GWT applications with a particular focus on some of the technologies useful for building social-media-oriented applications. The book is centered on a GAE+GWT Java application called Connectr that is developed throughout the chapters and demonstrates by example the use of the technologies described.
The remainder of this chapter will introduce the features of Google App Engine and Google Web Toolkit in more detail. Then we describe the Connectr application whose development provides a running example throughout the chapters, and highlight some of the technologies and approaches used by the application.
The final section of the chapter gives pointers to some useful online resources.
Google App Engine (GAE) is a platform and SDK for developing and hosting web applications using Google's servers and infrastructure. The most current version of GAE at the time of writing is 1.3.7. App Engine supports both Java and Python runtime environments and SDKs, with each environment supporting standard protocols and technologies for web application development. This book focuses on the Java runtime.
Google App Engine as a platform is designed for scalability, robustness, and performance. App Engine allows you to build applications that are scaled and load-balanced automatically for you as the demand for your application and your data storage needs increase. With App Engine applications, you do not need to perform any server or database maintenance yourself—you just upload and run your app. Google's infrastructure offers reliable performance even under a heavy load and when using very large amounts of data.
A request to an App Engine app is routed to a selected app server, and the application is started on the server if necessary. No state on the server is saved between requests. (Instead, GAE has other means to persist data between requests). There is no guarantee that the same server will handle two subsequent requests, even if the time period between them is very short. Thus, a runtime instance often comes into existence when a request handler begins, and is terminated when it ends—though the instance may sometimes be retained, depending upon app traffic.
By making no assumption that a runtime will be maintained on a given server between requests, App Engine can distribute request traffic across as many servers as necessary, and target requests at servers that it expects will provide the fastest response.
It is useful to view App Engine in terms of the several facets of its functionality—the App Engine Datastore, its scalable services, and its runtime environments.
App Engine's Datastore is based on Google's BigTable technology (http://labs.google.com/papers/bigtable.html). It is a non-centralized, persistent store, designed specifically to be distributed and to scale as it grows.
The Datastore is not a "join-query" relational database. Rather, it is essentially a property-value store, holding entities—each of a given kind—that contain property-value sets.
App Engine's Datastore is designed so that the request time is linear in the size of the results, not the size of the data stored. This is accomplished in part by the way in which Datastore builds indexes at write-time, and App Engine's imposition of constraints on the types of queries supported, resulting in extremely efficient reads over large distributed datasets.
The Datastore supports transactions and uses optimistic concurrency control. The transactional model is specific to how App Engine works: the allowed transactions are determined by how entities are defined and grouped in the Datastore.
App Engine supports a number of useful capabilities and services, all designed to scale with the size of your app. The Datastore is in fact one such service. Other services include:
APIs for authentication via both Google Accounts and OAuth
An API for sending and receiving e-mail
An API for sending and receiving XMPP messages
Support for Task Queues, allowing work to be performed asynchronously outside the scope of a web request
Support for running scheduled tasks at specified times or intervals: Cron jobs
Support for image processing
Integration with Google Apps
Both the Python and Java App Engine runtimes support the above-mentioned capabilities, and many discussions of App Engine capabilities and design approaches can be largely runtime-environment agnostic. In a few cases, the Python features or development tools, which have a head start on the Java version, are currently more capable, but in future their respective capabilities are intended to equalize.
Both runtime environments have some "sandbox" restrictions, imposed to allow App Engine to use any server for a new incoming request, and to control server responsiveness and security. For example, application code cannot access the server on which it is running, in the traditional sense.
SDKs are provided for both the Python and Java runtimes, along with tools for uploading and downloading app information, including bulk data uploads and downloads.
This book uses Google App Engine's Java platform, with the exception of a few forays into Python in order to access some administrative tools.
The Google App Engine Java (GAE/J) runtime includes the Java SE Runtime Environment (JRE) 6 platform and libraries (with some sandbox restrictions implemented in the JVM). The Java Servlet standard is supported, and is a basis for how your app interacts with its environment.
The App Engine Java SDK currently supports developing apps using either Java 5 or 6. However, support in the SDK for Java 1.5 is now deprecated. Languages that use a JVM-based interpreter or compiler (and do not violate the sandbox restrictions), such as JRuby, can be run on App Engine as well.
For most people, the easiest way to develop Java App Engine apps is via the Eclipse IDE (http://www.eclipse.org/). Eclipse has sophisticated general support for Java development. Google provides a plugin that supports development of GWT+GAE apps, and adds many additional capabilities. The plugin helps to generate the necessary directories for a GAE (and GWT) project, creates key files, sets up classpaths, and can do some code generation. The plugin also allows "push-button" uploading and deployment of an App Engine application, including automatic compilation of the GWT Java code to JavaScript as required, and copying and packaging of all the necessary files for the deployment.
Furthermore, the plugin integrates the SDK's development support for simulating GAE on your local computer, giving you the ability to set breakpoints and use an interactive debugger.
This book will use Eclipse for development.
GAE apps are hosted at http://appspot.com (you can map the apps to your own domain names as well, if you like). You can host multiple apps, and multiple versions of each app. You can host apps for free, with relatively limited resource quotas, or enable billing for an app in order to increase its available resources. The App Engine billing model is tied to the resources actually used by an app so if you enable billing you will be charged more only as your app becomes busier— up to fixed thresholds that you specify. An application will still have some resource quotas, even with billing enabled; if you are running up against these thresholds, you can request to have them raised.
To administer your app versions, track their resource usage and statistics and view their logs, manage app billing, and so on, an App Engine Admin Console web application is provided.
Google has announced a new platform—Google App Engine for Business (http://code.google.com/appengine/business/). Its features include a Service Level Agreement (SLA), a per-user-per-app pricing structure (up to a maximum threshold), and support for SQL and dedicated relational databases in addition to the Datastore.
At the time of writing, App Engine for Business has not been rolled out, its details are still subject to change, and it will not be covered in this book.
Some aspects of the GAE model and App Engine development can be surprising to developers who are new to the platform. They stem in part from the constraints imposed by App Engine in order to ensure scalability and reliability. We will list some of these aspects here, and discuss all of them in more detail, later in the book.
Not all core Java methods and networking capabilities are supported—so, not all third-party Java libraries port to App Engine. Many do, however, and we will use some in this book.
There is no file system in the usual sense. Dynamic data must be persisted by other means; however, it is possible to read static files associated with an app.
Datastore is not a "join-query" relational database. It does not support joins in the traditional sense, nor text search. So different approaches to data modeling and query construction may be required with App Engine apps.
There are a number of usage limitations, including limits on the size of the data chunks that can be transferred and stored.
There is a 30-second request limit; this includes background processes. Typically, long-running activities are split up, often using App Engine's task queue.
When a new app server is allocated for a request, "spin-up time" is required to load the app instance, increasing both request CPU usage and response time. Low-traffic apps are often evicted from their server, and thus such apps may see the spin-up time reflected in a large percentage of requests to the app. (In future, it may be possible to address this by paying to retain a warm instance for a given app).
GWT is a development toolkit for building complex AJAX-based web applications using Java, which is then compiled to highly optimized JavaScript. The current version as of this writing is 2.0.4.
GWT allows you to write your application frontend in Java, and then compile the source to highly optimized JavaScript that runs across all major browsers. It allows you to debug applications in an IDE as you would debug a "regular" Java app, interactively setting breakpoints, while at the same time you can access the app in the browser as if it were generated JavaScript. It does this via a browser developer plugin, which spans the gap between Java bytecode in the debugger and the browser's JavaScript.
Once you are ready to deploy, the GWT Java code is compiled to JavaScript. More exactly, a compiled application frontend consists of JavaScript, HTML, and XML.
There are several advantages to this compilation process in terms of efficiency, security, and browser compatibility. The compiled JavaScript is highly efficient and optimized, and helps protect against Cross-site scripting (XSS) attacks. (The GWT documentation provides more detail on the specifics of this; essentially, you are well protected as long as you do not add your own JavaScript, use the JSON API to parse untrusted strings, or write code to set "innerHTML" in GWT widgets).
In addition, compatible JavaScript is generated for all major browsers (a current list of supported browsers is here: http://code.google.com/webtoolkit/doc/latest/FAQ_GettingStarted.html#What_browsers_does_GWT_support?). Every web browser has its own idiosyncrasies, and GWT allows you to work at a level of abstraction above these differences. When the Java app is compiled into JavaScript, a different, efficient version is generated for each browser type. However, only the JavaScript required by a given client is downloaded when a request is made to the app. GWT uses a technique called deferred binding to accomplish this. A bootstrapping process at runtime loads the right version for the client.
GWT doesn't require you to write your app using any JavaScript—you can stick to Java, with its strict typing and powerful compile-time checks, to build your app. However, you can integrate raw JavaScript with a GWT app if you like, as when calling other native code using JNI. You can define an entire application frontend to use GWT, or you can integrate small amounts of GWT with an existing app.
GWT also allows developer-guided code splitting. Code splitting is useful when your application's compiled JavaScript code base is large enough to require significant startup time while downloading it. In this case, you can indicate places in your GWT program where you want the program to pause and download more code. Code will always be loaded automatically on initial download if it is part of a dependency chain for the application's initial entry point. So, code splitting can require some consideration of program design, using patterns that allow control of "modules" of functionality and their access. However, the process can provide a large payoff in terms of efficiency.
GWT provides many pre-defined widgets and panels that you can use as UI building blocks and also allows you to build your own widgets, either as composites or from scratch. It allows both programmatic and declarative definition of UIs, or a combination of both, and allows use of CSS for formatting and layout.
We will make use of many of the GWT widgets and panels in this book. For a full list of the GWT-provided widgets, see the GWT gallery: http://code.google.com/webtoolkit/doc/latest/RefWidgetGallery.html.
The GWT toolkit includes other support for building AJAX-based applications. It addresses the "browser back button" issue by providing support for history management. This issue stems from the fact that with AJAX web apps, it is possible for the page content to change without a change in the URL. In such cases, the back button won't work as the user expects; nor will bookmarking the URL necessarily return them to the UI state that they expect. GWT's history management provides a way to rewrite the URL upon important state changes, so that the back button and bookmarking work as expected.
GWT also provides strong support for event handling—both low-level browser events, and more abstract application-level events.
GWT has a Remote Procedure Call (RPC) API and framework that supports client-server communication. It allows you to pass objects between the client and server, instantiated from shared data classes, in a straightforward manner.
The RPC package runs on App Engine, making it easy to build GWT frontends to App Engine apps—exactly as we will do in this book. (The RPC package can also be used with other Servlet containers, such as Tomcat).
The RPC calls are all asynchronous, that is, non-blocking. They return immediately, and use a callback method, defined by the user, to handle the results later returned from the server.
GWT supports client-server communication via other means as well (for example, using JSON), but this book focuses on the RPC API.
GWT does not need to be paired with GAE/Java as the backend, but they are designed to be integrated, and work very effectively together. One goal of this book is to show how GWT combined with App Engine synergistically simplifies development and deployment in many ways.
A typical GWT+GAE code base will include client, server, and shared packages. The shared package includes classes that must be accessible to both the client GWT and server GAE code. Data Transfer Objects (those objects passed over GWT RPC) must be in the shared package and must not use any Java methods unavailable to GAE or GWT.
Third-party Java libraries cannot be used from client-side Java code unless their source code is available to the GWT compiler and this code uses only the supported subset of Java.
As indicated earlier, Google provides a GWT+GAE plugin for the Eclipse IDE, which provides a number of useful capabilities. The plugin includes the SDKs for both App Engine and GWT. The Eclipse plugin allows integrated use of the GWT SDK's developer browser plugin, letting you set Eclipse breakpoints in your frontend GWT Java code and debug and inspect interactively while accessing the app in the browser—a powerful development aid.
The Eclipse plugin also fully manages app deployment for you. It allows "push-button" uploading of an App Engine app, including automatic compilation of the GWT Java code, and copying and packaging of all the necessary files for the deployment.
GWT supports most but not all core Java capabilities and language semantics. Its scope is constrained by its purpose—the Java code must be compilable into JavaScript. As an example, Java threading is not supported by GWT. In some cases, the GWT emulation differs from the standard Java runtime, or provides a subset of the JRE functionality.
In general, GWT's JRE emulation supports the following packages (sometimes with only a subset of the methods): java.lang, java.lang.annotation, java.util, java.io, and java.sql. Use of an IDE such as Eclipse is helpful in indicating when a given Java method is not supported. The GWT documentation (http://code.google.com/webtoolkit/doc/latest/DevGuideCodingBasicsCompatibility.html, http://code.google.com/webtoolkit/doc/latest/RefJreEmulation.html) provides specifics on supported classes.
There are many online resources and books that introduce GWT. This book assumes some basic prior knowledge about how to use GWT. In particular, it would be helpful for you to be familiar with the following concepts when reading this book:
Have an understanding of the different types of files that are generated by the GWT compiler and what their purpose is. Similarly, have a general understanding of the "bootstrapping" process by which an application is loaded in a browser, and the files involved. (The Eclipse plugin will manage generation and uploading of these files for you on deployment of your App Engine app).
Have an understanding of GWT modules, be familiar with the definition of
gwt.xmlfiles and what they can contain, understand how to define and use app entry point classes, and the role of the Root panel.Understand the basics of building widgets and panels programmatically. The GWT API Reference (http://google-web-toolkit.googlecode.com/svn/javadoc/2.1/index.html) can help with the details.
Understand how to define and register GWT event handlers.
Be familiar with the general concept of making Remote Procedure Calls, and be comfortable defining callbacks and anonymous Java subclasses.
All of these topics are covered in Google's online GWT documentation (http://code.google.com/webtoolkit/).
This book is centered on a GWT + Google App Engine application, designed to show the power of combining these two technologies. The app is also designed to explore many of the important features and capabilities of both GAE/J and GWT, which will likely be used in building your own apps, and demonstrates a number of useful design patterns.
The example application is a social-media-oriented app named Connectr. The application includes social-media information gathering and aggregation activities, and incorporates the use of App Engine's services and APIs.
Connectr allows a user to create an account and log in by authenticating via the Google Accounts API (for example, with their Gmail account), or via OAuth using a Twitter or Facebook account.
Once logged in, the user can maintain and edit contact information for a set of friends, and can display a stream aggregating the friends' updates. Connectr supports login authentication via either the Google Accounts API (App Engine's support for authentication via a Google login), or via Twitter or Facebook using OAuth.
A user's customized information stream is based on an aggregation of feed data. As part of the information associated with each friend, a user can add any number of RSS or Atom feed URLs for that friend, for example, from services such as Twitter, Flickr, and blogs. Figure 1 shows the editing of a friend's details in Connectr, with the friend's associated URLs specified.
Figure 1: Editing Friend information in Connectr.
The feed data is fetched and integrated into a chronologically sorted and auto-updating "activity stream", supported by push from the App Engine's recently-introduced Channel API.
Figure 2: The main view of Connctr, showing a list of friends for a user, and a stream of items for selected friends.
The stream can show information from all of the user's friends' feeds, or be filtered to show the feeds for only a subset of friends. A user's activity stream can also be configured to include insertion of breaking news notifications, delivered to the app via XMPP.
Figure 2 shows a typical view of the main page of Connectr, with a user's friends listed on the left, and the activity stream for the selected users' associated feeds to the right (in this figure the feeds, for the purposes of the example, look suspiciously like news feeds).
Note
Our use of the phrase "activity stream" is not to be confused with its more technical reference to an extension of the Atom feed format (http://activitystrea.ms/)—though use of Activity Stream-formatted feeds would certainly be consistent with the semantics of Connectr's stream.
Behind the scenes, the application makes use of App Engine services such as Cron jobs and task queues to update feeds asynchronously in the background, to push new content to the client, and to perform other administrative tasks.
The Connectr application serves as a running example throughout the book. It is introduced in several progressive stages, with simpler versions of the application introduced in the first few chapters, and subsequent chapters further developing the app as functionality is added. One of the goals of this book is to be a resource for a number of design patterns, approaches, and idioms, as developed within Connectr.
As we introduce the various features of Connectr, we will develop these patterns and approaches both for GWT and App Engine, as well as for communication between the client and server components of an app.
For GWT, we introduce a number of design patterns for keeping code modular and manageable, including the MVP, Event Bus, and related patterns; and describe the use of GWT's UI Binder for declarative layout specification.
For App Engine, Connectr illustrates a number of approaches for data class design and querying, including design to get around the lack of a relational database, instead leveraging the Datastore's features. We describe models for decoupled and asynchronous processing and task design, in order to make an application more robust and scalable, including use of task queues and Cron Jobs to support background processing; show how to use query cursors to break data access into manageable chunks; and introduce some useful Memcache patterns.
Additionally, Connectr illustrates some useful patterns in client-server communication, including use of App Engine's recently developed Channel API to push content from the server to the client, and use of OAuth and Google accounts to support multiple login methods, so that the app need not store user passwords.
Much GAE and GWT information is available online. Some of the most useful resources are listed in the following sections.
Google maintains galleries of sites using GAE and GWT, both independently and together. The GWT galleries in particular can be useful in providing UI design inspiration and demonstrating the different ways in which GWT widgets can be used.
App Engine: http://appgallery.appspot.com/
Google Wave (http://wave.google.com) uses GWT as well
Google's associated documentation sites and blogs may be the first sites you bookmark:
GWT documentation : http://code.google.com/webtoolkit/, as well as a series of articles and tutorials: http://code.google.com/webtoolkit/articles.html
GAE documentation : http://code.google.com/appengine/, and
articles: http://code.google.com/appengine/articles/GAE/Java documentation : http://code.google.com/appengine/docs/java/overview.html
The official GAE blog : http://googleappengine.blogspot.com/
The official GWT blog : http://googlewebtoolkit.blogspot.com/
The Google Code blog : http://googlecode.blogspot.com/
If you are a Twitter user, Google's Twitter accounts (for example, @app_engine, @GoogleCode) are good sources for announcements as well.
The Google Groups lists for GWT and App Engine are monitored by Google employees:
The GWT Google Group : http://groups.google.com/group/google-web-toolkit/topics
The general GAE Google Group : http://groups.google.com/group/google-appengine/topics
The GAE/Java Google Group : http://groups.google.com/group/google-appengine-java/topics
Many other blogs are good resources as well. For example:
http://gae-java-persistence.blogspot.com/, written by Google employee Max Ross, is an excellent source of GAE/Java Datastore-related information and tips.
In this chapter, we have introduced Google App Engine and GWT, and given a preview of how these two technologies combine in a powerful and useful manner.
We have also described the social-media-oriented application, Connectr, which we will develop in stages as a running example throughout the book, and taken a look ahead at some of the primary approaches and patterns that the application will be used to illustrate.
In Chapter 2, Using Eclipse and the Google Plugin, we will describe how to use the Eclipse IDE and the Google Plugin for easy Google App Engine plus GWT development and deployment, and become familiar with the different components of a GAE+GWT project.