Marshalling Data Services with Ext.Direct

Learning Ext JS 3.2

, , ,
October 2010

$26.99

Build dynamic, desktop-style user interfaces for your data-driven web applications using Ext JS

        Read more about this book      

(For more resources on Ext JS, see here.)

What is Direct?

Part of the power of any client-side library is its ability to tap nearly any server-side technology. That said, with so many server-side options available there were many different implementations being written for accessing the data.

Direct is a means of marshalling those server-side connections, creating a 'one-stop-shop' for handling your basic Create, Read, Update, and Delete actions against that remote data. Through some basic configuration, we can now easily create entire server-side API's that we may programmatically expose to our Ext JS applications. In the process, we end up with one set of consistent, predefined methods for managing that data access.

Building server-side stacks

There are several examples of server-side stacks already available for Ext JS, directly from their site's Direct information. These are examples, showing you how you might use Direct with a particular server-side technology, but Ext provides us with a specification so that we might write our own. Current stack examples are available for:

  • PHP
  • .NET
  • Java
  • ColdFusion
  • Ruby
  • Perl

These are examples written directly by the Ext team, as guides, as to what we can do. Each of us writes applications differently, so it may be that our application requires a different way of handling things at the server level. The Direct specification, along with the examples, gives us the guideposts we need for writing our own stacks when necessary. We will deconstruct one such example here to help illustrate this point.

Each server-side stack is made up of three basic components:

  • Configuration— denoting which components/classes are available to Ext JS
  • API— client-side descriptors of our configuration
  • Router— a means to 'route' our requests to their proper API counterparts

To illustrate each of these pieces of the server-side stack we will deconstruct one of the example stacks provided by the Ext JS team. I have chosen the ColdFusion stack because:

  • It is a good example of using a metadata configuration
  • DirectCFM (the ColdFusion stack example) was written by Aaron Conran, who is the Senior Software Architect and Ext Services Team Leader for Ext, LLC

Each of the following sections will contain a "Stack Deconstruction" section to illustrate each of the concepts. These are to show you how these concepts might be written in a server-side language, but you are welcome to move on if you feel you have a good grasp of the material.

Configuration

Ultimately the configuration must define the classes/objects being accessed, the functions of those objects that can be called, and the length (number) of arguments that the method is expecting. Different servers will allow us to define our configuration in different ways. The method we choose will sometimes depend upon the capabilities or deficiencies of the platform we're coding to. Some platforms provide the ability to introspect components/classes at runtime to build configurations, while others require a far more manual approach. You can also include an optional formHandler attribute to your method definitions, if the method can take form submissions directly. There are four basic ways to write a configuration.

Programmatic

A programmatic configuration may be achieved by creating a simple API object of key/value pairs in the native language. A key/value pair object is known by many different names, depending upon the platform to which we're writing for: HashMap, Structure, Object, Dictionary, or an Associative Array. For example, in PHP you might write something like this:

$API = array(
'Authors'=>array(
'methods'=>array(
'GetAll'=>array(
'len'=>0
),
'add'=>array(
'len'=>1
),
'update'=>array(
'len'=>1
)
)
)
);

Look familiar? It should, in some way, as it's very similar to a JavaScript object. The same basic structure is true for our next two methods of configuration as well.

JSON and XML

For this configuration, we can pass in a basic JSON configuration of our API:


{
Authors:{
methods:{
GetAll:{
len:0
},
add:{
len:1
},
update:{
len:1
}
}
}
}

Or we could return an XML configuration object:

<Authors>
<methods>
<method name="GetAll" len="0" />
<method name="add" len="1" />
<method name="update" len="1" />
</methods>
</Authors>

All of these forms have given us the same basic outcome, by providing a basic definition of server-side classes/objects to be exposed for use with our Ext applications. But, each of these methods require us to build these configurations basically by hand. Some server-side options make it a little easier.

Metadata

There are a few server-side technologies that allow us to add additional metadata to classes and function definitions, using which we can then introspect objects at runtime to create our configurations. The following example demonstrates this by adding additional metadata to a ColdFusion component (CFC):

<cfcomponent name="Authors" ExtDirect="true">
<cffunction name="GetAll" ExtDirect="true">
<cfreturn true />
</cffunction>
<cffunction name="add" ExtDirect="true">
<cfargument name="author" />
<cfreturn true />
</cffunction>
<cffunction name="update" ExtDirect="true">
<cfargument name="author" />
<cfreturn true />
</cffunction>
</cfcomponent>

This is a very powerful method for creating our configuration, as it means adding a single name/value attribute (ExtDirect="true") to any object and function we want to make available to our Ext application. The ColdFusion server is able to introspect this metadata at runtime, passing the configuration object back to our Ext application for use.

Stack deconstruction—configuration

The example ColdFusion Component provided with the DirectCFM stack is pretty basic, so we'll write one slightly more detailed to illustrate the configuration. ColdFusion has a facility for attaching additional metadata to classes and methods, so we'll use the fourth configuration method for this example, Metadata.

We'll start off with creating the Authors.cfc class:

<cfcomponent name="Authors" ExtDirect="true">
</cfcomponent>

Next we'll create our GetAll method for returning all the authors in the database:

<cffunction name="GetAll" ExtDirect="true">
<cfset var q = "" />
<cfquery name="q" datasource="cfbookclub">
SELECT AuthorID,
FirstName,
LastName
FROM Authors
ORDER BY LastName
</cfquery>
<cfreturn q />
</cffunction>

We're leaving out basic error handling and stuff, but these are the basics behind it. The classes and methods we want to make available will all contain the additional metadata.

Building your API

So now that we've explored how to create a configuration at the server, we need to take the next step by passing that configuration to our Ext application. We do this by writing a server-side template that will output our JavaScript configuration. Yes, we'll actually dynamically produce a JavaScript include, calling the server-side template directly from within our <script> tag:

<script src="Api.cfm"></script>

How we write our server-side file really depends on the platform, but ultimately we just want it to return a block of JavaScript (just like calling a .js file) containing our API configuration description. The configuration will appear as part of the actions attribute, but we must also pass the url of our Router, the type of connection, and our namespace. That API return might look something like this:

Ext.ns("com.cc");
com.cc.APIDesc = {
"url": "\/remote\/Router.cfm",
"type": "remoting"
"namespace": "com.cc",
"actions": {
"Authors": [{
"name": "GetAll",
"len": 0
},{
"name": "add",
"len": 1
},{
"name": "update",
"len": 1
}]
}
};

This now exposes our server-side configuration to our Ext application.

Stack deconstruction—API

The purpose here is to create a JavaScript document, dynamically, of your configuration. Earlier we defined configuration via metadata. The DirectCFM API now has to convert that metadata into JavaScript. The first step is including the Api.cfm in a <script> tag on the page, but we need to know what's going on "under the hood."

Api.cfm:
<!--- Configure API Namespace and Description variable names --->
<cfset args = StructNew() />
<cfset args['ns'] = "com.cc" />
<cfset args['desc'] = "APIDesc" />
<cfinvoke component="Direct" method="getAPIScript"
argumentcollection="#args#" returnVariable="apiScript" />
<cfcontent reset="true" />
<cfoutput>#apiScript#</cfoutput>

Here we set a few variables, that will then be used in a method call. The getAPIScript method, of the Direct.cfc class, will construct our API from metadata.

Direct.cfc getAPIScript() method:

<cffunction name="getAPIScript">
<cfargument name="ns" />
<cfargument name="desc" />
<cfset var totalCFCs = '' />
<cfset var cfcName = '' />
<cfset var CFCApi = '' />
<cfset var fnLen = '' />
<cfset var Fn = '' />
<cfset var currFn = '' />
<cfset var newCfComponentMeta = '' />
<cfset var script = '' />
<cfset var jsonPacket = StructNew() />
<cfset jsonPacket['url'] = variables.routerUrl />
<cfset jsonPacket['type'] = variables.remotingType />
<cfset jsonPacket['namespace'] = ARGUMENTS.ns />
<cfset jsonPacket['actions'] = StructNew() />
<cfdirectory action="list" directory="#expandPath('.')#"
name="totalCFCs" filter="*.cfc" recurse="false" />
<cfloop query="totalCFCs">
<cfset cfcName = ListFirst(totalCFCs.name, '.') />
<cfset newCfComponentMeta = GetComponentMetaData(cfcName) />
<cfif StructKeyExists(newCfComponentMeta, "ExtDirect")>
<cfset CFCApi = ArrayNew(1) />
<cfset fnLen = ArrayLen(newCFComponentMeta.Functions) />
<cfloop from="1" to="#fnLen#" index="i">
<cfset currFn = newCfComponentMeta.Functions[i] />
<cfif StructKeyExists(currFn, "ExtDirect")>
<cfset Fn = StructNew() />
<cfset Fn['name'] = currFn.Name/>
<cfset Fn['len'] = ArrayLen(currFn.Parameters) />
<cfif StructKeyExists(currFn, "ExtFormHandler")>
<cfset Fn['formHandler'] = true />
</cfif>
<cfset ArrayAppend(CFCApi, Fn) />
</cfif>
</cfloop>
<cfset jsonPacket['actions'][cfcName] = CFCApi />
</cfif>
</cfloop>
<cfoutput><cfsavecontent variable="script">Ext.ns('#arguments.
ns#');#arguments.ns#.#desc# = #SerializeJson(jsonPacket)#;</
cfsavecontent></cfoutput>
<cfreturn script />
</cffunction>

The getAPIScript method sets a few variables (including the 'actions' array), pulls a listing of all ColdFusion Components from the directory, loops over that listing, and finds any components containing "ExtDirect" in their root meta. With every component that does contain that meta, it then loops over each method, finds methods with "ExtDirect" in the function meta, and creates a structure with the function name and number of arguments, which is then added to an array of methods. When all methods have been introspected, the array of methods is added to the 'actions' array. Once all ColdFusion Components have been introspected, the entire packet is serialized into JSON, and returned to API.cfm for output.

One item to note is that the script, when introspecting method metadata, also looks for a "ExtFormHandler" attribute. If it finds the attribute, it will include that in the method struct prior to placing the struct in the 'actions' array.

        Read more about this book      

(For more resources on Ext JS, see here.)

Routing requests

By looking at our previous example of the API output, we notice that the url attribute is pointing to a server-side file. This is the final piece of the Direct stack, a file that will 'route' requests to the proper server-side class and method for processing./p>

What is a Router

A Router is a server-side template that takes in all requests from our Ext JS application, and passes those requests off to the respective object and method for action. Ultimately, a Router is fairly agnostic; it will take any request, maybe apply some basic logic to the incoming variables, and then pass the request along according to data in the transaction.

Transactions

When we send data to our Router, we're creating a transaction. Data coming from our Ext JS application will come in one of the two ways: form post (like when we upload files), or a raw HTTP post of a JSON packet. The parameters of each type are slightly different.

A standard JSON post will look like this:

{"action":"Authors","method":"GetAll","data":[],"type":"rpc","tid":27}

It's important to notice that Direct passed along information that came directly from our configuration, such as the action and method we're trying to access. Each of these transactions would contain the following:

  • action— the class/ object/component we're attempting to access.
  • method— the method/function we're trying to invoke.
  • data— the arguments being sent to the method, in the form of an array.
  • type— we'll use "rpc" for all remote calls.
  • tid— our own transaction ID. This helps us identify responses from the server, connecting them to whichever transaction was initiated. This is especially beneficial whenever you batch process multiple requests, as it helps to tell us what was requested, for which, by what, and place our return results where they are needed.

Stack deconstruction—HTTP post transaction

DirectCFM's Router will first create an instance of it's Direct.cfc class, then pull in the request and deserialize it , casting it into a native ColdFusion variable.

<cfset direct = CreateObject('component', 'Direct') />
<cfset postBody = direct.getPostBody() />
<cfset requests = DeserializeJSON(postBody) />

If the variable isn't an array, it will copy it into a temp variable, create an array, and place the temp object as the first element of the array. This is to maintain consistent process.

<cfif NOT IsArray(requests)>
<cfset tmp = requests />
<cfset requests = ArrayNew(1) />
<cfset requests[1] = tmp />
</cfif>

We now need to loop over the requests that came from our Ext JS application. For each element of the array, we call the method that was requested.

<cfset result = direct.invokeCall(curReq) />

invokeCall() is a method of the Direct.cfc class, which is used to dynamically call a method. It's very generic by nature. The curReq variable, being passed in as an object, is an element of our requests array.

Direct.cfc invokeCall() method:

<cffunction name="invokeCall">
<cfargument name="request" />
<cfset var idx = 1 />
<cfset var mthIdx = 1 />
<cfset var result = '' />
<cfset var args = StructNew() />
<!--- find the methods index in the metadata --->
<cfset newCfComponentMeta = GetComponentMetaData(request.action) />
<cfloop from="1" to="#arrayLen(newCfComponentMeta.Functions)#"
index="idx">
<cfif newCfComponentMeta.Functions[idx]['name'] eq request.method>
<cfset mthIdx = idx />
<cfbreak />
</cfif>
</cfloop>
<cfif NOT IsArray(request.data)>
<cfset maxParams = 0 />
<cfelseif ArrayLen(request.data) lt ArrayLen(newCfComponentMeta.
Functions[mthIdx].parameters)>
<cfset maxParams = ArrayLen(request.data) />
<cfelse>
<cfset maxParams = ArrayLen(newCfComponentMeta.Functions[mthIdx].
parameters) />
</cfif>
<!--- marry the parameters in the metadata to params passed in the
request. --->
<cfloop from="0" to="#maxParams - 1#" index="idx">
<cfset args[newCfComponentMeta['Functions'][mthIdx].
parameters[idx+1].name] = request.data[idx+1] />
</cfloop>
<cfinvoke component="#request.Action#" method="#request.method#"
argumentcollection="#args#" returnvariable="result">
<cfreturn result />
</cffunction>

Here we find the action class, then the method, and then verify that the method was passed with enough parameters in the request. After an arguments object is created we finally dynamically invoke the class related to the action, calling the method requested. The result is sent back to the Router.

Form transactions

Form posts send a slightly different data set, forming their attributes in such a way as to easily differentiate a form post from a standard HTTP post:

  • extAction—The class/object/component to use
  • extMethod—The method that will process our form post
  • extTID—The transaction ID of the request (form posts cannot batch requests)
  • extUpload—An optional argument for a file field used for file upload
  • Any other fields needed for our method

Our Router should take in the request, call the correct class/object/component, and invoke the appropriate method. We know the object to call, from our action/ extAction parameters, as well as the method, from our method parameter. This makes it simple to write dynamic statements for invoking the proper server-side architecture for our requests.

Stack deconstruction—form transactions

DirectCFM handles form post transactions in nearly the same manner that it handled the JSON requests. First, we see if it is a form post. In ColdFusion, this is done by looking for values in the form scope.

<cfif NOT StructIsEmpty(form)>
</cfif>

If it is, we'll set up our initial object to hold our JSON return.

<cfset jsonPacket = StructNew() />
<cfset jsonPacket['tid'] = form.extTID />
<cfset jsonPacket['action'] = form.extAction />
<cfset jsonPacket['method'] = form.extMethod />
<cfset jsonPacket['type'] = 'rpc' />

We can then dynamically invoke the action (class) and method requested during the form post.

<cfinvoke component="#form.extAction#" method="#form.extMethod#"
argumentcollection="#form#" returnVariable="result" />

Now all we have to do is return the results of our method calls.

Response

The final piece of the puzzle is to take the return from our dynamically invoked methods and formulate a proper response object to pass back to our Ext JS applications. The response should be a JSON encoded array of each transaction. The response of each transaction should contain the following:

  • type—"rpc"
  • tid—the transaction ID
  • action—the class/object/component that was called
  • method—the method we invoked
  • result—the result of the method call

This would give us a response similar to:

[{
"type":"rpc",
"tid":14,
"action":"Authors",
"method":"GetAll",
"result":{
"COLUMNS":"ID,FIRSTNAME,LASTNAME",
"DATA":[
[1,"Stephen","King"],
[2,"Robert","Ludlum"]
]
}
},{
"type":"rpc",
"tid":15,
"action":"Authors",
"method":"add",
"result":{
"success":true,
"ID":3
}
}]

Stack deconstruction—JSON HTTP response

The Router builds out the rest of the response object, carefully to remove the "DATA" used to make the request.

Router.cfm:
<cfif IsStruct(result) AND StructKeyExists(result, 'name') AND
StructKeyExists(result, 'result')>
<cfset curReq['name'] = result.name />
<cfset curReq['result'] = result.result />
<cfelse>
<cfset curReq['result'] = result />
</cfif>
<cfset StructDelete(curReq, 'data') />

The last thing to be done is to return the data as JSON output:

<cfcontent reset="true" /><cfoutput>#SerializeJson(requests)#</
cfoutput>

Form post responses are only slightly different. A standard post method will return the result in the same manner, except the JSON response is not part of an array. In the case of a file upload, the JSON should be wrapped inside a <textarea> element within a valid HTML document:

<html>
<body>
<textarea>{"type":"rpc","tid":15,"action":"Authors","method":"uplo
adPhoto","result":{"success":true,"ID":3}}</textarea>
</body>
</html>

Notice there's only one response here. Remember that form posts cannot handle batch requests, so we can only do one at a time.

Stack deconstruction—form post response

We can serialize the JSON result of the method, and output the response back to the browser, conditionalizing the response for those requests that were file uploads.

<cfset jsonPacket['result'] = result />
<cfset json = SerializeJson(jsonPacket) />
<cfif form.extUpload eq "true">
<cfoutput>
<cfsavecontent variable="output"><html><body><textarea>#json#</
textarea></body></html></cfsavecontent>
</cfoutput>
<cfelse>
<cfset output = json />
</cfif>
<cfcontent reset="true" />
<cfoutput>#output#</cfoutput>

Exception responses

Ok, we all make mistakes. Because of this, we have to be able to write in exception handling. An exception response is a basic JSON packet, containing the following info:

  • type—'exception'
  • message—an informative message about the error that occurred
  • where—tells us where the error occurred on the server

Exception responses are only handled when the Router is configured in debugging mode, and it is suggested that you do not return these responses in a production environment, as it could expose vital information about your server's file system that would pose a security risk.

Stack deconstruction—exceptions

ColdFusion allows us to use standard try/catch error handling for capturing errors for graceful error handling. If an error were to occur in our method handling, we could 'catch' that error and form our Exception Response:

<cfcatch type="any">
<cfset jsonPacket = StructNew() />
<cfset jsonPacket['type'] = 'exception' />
<cfset jsonPacket['tid'] = curReq['tId'] />
<cfset jsonPacket['message'] = cfcatch.Message />
<cfset jsonPacket['where'] = cfcatch.TagContext.Line/>
<cfcontent reset="true" />
<cfoutput>#SerializeJson(jsonPacket)#</cfoutput><cfabort/>
</cfcatch>

This creates a new jsonPacket structure, to which we apply the necessary key/ value pairs required for an exception object. We then output the JSON response, and abort further processing of the method.

Putting the pieces together

Now that we have a basic understanding of how we can create a stack, it's time to tie all of the pieces together to use it with our Ext JS applications.

Make your API available

The first thing we need to do, in order to be able to access our configuration, is include the API within our application. This was touched on briefly in our review of the API. All it requires us to do is include a <script> tag within our application, calling the server-side script that renders the API configuration:

<script src="Api.cfm"></script>

We need to include this after our Ext base script files, but prior to our own application files. Next, we need to attach that API as a Provider for Ext.Direct, within our JavaScript application file:

Ext.Direct.addProvider(com.cc.APIDesc);

This now gives us access to any actions and methods provided through our configuration.

Making API calls

Now that our actions and methods are available to our application, it comes down to making actual requests. Say, for instance, that you wanted to create a data store object to fill in a grid. You could do this by creating a DirectStore object, calling one of the exposed methods to populate the store:

var dirStore = new Ext.data.DirectStore({
storeId: 'Authors',
api: {
read: com.cc.Authors.GetAll,
create: com.cc.Authors.add,
update: com.cc.Authors.update,
delete: com.cc.Authors.delete
},
paramsAsHash: false,
autoSave: false,
reader: new Ext.data.CFQueryReader({
idProperty: 'AuthorId'
},[
{name:'AuthorID',type:'int'},
'FirstName',
'LastName'
]),
writer: new Ext.data.JsonWriter({
encode: false,
writeAllFields: true
})
});

We begin with our DirectStore configuration. The DirectStore is a new object of the Ext.data package, specifically designed to call data from an Ext.Direct source. The api attribute is now used to define actions and basic CRUD methods against our store, but notice that the majority of the configuration is just like any other data store.

Next, we'll apply this store to a basic data EditorGridPanel:

// shorthand alias
var fm = Ext.form;
var dirGrid = new Ext.grid.EditorGridPanel ({
width: 400,
height: 500,
title: 'Ext.DirectStore Example',
store: dirStore,
loadMask: true,
clicksToEdit: 1,
columns: [{
header: 'AuthorID',
dataIndex: 'AuthorID',
width: 60
},{
header: 'First Name',
dataIndex: 'FirstName',
width: 150,
editor: new fm.TextField({
allowBlank: false
})
},{
header: 'Last Name',
dataIndex: 'LastName',
width: 150,
editor: new fm.TextField({
allowBlank: false
})
}],
viewConfig: {
forceFit: true
},
tbar:[{
text: 'New Author',
handler: function(){
var author = dirGrid.getStore().recordType;
var a = new author({
AuthorID: 0,
});
dirGrid.stopEditing();
dirStore.insert(0,a);
dirGrid.startEditing(0,1);
}
}],
bbar:[{
text: 'Save Changes',
handler: function(){
console.log(dirStore.save());
}
}]
});
dirGrid.render('chap16');
dirStore.load();

There's nothing new here. We apply our store to our editor grid configuration just as we would apply one to any other store object. Once we render the grid, we then load the store from our Direct exposed configuration, just as we would load any other store.

Going back to our class object, we have an add and an update method, both taking an author as an argument. The add method should return a valid AuthorID in its response, which we might want to apply to a new record in our store. We've added a button to the TopToolbar to add records to the grid for editing:

tbar:[{
text: 'New Author',
handler: function(){
var author = dirGrid.getStore().recordType;
var a = new author({
AuthorID: 0,
});
dirGrid.stopEditing();
dirStore.insert(0,a);
dirGrid.startEditing(0,1);
}
}],

Once a new record has been added, or an existing record has been edited, we need to save those changes to our remote server. Here we've added a button to the bottom Toolbar that will save any new and changed records to the server, by invoking the appropriate method to take upon the record:

bbar:[{
text: 'Save Changes',
handler: function(){
dirStore.save();
}
}]

New records will go to the add method of the Author action. Changed records will go to the update method of the Author action. Since this is a JSON request, we can batch-process our changes.

Summary

In this article, we've discussed how a developer can write his/her own Ext.Direct server-side stacks for marshalling data services under a single configuration.

We've talked about the different ways of writing a configuration.

We've also deconstructed one example stack provided by the Ext JS development team, illustrating how the server-side code might generate our APIs, and route our requests. Finally, we talked about how we implement Direct within our own Ext applications.


Further resources on this subject:


Books to Consider

comments powered by Disqus