Debugging REST Web Services

Exclusive offer: get 50% off this eBook here
RESTful PHP Web Services

RESTful PHP Web Services — Save 50%

Learn the basic architectural concepts and step through examples of consuming and creating RESTful web services in PHP

$23.99    $12.00
by Samisa Abeysinghe | July 2009 | Architecture & Analysis MySQL PHP Web Development

Learning how to figure out why things are going wrong is one of the key aspects of developing software. We call it debugging. While dealing with REST services and clients, things can go wrong and it would help a great deal to know how to find out what is causing the problems.

In this article by Samisa Abeysinghe, we will look into the techniques such as message capturing and analyzing to get to know if things are going fine, and if not, what sort of problems are causing trouble.

(For more resources on this subject, see here.)

Message tracing

The first symptom that you will notice when you are running into problems is that the client would not behave the way you want it to behave. As an example, there would be no output, or the wrong output.

Since the outcome of running a REST client depends on the request that you send over the wire and the response that you receive over the wire, one of the first things is to capture the messages and verify that those are in the correct expected format.

REST Services and clients interact using messages, usually in pairs of request and response. So if there are problems, they are caused by errors in the messages being exchanged.

Sometimes the user only has control over a REST client and does not have access to the implementation details of the service. Sometimes the user will implement the REST service for others to consume the service. Sometimes the Web browser can act as a client. Sometimes a PHP application on a server can act as a REST client. Irrespective of where the client is and where the service is, you can use message capturing tools to capture messages and try to figure out the problem. Thanks to the fact that the service and client use messages to interact with each other, we can always use a message capturing tool in the middle to capture messages. It is not that we must run the message capturing tool on the same machine where the client is running or the service is running; the message capturing tool can be run on either machine, or it can be run on a third machine.

The following figure illustrates how the message interaction would look with a message capturing tool in place.

Debugging REST Web Services

If the REST client is a Web browser and we want to capture the request and response involved in a message interaction, we would have to point the Web browser to message capturing tool and let the tool send the request to the service on behalf of the Web browser. Then, since it is the tool that sent the request to the service, the service would respond to the tool. The message capturing tool in turn would send the response it received from the service to the Web browser. In this scenario, the tool in the middle would gain access to both the request and response. Hence it can reveal those messages for us to have a look.

When you are not seeing the client to work, here is the list of things that you might need to look for:

  • If the client sends a message
  • If you are able to receive a response from a service
  • If the request message sent by the client is in the correct format, including HTTP headers
  • If the response sent by the server is in the correct format, including the HTTP headers

In order to check for the above, you would require a message-capturing tool to trace the messages.

There are multiple tools that you can use to capture the messages that are sent from the client to the service and vice versa. Wireshark (http://www.wireshark.org/) is one such tool that can be used to capture any network traffic. It is an open-source tool and is available under the GNU General Public License version 2. However this tool can be a bit complicated if you are looking for a simple tool.

Apache TCPMon (http://ws.apache.org/commons/tcpmon/) is another tool that is designed to trace web services messages. This is a Java based tool that can be used with web services to capture the messages. Because TCPMon is a message capturing tool, it can be used to intercept messages sent between client and service, and as explained earlier, can be run on the client machine, the server machine or on a third independent machine. The only catch is that you need Java installed in your system to run this tool. You can also find a C-based implementation of a similar tool with Apache Axis2/C (http://ws.apache.org/axis2/c). However, that tool does not have a graphical user interface.

There is a set of steps that you need to follow, which are more or less the same across all of these tools, in order to prepare the tool for capturing messages.

  • Define the target host name
  • Define the target port number
  • Define the listen port number

Target host name is the name of the host machine on which the service is running. As an example, if we want to debug the request sent to the Yahoo spelling suggestion service, hosted at http://search.yahooapis.com/WebSearchService/V1/spellingSuggestion, the host name would be search.yahooapis.com. We can either use the name of the host or we can use the IP address of the host because the tools are capable of dealing with both formats in place of the host name. As an example, if the service is hosted on the local machine, we could either use localhost or 127.0.0.1 in place of the host name.

Target port number is the port number on which the service hosting web server is listening; usually this is 80. As an example, for the Yahoo spelling suggestion service, hosted at http://search.yahooapis.com/WebSearchService/V1/spellingSuggestion, the target port number is 80. Note that, when the service URL does not mention any number, we can always use the default number. If it was running on a port other than port 80, we can find the port number followed by the host name and preceded with character ':'. As an example, if we have our web server running on port 8080 on the local machine, we would have service URL similar to http://localhost:8080/rest/04/library/book.php. Here, the host name is localhost and the target port is 8080.

Listen port is the port on which the tool will be listening to capture the messages from the client before sending it to the service. For an example, say that we want to use port 9090 as our listen port to capture the messages while using the Yahoo spelling suggestion service. Under normal circumstances, we will be using a URL similar to the following with the web browser to send the request to the service.

http://search.yahooapis.com/WebSearchService/V1/spellingSuggestion
?appid=YahooDemo&query=apocalipto

When we want to send this request through the message capturing tool and since we decided to make the tools listen port to be 9090 with the tool in the middle and assuming that the tool is running on the local machine, we would now use the following URL with the web browser in place of the original URL.

http://localhost:9090/WebSearchService/V1/spellingSuggestion?appid
=YahooDemo&query=apocalipto

Note that we are not sending this request directly to search.yahooapis.com, but rather to the tool listening on port 9090 on local host. Once the tool receives the request, it will capture the request, forward that to the target host, receive the response and forward that response to the web browser.

The following figure shows the Apache TCPMon tool. You can see localhost being used as the target host, 80 being the target port number and 9090 being the listening port number. Once you fill in these fields you can see a new tab being added in the tool showing the messages being captured.

Debugging REST Web Services

Once you click on the Add button, you will see a new pane as shown in the next figure where it will show the messages and pass the messages to and from the client and service.

Debugging REST Web Services

Before you can capture the messages, there is one more step. That is to change the client code to point to the port number 9090, since our monitoring tool is now listening on that port. Originally, we were using port 80


$url = 'http://localhost:80/rest/04/library/book.php';

or just

$url = 'http://localhost/rest/04/library/book.php';

because the default port number used by a web server is port 80, and the client was directly talking to the service. However, with the tool in place, we are going to make the client talk to the tool listening on port 9090. The tool in turn will talk to the service. Note that in this sample we have all three parties, the client, the service, and the tool running on the same machine. So we will keep using localhost as our host name.

Now we are going to change the service endpoint address used by the client to contain port 9090. This will make sure that the client will be talking to the tool.

$url = 'http://localhost:9090/rest/04/library/book.php';

Debugging REST Web Services

As you can see, the tool has captured the request and the response. The request appears at the top and the response at the bottom. The request is a GET request to the resource located at /rest/04/library/book.php. The response is a success response, with HTTP 200 OK code. And after the HTTP headers, the response body, which is in XML follows.

As mentioned earlier, the first step in debugging is to verify if the client has sent a request and if the service responded. In the above example, we have both the request and response in place. If both were missing then we need to check what is wrong on either side.

If the client request was missing, you can check for the following in the code.

  • Are you using the correct URL in client
  • Have you written the request to the wire in the client? Usually this is done by the function curl_exec when using Curl

If the response was missing, you can check for the following.

  • Are you connected to the network? Because your service can be hosted on a remote machine
  • Have you written a response from the service? That is, basically, have you returned the correct string value from the service? In PHP wire, using the echo function to write the required response to the wire usually does this. If you are using a PHP framework, you may have to use the framework specific mechanisms to do this. As an example, if you are using the Zend_Rest_Server class, you have to use handle() method to make sure that the response is sent to the client.

Here is a sample error scenario.

Debugging REST Web Services

As you can see, the response is 404 not found. And if you look at the request, you see that there is a typo in the request. We have missed 'k' from our resource URL, hence we have sent the request to /rest/04/library/boo.php, which does not exist, whereas the correct resource URL is /rest/04/library/book.php.

Next let us look at the Yahoo search example that was discussed earlier to identify some advanced concepts. We want to capture the request sent by the web browser and the response sent by the server for the request. http://search.yahooapis.com/WebSearchService/V1/spellingSuggestion?appid=YahooDemo&ampquery=apocalipto.

As discussed earlier, the target host name is search.yahooapis.com. The target port number is 80. Let's use 9091 as the listen.

Debugging REST Web Services

Let us use the web browser to send the request through the tool so that we can capture the request and response. Since the tool is listening on port 9091, we would use the following URL with the web browser. http://localhost:9091/WebSearchService/V1/spellingSuggestion?appid=YahooDemo&ampquery=apocalipto

Debugging REST Web Services

When you use the above URL with the web browser, the web browser would send the request to the tool and the tool will get the response from the service and forward that to the web browser. We can see that the web browser gets the response.

However, if we have a look at the TCPMon tool's captured messages, we see that the service has sent some binary data instead of XML data even though the Web browser is displaying the response in XML format.

Debugging REST Web Services

So what went wrong? In fact, nothing is wrong. The service sent the data in binary format because the web browser requested that format. If you look closely at the request sent you will see the following.

GET /WebSearchService/V1/spellingSuggestion?appid=YahooDemo&query=apocalipto HTTP/1.1
Host: search.yahooapis.com:9091
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9)
Gecko/20071025 Firefox/2.0.0.9
Accept: text/xml,application/xml,application/xhtml+xml,
text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.7,zh-cn;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

In the request, the web browser has used the HTTP header.

Accept-Encoding: gzip,deflate

This tells the service that the web browser can handle data that comes in gzip compressed format. Hence the service sends the data compressed to the web browser. Obviously, it is not possible to look into the XML messages and debug them if the response is compressed. Hence we should ideally capture the messages in XML format. To do this, we can modify the request message on the TCPMon pane itself and resend the message.

First remove the line

Accept-Encoding: gzip,deflate

Then click on the Resend button.

Debugging REST Web Services

Once we click on the Resend button, we will get the response in XML format.

Debugging REST Web Services

Errors in building XML

While forming XML as request payload or response payload, we can run into errors through simple mistakes. Some would be easy to spot but some are not. Most of the XML errors could be avoided by following a simple rule of thumb-each opening XML tag should have an equivalent closing tag. That is the common mistake that can happen while building XML payloads.

Debugging REST Web Services

In the above diagram, if you look carefully in the circle, the ending tag for the book element is missing. A new starting tag for a new book is started before the first book is closed. This would cause the XML parsing on the client side to fail. In this case I am using the Library system sample and here is the PHP source code causing the problem.

echo "<books>";
while ($line = mysql_fetch_array($result, MYSQL_ASSOC)) {
echo "<book>";
foreach ($line as $key => $col_value) {
echo "<$key>$col_value</$key>";
}
//echo "</book>";
}
echo "</books>";

Here I have intentionally commented out printing the closing tag to demonstrate the error scenario. However, while writing this code, I could have missed that as well, causing the system to be buggy.

While looking for XML related errors, you can use the manual technique that we just used. Look for missing tags. If the process looks complicated and you cannot seem to find any XML errors in the response or request that you are trying to debug, you can copy the XML captured with the tool and run it through an XML validator tool. For example, you can use an online tool such as http://www.w3schools.com/XML/xml_validator.asp. You can also check if the XML file is well formed using an XML parser

RESTful PHP Web Services Learn the basic architectural concepts and step through examples of consuming and creating RESTful web services in PHP
Published: October 2008
eBook Price: $23.99
Book Price: $39.99
See more
Select your format and quantity:

 

Errors in parsing XML

There also can be situations where you could make mistakes while parsing an incoming XML message. As an example, have a look at the following code.

foreach ($xml->book as $book) {
echo "$book->id, $book->name, $book->author, $book->isbn <br/>n";
}

Here, the parsing logic assumes the following XML format.

<books>
<book>
<id>1</id>
<name>Book1</name>
<author>Auth1</author>
<isbn>ISBN0001</isbn>
</book>
</books>

There are two common possibilities where typos can happen while parsing an XML. First, the foreach statement.

foreach ($xml->book as $book) {

Here, the common mistake is to forget that the root element, book in this example corresponds to the $xml element. It is a common mistake to use

$xml->books->book

in place of

$xml->book

The second possibility is to use incorrect element names while using the child elements of a given element. In this example, we use

echo "$book->id, $book->name, $book->author, $book->isbn <br/>n";

If we were to use

$book->ISBN

instead of

$book->isbn

The client would not behave as expected and would not print the ISBN number in the output. This is because XML is case sensitive and the incoming XML has the element name <isbn> and not <ISBN>.

Best practices

Best practices would help us avoid common mistakes while using REST style services and clients.

  • Make sure that you are using the correct HTTP method. You can trace the messages and have a look at the HTTP method used in the request. Note that the HTTP verb being used has significance in the REST operations. Therefore, it is good practice to always pay attention to the HTTP verb being used while invoking operations.
  • Always verify the integrity of your XML messages, both request and response. Check for exact spelling of field key names because, as mentioned earlier, XML is case sensitive. Also check if all mandatory fields are in place. Often the service API document would clearly highlight and distinguish mandatory and optional fields. Some API documents would not mention if a field is mandatory or not. In that case you might have to assume that every field is mandatory.
  • If you happen to run into problems and cannot figure out what is going wrong in case of the business logic of a service, try to run the business logic as a standalone program rather than making it a service. In other words, make sure you do comprehensive unit testing of the functionality. That will help reduce the complexity to help locate the problems easily
  • The above style of divide and conquer approach can also be applied on client side as well. As an example, if you seem to have problems with parsing the response, use a sample XML file that has the identical structure to that of response and try to test the parsing logic standalone.
  • If you are using third-party services that you did not write, make sure you have access to comprehensive documentation that explains the message formats and request parameters. Especially the sample messages, which indicate what the request and response would look like, would help with debugging.
  • Look for sample code segments provided to you by the service provider. Most public service APIs come with such examples.
  • If you are using public services such as Google, Yahoo, etc. there are wrapper PHP classes provided by the service vendors to help you consume the services. Look for those classes, rather than writing it from scratch. The Zend framework also provides a set of such wrapper classes for some public REST services.
  • If you are using a framework for your application, look for framework specific logging and debugging features. Most frameworks provide you with comprehensive logging along with the ability to customize log levels to help you locate problems.
  • If you are planning on providing REST services of your own, make sure that you clearly document the service API along with all mandatory and optional fields. Most of the time, just pointing users to the documents would solve a considerable amount of problems.
  • Try to avoid re-inventing the HTTP verbs mappings if you are designing services. Stick to the basic meanings of HTTP verbs. For example, use GET to retrieve data and POST for updating data. Do not use POST to retrieve data.
  • Keep in mind that the client and the service are two independent entities. All the interaction between those two takes place using messages. Hence avoid making any assumptions about either party. All information that is required for the interaction must be self-contained within the messages passed between the client and the service. This interaction must be stateless, meaning that each request/response pair is executed independently without any knowledge of the request/response that happened before or that is to happen in the future. This will make sure that the client and service interaction is kept atomic, and ensure that the interactions are simple.
  • While designing services, make sure the request and response message formats are kept simple. This is because, more complex the messages formats are the harder it is to debug applications. At the same time, the number of interactions between the client and the service that are required to get some effective work done must also be kept to a minimum. This is because the more interactions we need the more we will have to use the network and the latency of the network would be the governing factor that would determine our application's performance. Hence the right balance between the message size and the number of request/response interactions must be determined. The rule of thumb to follow would be to do whatever necessary, not more and not less.

Summary

We looked into the use of tools to trace and look into the messages to figure out possible problems with request and response pairs passed between clients and services. We also looked into how we could look at the XML to figure out possible problems in building XML. We also discussed how we can locate problems in parsing an incoming XML.


Further resources on this subject:


RESTful PHP Web Services Learn the basic architectural concepts and step through examples of consuming and creating RESTful web services in PHP
Published: October 2008
eBook Price: $23.99
Book Price: $39.99
See more
Select your format and quantity:

About the Author :


Samisa Abeysinghe

Samisa Abeysinghe is Director, Engineering at WSO2. Samisa pioneered the Apache Axis2/C effort and architected the core of the Apache Axis2/C Web services engine. He continues to be an active contributor in the Apache Axis2/C project. His involvement in open-source projects began in 2004 when he started working with the Apache Axis C/C++ project.

Prior to his current role, Samisa played the project lead role for the WSO2 Web services Framework for PHP project, which provides comprehensive support for building both SOAP and REST services and clients.

Books From Packt

Grails 1.1 Web Application Development
Grails 1.1 Web Application Development

PHP and script.aculo.us Web 2.0 Application Interfaces
PHP and script.aculo.us Web 2.0 Application Interfaces

JBoss Tools 3 Developers Guide
JBoss Tools 3 Developers Guide

WordPress 2.7 Complete
WordPress 2.7 Complete

Drupal 5 Views Recipes
Drupal 5 Views Recipes

Flash with Drupal
Flash with Drupal

SAP Business ONE Implementation
SAP Business ONE Implementation

Choosing an Open Source CMS: Beginner's Guide
Choosing an Open Source CMS: Beginner's Guide

No votes yet

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
M
q
K
P
c
U
Enter the code without spaces and pay attention to upper/lower case.
Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software