Building RESTful Web services with Go

There are many types of web services which have evolved over time. Prominent ones are :

Out of these, SOAP became popular in the early 2000s, when XML was on the top wave. The XML data format is used by various distributed systems to communicate with each other. SOAP is too complex to implement. Criticizers of SOAP point out how bulky the SOAP HTTP request is.

A SOAP request usually consists of these three basic components:

Just to perform an HTTP request and response cycle, we have to attach a lot of additional data in SOAP. A sample SOAP request looks like this:

POST /StockQuote HTTP/1.1
Host: www.stockquoteserver.com
Content-Type: text/xml; charset="utf-8"
Content-Length: nnnn
SOAPAction: "Some-URI"

<SOAP-ENV:Envelope
  xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
  SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
   <SOAP-ENV:Body>
       <m:GetLastTradePrice xmlns:m="Some-URI">
           <symbol>DIS</symbol>
       </m:GetLastTradePrice>
   </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

This is a standard example of SOAP from the W3C standard (https://www.w3.org/TR/2000/NOTE-SOAP-20000508/). If we observe carefully, it is in XML format, with special tags specifying the envelope and body. Since XML operates on a lot of namespaces to function, additional information comes into play. 

REST verbs specify an action to be performed on a specific resource or a collection of resources. When a request is made by the client, it should send this information in the HTTP request:

As we mentioned previously, REST uses the URI to decode its resource to be handled. There are quite a few REST verbs available, but six of them are used frequently. They are as follows:

If you are a software developer, you will be dealing with these six most of the time. The following table explains the operation, target resource, and what happens if the request succeeds or fails:

The numbers in the Success and Failure columns of the preceding table are HTTP status codes. Whenever a client initiates a REST operation, since REST is stateless, the client should know a way to find out whether the operation was successful or not. For that reason, HTTP has status codes for the response. REST defines the preceding status code types for a given operation. This means a REST API should strictly follow the preceding rules to achieve client-server communication.

All defined REST services have the following format. It consists of the host and API endpoint. The API endpoint is the URL path which is predefined by the server. Every REST request should hit that path.

Let us look at all the verbs in more detail. The REST API design starts with the defining of operations and API endpoints. Before implementing the API, the design document should list all the endpoints for the given resources. In the following section, we carefully observe the REST API endpoints using PayPal's REST API as a use case.

      /v1/books/?category=fiction&publish_date=2017

{"name" : "Lord of the rings", "year": 1954, "author" : "J. R. R. Tolkien"}

/v1/books/1256

{"name" : "Lord of the rings", "year": 1955, "author" : "J. R. R. Tolkien"}

/v1/books/1256

{"isbn" : "0618640150"}

Cross-Origin Resource Sharing (CORS)

The most important application of this OPTIONS method is Cross-Origin Resource Sharing (CORS). Initially, browser security prevented the client from making cross-origin requests. It means a site loaded with the URL www.foo.com can only make API calls to that host. If the client code needs to request files or data from www.bar.com, then the second server, bar.com, should have a mechanism to recognize foo.com to get its resources.

This process explains the CORS:

1. foo.com requests the OPTIONS method on bar.com.
2. bar.com sends a header like Access-Control-Allow-Origin: http://foo.com in response to the client.
3. Next, foo.com can access the resources on bar.com without any restrictions that call any REST method.

If bar.com feels like supplying resources to any host after one initial request, it can set Access control to * (that is, any).

The following is the diagram depicting the process happening one after the other:

 

You need to understand why Single Page Applications (SPA) are the hot topic today. Instead of building the UI in a traditional way (request web pages), these SPA designs make developers write code in a totally different way. There are many MVC frameworks, like AngularJS, Angular2, React JS, Knockout JS, Aurelia, and so on, to develop web UIs rapidly, but the essence of each of them is pretty simple. All MVC frameworks help us to implement one design pattern. That design pattern is No requesting of web pages, only REST API.

The modern web frontend development has advanced a lot since 2010. In order to exploit the features of Model-View-Controller (MVC) architecture, we need to consider the frontend as a separate entity which talks to the backend only using the REST API (most preferably, REST JSON).

Old and new ways of data flow in SPA

All websites go through the following steps:

1. Request a web page from the server.
2. Authenticate and show the Dashboard UI.
3. Allow the user to modify and save.
4. Request as many web pages from the server as needed to show individual pages on the site.

But in the SPA, the flow is quite different:

1. Request the HTML template/s to the browser in one single go.
2. Then, query the JSON REST API to fill a model (data object).
3. Adjust the UI according to the data in the model (JSON).
4. When users modify the UI, the model (data object) should change automatically. For example, in AngularJS, it is possible with two-way data binding. Finally, make REST API calls to notify the server about changes whenever you want.

In this way, communication happens only in the form of the REST API. The client takes care of logically representing the data. This causes systems to move from Response Oriented Architecture (ROA) to Service Oriented Architecture (SOA). Take a look at the following diagram:

 

Note

SPA reduces the bandwidth and improves the site performance.

This is a building series book. It assumes you already know the basics of Go. If not, no worries. You can jump start and learn them quickly from Go's official site at https://golang.org/. Go uses a different way of developing projects. Writing a standalone, simple program doesn't bother you much. But after learning the basics, people try to advance a step further. For that reason, as a Go developer, you should know how Go projects are laid out and the best practices to keep your code clean.

Make sure you have done the following things before proceeding:

There are many online references from which you can get to know the preceding details. Depending on your machine type (Windows, Linux, or macOS X), set up a working Go compiler. We see more details about GOPATH in the following section.

Demystifying GOPATH 

GOPATH is nothing but the current appointed workspace on your machine. It is an environment variable that tells the Go compiler about where your source code, binaries, and packages are placed.

The programmers coming from a Python background may know the Virtualenv tool to create multiple projects (with different Python interpreter versions) at the same time. But at a given time, one activates the environment and develops his project. Similarly,  you can have any number of Go projects on your machine. While developing, set the GOPATH to one of your projects. The Go compiler now activates that project.

It is a common practice to create a project under the home directory and set the GOPATH environment variable like this:

>mkdir /home/naren/myproject
export GOPATH=/home/naren/myproject

Now we install external packages like this:

go get -u -v github.com/gorilla/mux

 Go copies the project called mux into the currently activated project myproject.

Note

For Go get, use the -u flag to install updated dependencies of the external package and -v to see the verbose details of installation.

A typical Go project has the following structure, as mentioned on the official Go website:

                     

 

Let us understand this structure before digging further:

• bin: Stores the binary of our project; a shippable binary which can be run directly
• pkg: Contains the package objects; a compiled program which supplies package methods
• src: The place for your project source code, tests, and user packages

In Go, all the packages which you import into your main program have an identical structure, github.com/user/project. But who creates all these directories? Should the developer do that? Nope. It is the developer's responsibility to create directories for his/her project. It means he/she only creates the directory src/github.com/user/hello.

When a developer runs the following command, the directories bin and package are created if they did not exist before. .bin consists of the binary of our project source code and .pkg consists of all internal and external packages we use in our Go programs:

 go install github.com/user/project

With the concepts we have built upto now, let us write our first basic REST service. This service takes the number range (1-10) from the client and returns its Roman string. Very primitive, but better than Hello World.

Design:

Our REST API should take an integer number from the client and serve back the Roman equivalent.

The block of the API design document may look like this:

Implementation:

Now we are going to implement the preceding simple API step-by-step.

As we previously discussed, you should set the GOPATH first. Let us assume the GOPATH is /home/naren/go. Create a directory called romanserver in the following path. Replace narenaryan with your GitHub username (this is just a namespace for the code belonging to different users):

mkdir -p $GOPATH/src/github.com/narenaryan/romanserver

Our project is ready. We don't have any database configured yet. Create an empty file called main.go:

touch $GOPATH/src/github.com/narenaryan/romanserver/main.go

Our main logic for the API server goes into this file. For now, we can create a data file which works as a data service for our main program. Create one more directory for packaging the Roman numeral data: 

mkdir $GOPATH/src/github.com/narenaryan/romanNumerals

Now, create an empty file called data.go in the romanNumerals directory.  The src directory structure so far looks like this:

 ;

Now let us start adding code to the files. Create data for the Roman numerals:

// data.go
package romanNumerals

var Numerals = map[int]string{
  10: "X",
  9: "IX",
  8: "VIII",
  7: "VII",
  6: "VI",
  5: "V",
  4: "IV",
  3: "III",
  2: "II",
  1: "I",
}

We are creating a map called Numerals. This map holds information for converting a given integer to its Roman equivalent. We are going to import this variable into our main program to serve the request from the client. 

Open main.go and add the following code:

// main.go
package main

import (
   "fmt"
   "github.com/narenaryan/romanNumerals"
   "html"
   "net/http"
   "strconv"
   "strings"
   "time"
)

func main() {
   // http package has methods for dealing with requests
   http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
       urlPathElements := strings.Split(r.URL.Path, "/")
       // If request is GET with correct syntax
       if urlPathElements[1] == "roman_number" {
           number, _ := strconv.Atoi(strings.TrimSpace(urlPathElements[2]))
           if number == 0 || number > 10 {
           // If resource is not in the list, send Not Found status
               w.WriteHeader(http.StatusNotFound)
               w.Write([]byte("404 - Not Found"))
           } else {
             fmt.Fprintf(w, "%q", html.EscapeString(romanNumerals.Numerals[number]))
           }
       } else {
           // For all other requests, tell that Client sent a bad request
           w.WriteHeader(http.StatusBadRequest)
           w.Write([]byte("400 - Bad request"))
       }
   })
 // Create a server and run it on 8000 port
   s := &http.Server{
     Addr: ":8000",
     ReadTimeout: 10 * time.Second,
     WriteTimeout: 10 * time.Second,
     MaxHeaderBytes: 1 << 20,
   }
   s.ListenAndServe()
}

Now, install this project with the Go command install:

go install github.com/narenaryan/romanserver

This step does two things:

We can run the preceding API server as this:

$GOPATH/bin/romanserver

The server is up and running on http://localhost:8000. Now we can make a GET request to the API using a client like Browser or the CURL command. Let us fire a CURL command with a proper API GET request.

Request one is as follows:

curl -X GET "http://localhost:8000/roman_number/5" # Valid request

The response is as follows:

HTTP/1.1 200 OK
Date: Sun, 07 May 2017 11:24:32 GMT
Content-Length: 3
Content-Type: text/plain; charset=utf-8

"V"

Let us try a few incorrectly formed requests.

Request two is as follows:

curl -X GET "http://localhost:8000/roman_number/12" # Resource out of range

The response is as follows: 

HTTP/1.1 404 Not Found
Date: Sun, 07 May 2017 11:22:38 GMT
Content-Length: 15
Content-Type: text/plain; charset=utf-8

404 - Not Found

Request three is as follows:

curl -X GET "http://localhost:8000/random_resource/3" # Invalid resource

The response is as follows:

"HTTP/1.1 400 Bad request
Date: Sun, 07 May 2017 11:22:38 GMT
Content-Length: 15
Content-Type: text/plain; charset=utf-8
400 - Bad request

Our little Roman numerals API is doing the right thing. The right status codes are being returned. That is the point all API developers should keep in mind. The client should be informed why something went wrong.

Live reloading the application with supervisord and Gulp

Gulp is a nice tool for creating workflows. A workflow is a step-by-step process. It is nothing but a task streamlining application. You need NPM and Node installed on your machine. We use Gulp to watch the files and then update the binary and restart the API server. Sounds cool, right?

The supervisor is an application to reload your server whenever the application gets killed. A process ID will be assigned to your server. To restart the app properly, we need to kill the existing instances and restart the application. We can write one such program in Go. But in order to not reinvent the wheel, we are using a popular program called supervisord.  

Monitoring your Go web server with supervisord

Sometimes your web application may stop due to operating system restarts or crashes. Whenever your web server gets killed, it is supervisor's job to bring it back to life. Even the system restart cannot take your web server away from the customers. So, strictly use supervisord for your app monitoring.

Installing supervisord

We can easily install supervisord on Ubuntu 16.04, with the apt-get command:

sudo apt-get install -y supervisor

This installs two tools, supervisor and supervisorctl. supervisorctl is intended to control the supervisord and add tasks, restart tasks, and so on.

On macOS X, we can install supervisor using the brew command:

brew install supervisor

Now, create a configuration file at:

/etc/supervisor/conf.d/goproject.conf

You can add any number of configuration files, and supervisord treats them as separate processes to run. Add the following content to the preceding file:

[supervisord]
logfile = /tmp/supervisord.log

[program:myserver]
command=$GOPATH/bin/romanserver
autostart=true
autorestart=true
redirect_stderr=true

By default, we have a file called .supervisord.conf at /etc/supervisor/. Look at it for more reference. In macOS X, the same file will be located at /usr/local/etc/supervisord.ini.

Coming to the preceding configuration:

• The [supervisord] section tells the location of the log file for supervisord
• [program:myserver] is the task block which traverses to the given directory and executes the command given

Now we can ask our supervisorctl to re-read the configuration and restart the tasks (process). For that, just say:

• supervisorctl reread
• supervisorctl update

Then, launch supervisorctl with the command:

supervisorctl

You will see something like this:

Note

supervisorctl is a great tool for controlling supervisor programs.

   

Since we named our romanserver myserver in the supervisor configuration file, we can start, stop, and restart that program from supervisorctl.

With the little introduction we gave about Gulp in the preceding section, we are going to write a gulpfile for telling the computer to execute a few tasks.

I install Gulp and Gulp-shell using npm:

npm install gulp gulp-shell

After this, create a gulpfile.js in the root directory of the project. Here, it is github.com/src/narenaryan/romanserver. Now add this content to gulpfile.js. First, whenever a file changes, install binary task gets executed. Then, the supervisor will be restarted. The watch task looks for any file change and executes the preceding tasks. We are also ordering the tasks so that they occur one after the other synchronously. All of these tasks are Gulp tasks and can be defined by the gulp.task function. It takes two arguments with task name, task. sell.task allows Gulp to execute system commands:

var gulp = require("gulp");
var shell = require('gulp-shell');

// This compiles new binary with source change
gulp.task("install-binary", shell.task([
 'go install github.com/narenaryan/romanserver'
]));

// Second argument tells install-binary is a deapendency for restart-supervisor
gulp.task("restart-supervisor", ["install-binary"], shell.task([
 'supervisorctl restart myserver'
]))

gulp.task('watch', function() {
 // Watch the source code for all changes
 gulp.watch("*", ['install-binary', 'restart-supervisor']);

});

gulp.task('default', ['watch']);

Now, if you run the  gulp command in the source directory, it starts watching your source code changes:

gulp

Now, if we modify the code, then the code is compiled, installed, and the server restarted in a flash:

In this chapter, we gave an introduction to the REST API. We saw that REST is not a protocol, but an architectural pattern. HTTP is the actual protocol on which we can implement our REST service. We jumped into the fundamentals of the REST API to be clear about what they actually are. Then we explored types of web services. Before REST, we have something called SOAP, which uses XML as the data format. REST operates on JSON as the primary format. REST has verbs and status codes. We saw what a given status code refers to. We built a simple service which serves the Roman numerals for given numbers. In this process, we also saw how to package a Go project. We understood the GOPATH environment variable. It is a workspace defining a variable in Go. All packages and projects reside in that path. We then saw how to reload a development project on the fly with the help of supervisord and Gulp. These are node tools but can help us to keep our Go project up and running.

In the next chapter, we dig deeper into URL routing. Starting from the built-in router, we explore Gorilla Mux, a powerful URL routing library.