MASTERING NSERVICEBUS AND PERSISTENCE

During the course of building enterprise systems, there are functional and nonfunctional requirements. Functional requirements describe the business rules, and nonfunctional requirements are system characteristics with non-business rules. A simple nonfunctional requirement for a system is, for instance, that any SSN must be encrypted both at rest and in-transient states. Nonfunctional requirements simply go beyond security requirements; nonfunctional requirements include notifications, alerts, monitoring, logging, and other software qualities.

Nonfunctional requirements include many of the components that make up software quality http://en.wikipedia.org/wiki/Software_quality. Software quality includes some of the software characteristics already mentioned, such as maintainability, security, code quality, reliability, integrity, and so on. Software quality is the ideal state for software to achieve; nonfunctional requirements form the specifics of how to achieve certain pieces.

The problem is that, while business requirements may be clearly spelled out, nonfunctional requirements may not be defined clearly or negotiated enough ahead of time. Therefore, tweaks are required along the way during the application life cycle, including development or maintenance. Metadata and precreated frameworks are the key players of this tweaking.

Consider an administration application that business analysts (BAs) and operational teams use to check the current state of an enterprise application. The application takes orders for aircraft maps and equipment, and customer service representatives (CSRs) have an interface for working with the customers and changing their data at will. Operations use an administration application to monitor the end-to-end throughput from a browser to a database and receive notifications if the levels are not achieved.

In the previous example, notifications and monitoring are nonfunctional requirements. BAs may use the administration application to handle special customer cases and monitor the number of orders, customers, and other reports. The generation of the reports, the data for monitoring, is based on the business data and generates metadata. This metadata is used to check the business data.

The following is a common 3-tier diagram for an application that gathers sales information:

The application has a frontend, a logic tier (middle tier), and a data tier. So far, this is a very common design for an application. The frontend is done in HTML or ASP.NET to control the presentation layer in a browser. The logic tier contains the workflow and messaging to handle business logic. Finally, the data tier is the storage to hold the information in a persisted repository—usually a database, mainframe, file I/O, or third-party server among other options.

When you look at this basic application, you'll realize that many endpoints are missing. These endpoints are used to monitor the application, to log the application, and perform other operational and administration tasks previously mentioned. Therefore, this model is incomplete since it does not address nonfunctional requirements.

Many software projects seem to need continuous enhancements because the developer keeps on adding components for security, operational reports, and other application characteristics that were not mentioned in the list of business requirements, even though they are components required to ensure the integrity of the application itself.

To paraphrase what's written in http://en.wikipedia.org/wiki/Service_oriented_architecture, the idea behind Service-oriented Architecture (SOA) is to decouple the end-to-end application functionality between discreet services.

So far, we have discussed sagas and some metadata of applications. There are other types of data that are saved to the data store, including business objects that contain the information used for business rules. Business rules run the business engines and are used to execute business logic.

In the ESB world, the bus transports (moves) objects that could be considered business objects; these business objects move through sagas. These objects are the pieces of NSBs that are used for notifications, timeouts, gateways for message distribution, Second–level Retries (SLRs), and even endpoints to where the messages are sent.

The preceding objects make up many of the application metadata. Many of these are the configurations of the services that make up the distribution of the messages and the behavior of the transactions. The metadata that NSB keeps track of during a publish-subscribe message pattern is the same subscription information required for NSB to keep track of the publish-subscribe endpoints. The subscription information is needed for the subscribers to keep track of the message types and queue endpoints. This is needed to subscribe to the publishers. NSB uses the database to keep track of these types of endpoints.

A small table of what is available can be seen at http://docs.particular.net/nservicebus/persistence-in-nservicebus.

The persistence configurations are just some of the typical ESB service configurations in NSB. There are many more configurations as NSB is meant to do so much more as a complete automation framework for the middleware. We will be discussing the various features and their associated configurations on the bus called IBus throughout this book.

Through this table, we know that the timeout for sagas, the saga object itself, the subscription information for publish-subscribe, the second-level retries, the fault management, notification, the gateway, and distributor can be supported in MSMQ. Some of these pieces can be stored in the local memory of the host application; it cannot be saved when the application is not running. Pieces can be saved in the RavenDB database, which is a NoSQL document-oriented database. Pieces can also be saved using the NHibernate database connecter, which is an ORM mapper to various relational databases, such as SQL Server, MySQL, and Oracle. Some of the items have been referred to as data, which is data that describes the messages versus the messages themselves that will be part of the ESB workflow. The workflow itself makes up the business logic, while the messages themselves could be considered as business objects.

The benefit of NServiceBus is that it will handle the persisting of the object's messages and various pieces for the developer, as long as the developer has configured NSB correctly.

For instance, when using NHibernate, NSB will perform the mapping of the messages to the relational database, and the developer does not have to configure the NHibernate-mapping properties to map the objects to the relational database. This saves the developer a lot of time and effort. The messages themselves can also be persisted through various means using the settings for using the transport in IBus configurations. These message queues include MSMQ, Azure queues, SQL Server queues, ActiveMQ, and RabbitMQ.

Many books are written on just various troubleshooting issues over networks and servers. There are many issues that come up in operations and maintenance that were never conceived as potential issues, anywhere from intermittent routers due to a power cord not being plugged in all the way, patches that left the servers in a hung state, DNS errors from a domain controller, and so on. There is no guarantee that the networks, or servers, are secure, remain unchanged, and all the routes remain reliable for the application that was built. Not having to deal with these abnormal issues by having someone else deal with the uptime issues is what makes cloud computing so attractive. In many enterprise applications, as in this usage, we discuss where uptime is critical, and where it is normal to have to code, notification, and monitoring, for failure along every step of the way between services and clients. There are many assumptions that we can make, including the one that it is someone else's concern; however, in the end, it becomes a piece of the application's responsibility to describe how it is working.

Because the network may not be reliable, there may be a changeover in staff and servers. The need for persistent enterprise objects, such as bus technology and persistent messaging, has evolved. Also, the need for instrumentation has grown to track the messages and objects. Not knowing where payments and orders are in a system can be bad for any organization that needs to track them. In the end, the data that runs through applications is owned by the organization; if it is hacked, if financial data is lost, or if employees are not paid, it is their responsibility, rather than considering that it lives in the cloud or it is the fault of a bad network or any other condition. Because of this need for reporting on the systems, there is a need for metadata, which is just another form of persisting the company's data, except for business data such as a customer's address. Metadata is a form of reporting data, such as the current state of a message or if there was an error with a message reaching its endpoint. It is a snapshot in the organization's operations of applications. Sometimes these snapshots are very important; in many cases, where money and personal identifying information are involved, they are used to provide information, even to courts, on what happened when the money goes missing. We will start on this journey of running through the designing of systems with a common SOA design pattern called saga that will assist us in providing these pieces discussed thus far.

A saga is a design pattern that was originally coined in a paper by Hector Garcia-Molina in 1987, http://www.amundsen.com/downloads/sagas.pdf. To quote a piece:

In Arnon Rotem-Gal-Oz's book on SOA Patterns, page 137 says:

It is expressed by many references that sagas may be built differently, depending on the need.

A saga pattern is supported by NServiceBus; for more information see http://docs.particular.net/nservicebus/sagas-in-nservicebus. A saga handles the persisting of pieces of messages as part of an ESB. During a workflow of messages, a message is sent to a saga; the saga persists the needed data and responds to the original client with messages. A saga itself is a data object with an ID, getters, and setters. As messages are passed back and forth between services, the saga is an intermediate to save valuable data. The data are message parts.

The messages of a service bus are persisted by nature and can be replayed when there is an issue with the delivery of the message with the endpoint; however, the saga keeps track of the originator and can store other data to be associated with the original message. This updated data, which is defined by the developer, may be the state of the message, the session information related to the message, or any other data needed by the application. The saga correlates messages it receives, synchronizes the activity using the corresponding ID, and deals with other features such as timeouts and lookups.

The saga evolves in the ServiceBus architecture as a pattern; it is discussed in greater detail in the next chapters.

Many common frameworks such as Microsoft MVC and EF are designed for business requirements only, with additional frameworks to assist in nonfunctional requirements; this point is stressed throughout this book. Also, we emphasize the concept of ServiceBus.

ServiceBus is a messaging workflow; it stores messages along the way. It is a workflow since it incorporates both business and nonfunctional requirements. ServiceBus does have transactional persistence to perform second-level retries if there is an error in the server or the network. The saga pattern extends that concept by giving feedback to services along the way to the originator and timing out messages. Also, it provides feedback on which operations business analysts and CSRs normally require to perform day-to-day operations. This information is used to correct issues that are of interest to the business. Remember that the saga pattern is a framework that is easily extensible, and so it is not a stress to use it for more than just retries.

NServiceBus simplifies the implementation of the concepts in the previous section; the following is a real-life scenario to illustrate them and multiple services that communicate with each other.

Recall the pizza-ordering example we discussed earlier where the Please do not refresh the page and wait for the order to complete message is displayed when a user places an order. We discussed the concern that the user may have doubts about whether the order is completed, and there is the implication that a browser refresh could cause order issues. Obviously, an ASP or JSP web page waits for some web service to go out and charge my card as it waits for the result. To avoid this behavior, a better solution is needed. One such solution is a workflow for passing messages around so that the system fires off a transaction to process the payments, allowing user interaction to continue; eventually, the system is to receive an update once the payment is processed.

There are a few possible solutions for the preceding example, and all of them have one thing in common: combining a workflow with a middle layer simplifies the solution.

One possible solution is to have several services that are responsible for different actions. We need to save data entered by a user to a database; this can be accomplished via some backend services. These services handle all the transactions needed. A service, say Service1, can pick up the data and pass it into a MSMQ for processing. This provides the separation of knowing which messages are in the state of processing. Another service, say Service2, can be responsible for the interaction with a payment engine.

Continuing with the pizza-ordering example, Service1 is responsible for getting the data entered by the customer and Service2 is responsible for processing the credit card payment. If there are errors with the payment engine, Service2 and the ServiceBus have the logic to retry again. However, Service1 remains unaware that there are errors with the payment. Service2 is atomic and does not provide notifications and feedback to the user. The payment service may place the error in an error queue, but some information, such as why the payment was not processed, will remain missing.

Using the saga pattern provides many of the features that are currently missing in the solution presented thus far. The saga is the end-to-end message workflow that can be used to save the state in an intermediate process. This can be accomplished by saving an intermediate saga data object. This persistence typically is done to a database and looked up when the same message is passed back through. Sagas can get complicated but, because very little code is required—since the ServiceBus handles most of the work—sagas can be simple to use.

As hinted previously, a saga can be created as an intermediate between the services to keep the client, in our example Service1, informed about the progress of the message.

The saga can update other endpoints of the message status and change the message if it needs updating as it moves through the workflow. The important piece of a saga is the one-to-one lookup of the data related to the message and the message itself. This allows the workflow to follow a message's progress and know where it is at a given moment along multiple services. We could define a timer to fail the message if it continually errors out, since we don't want messages to live forever.

Returning back to the pizza-ordering example, instead of waiting and not refreshing the page, we can create a page where the user can go to and check the status as the order progresses through the ServiceBus workflow. Notice that this allows many nonfunctional requirements to be addressed.

Nonfunctional requirements (such as monitoring, logging, manual retries, timeouts, checking encryption, and the message) can be addressed by monitoring the services and messages.

To recap, we can address the payment engine errors by adding logic to the saga to notify the user, operations, and the organization of specific errors. For instance, we could add logic to the saga to send an e-mail to the user saying that the order was denied due to insufficient funds. In addition, we could add another error-checking option into the workflow for network failure and other unexpected events. When such events happen, have a notification sent to operations stating that the payment engine server is not available at this time. Notice that the user does not need to be notified of these errors. Therefore, the saga becomes the focal point for checking the status of the message.