Service Oriented Java Business Integration

In a home business or in small scale ventures, we start up with systems and applications in silos which cater to the entire setup. When the business grows, we add more verticals, which are functionally separated entities within the same organization. It goes without saying that, each of these verticals or LOB will have their own systems. People ask why they have different systems. The answer is because they are doing functionally different operations in an enterprise and hence they need systems carrying out different functionality for them. For example, an HR system will manage and maintain employee related data whereas the marketing relations will use Customer Relationship Management (CRM) systems. These systems may not be interconnected and hence impede information flow across LOBs. Adding to this, each LOB will have their own budgeting and cost constraints which determine how often systems are upgraded or introduced.

Multiple LOBs will have their own systems, and hence their own data schemas, and a way of accessing the data. This leads to duplication of data, which in turn leads to multiple services providing views for the same entity in different ways.

Let's consider the example shown in the following figure. There is one LOB system, Product Information System, which will keep track of customers who have registered their interest for a particular product or service. Another LOB system, Event Registration System, will provide membership management tools to customers, for a sales discount program. Sales Information System will have to track all customers who have registered their interest for any upcoming products or services. Thus, we can see there is no unified view of the Customer entity at the enterprise-level. Each LOB will have its own systems and their own view of Customer. Some will have more attributes for the Customer, while others a few. Now the question is which system owns the Customer entity? Rather, how do we address the data stewardship issue(This is represented in the figure using the symbols "?" and "?") ?

Now the question is how long can these systems remain isolated? Rather, can we bring all these systems under a central, single point of control? The fact is, systems cannot remain isolated, and also they cannot be controlled centrally. This is because every system needs data from every (or most) other systems. Can we then integrate everything together into a single big system so that we don't have the problem of integration at all? The question seems tempting, but this is analogous to attempting to boil sea water.

Different departments or verticals within the same organization need autonomy and so do their systems. Without the required level of autonomy, there is no freedom which will hinder innovation. Constant innovation is the key to success, in any walk of life. So, departments and their systems need to be autonomous. But, since they require each other's data, they need to integrate. This leads to the necessity for a farm of autonomous systems, which are all federated to the required level to facilitate information flow.

The following figure represents systems that are autonomous, but federated to facilitate information flow:

Integrating different functional departments within an organization is not a big hurdle, since everything is under the control of the organization. But the picture changes the moment the organization grows beyond a certain limit. Twenty first century organizations are growing beyond a single location; many of them are truly global organizations. They have operations around the globe, in multiple countries, with multiple legal, technical, and operational environments. The good news is that we have technologies like Internet, Wide Area Networks, and Virtual Private Networks for connectivity. This means global organizations will have their systems federated across the globe. All these systems will evolve due to constant innovation and business changes. They have to integrate across the firewall, and not every protocol and format can be easily traversed across the firewall.

Organizations conduct business with other organizations. An online retailer's system has to partner with its wholesaler's inventory systems. These systems are not federated in any manner due to the fact that they belong to multiple organizations. There exists no federation due to the competitive nature between organizations too. But they have to collaborate; otherwise there is no business at all. So, information has to flow across organizational systems in the Internet. This is what we mean by a permeable organization boundary which allows for constant information exchange on 24 X 7 bases, with adequate Quality of Service (QOS) features. Thus the necessity is to extend the enterprise over the edge (firewall), and this activity has to happen based on pre-plans and not as an afterthought.

The following figure explains a trading partners system that is not federated but the information flow is through the Internet:

David S. Linthicum defined EAI as:

This is a simple, straightforward definition. In my entire career, I have been fortunate enough to participate in much new generation IT system development for domains such as Airline, Healthcare, and Communications. Most of the time, I've been writing either adapters between systems, or negotiating and formalizing data formats between desperate systems. I know this is not because the former system's architects haven't put a long term vision to their systems in the angle of interoperability, but because systems have to evolve and interoperate in many new ways which were not foreseen earlier. This pushes integration providers to define new software pipes across applications. When we start this activity it might be elegant and straight forward, but sooner than later we realize that our integration pipes have no central control, administration, or management provisions.

EAI has traditionally been done using point-to-point integration solutions. In point-to-point, we define an integration solution for a pair of applications. Thus we have two end points to be integrated. We can build protocol and/or format adapters, or transformers at one or either end. This is the easiest way to integrate, as long as the volume of integration is low. We normally use technology-specific APIs like FTP, IIOP, remoting or batch interfaces, to realize integration. The advantage is that between these two points, we have tight coupling, since both ends have knowledge about their peers.

The following figure is the diagrammatic representation of the point-to-point integration:

Hub-and-spoke architecture is also called the message broker and is similar to point-to-point architecture in that it provides a centralized hub (broker) to which all applications are connected. When multiple applications connect in this manner, we get the typical hub-and-spoke architecture. Another distinguishing feature of the hub-and-spoke architecture is that each application connects with the central hub through lightweight connectors. The lightweight connectors facilitates for application integration with minimum or no changes to the existing applications. Message transformation and routing takes place within the hub. This architecture is a major improvement to the point-to-point solution since it reduces the number of connections required for integration. Since applications do not connect to other applications directly, they can be removed from the integration topology by removing from the hub. This insulation reduces disruption in the integration setup.

There is a major drawback with the hub-and-spoke architecture. Due to the centralized nature of the hub, it is a single point of failure. If the hub fails, the entire integration topology fails. A solution to this problem is to cluster the hub. A cluster is a logical grouping of multiple instances running simultaneously and working together. But clustering is not the right solution to the problem of single point of failure. This is due to the fact that the very point in having a hub-and-spoke architecture is to have a single point of control. This drawback may be offset to some extent by the fact that most of the clustering solutions provide central management and monitoring facilities, which will provide some form of centralized control.

The following figure is the diagrammatic representation of the hub-and-spoke integration:

While the hub-and-spoke architecture makes use of lightweight connectors for applications to integrate together through a central hub, many a times the integrating applications need to interact in a decoupled fashion, so that they can be easily added or removed without affecting the others. An enterprise message bus provides a common communication infrastructure, which acts as a platform-neutral and language-neutral adapter between applications.

This communication infrastructure may include a message router and/or Publish-Subscribe channels. So applications interact with each other through the message bus with the help of request-response queues. If a consumer application wants to invoke a particular service on a different provider application, it places an appropriately formatted request message on the request queue for that service. It then listens for the response message on the service's reply queue. The provider application listens for requests on the service's request queue, performs the service, and then sends (if) any response to the service's reply queue.

We normally use vendor products like IBM's Websphere MQ (WMQ) and Microsoft MQ (MSMQ), which are the best class message queue solution to integrate applications in the message bus topology. As shown in the following figure, sometimes the applications have to use adapter which handle scenarios such as invoking CICS transactions. Such an adapter may provide connectivity between the applications and the message bus using proprietary bus APIs, and application APIs. The Message bus also requires a common command structure representing the different possible operations on the bus. These command sets invoke bus-level primitives which includes listening to an address, reading bytes from an address, and writing bytes to an address.

The service bus approach to integration makes use of technology stacks to provide a bus for application integration. Different applications will not communicate directly with each other for integration; instead they communicate through this middleware Service Oriented Architecture (SOA) backbone. The most distinguishing feature of the ESB architecture is the distributed nature of the integration topology. Most ESB solutions are based on Web Services Description Language (WSDL) technologies, and they use Extensible Markup Language (XML) formats for message translation and transformation.

ESB is a collection of middleware services which provides integration capabilities. These middleware services sit in the heart of the ESB architecture upon which applications place messages to be routed and transformed. Similar to the hub-and-spoke architecture, in ESB architecture too, applications connect to the ESB through abstract, intelligent connectors. Connectors are abstract in the sense that they only define the transport binding protocols and service interface, not the real implementation details. They are intelligent because they have logic built-in along with the ESB to selectively bind to services at run time. This capability enhances agility for applications by allowing late binding of services and deferred service choice.

The following figure is the diagrammatic representation of ESB integration:

The above qualities of ESB provides for a true open enterprise approach. As we have discussed above, ESB is neither a product nor a separate technology; instead, ESB is a platform-level concept, a set of tools, and a design philosophy. What this means is, if we just buy a vendor product and install it in our IT infrastructure, we cannot say that we have ESB-based integration architecture. Instead ESB-based integration solutions are to be designed and built in the "ESB way". Tools and products help us to do this.

A list of major features and functionalities supported by an ESB will help us to understand the architecture better, which are listed as follows:

Let us first see how the message bus and the service bus are alike. In fact, both of them are aimed to solve the problem of integration and provide a common communication infrastructure, which acts as a platform-neutral and language-neutral adapter between applications. So mediation is the prime functionality provided by both the architectures. Applications can integrate each other in a loosely coupled manner through this mediator, so that they will not be dependent on each other's interfaces. Last but not the least, using either the message bus or the service bus architecture, you can implement SOA-based integration solutions!

The last statement might be hard to digest—at least some of you might have thought that one is purely SOA-based while the other is not. But the fact is that both the message bus and the service bus helps enterprises to attain an SOA ecosystem, if architected properly, but neither of them by itself will automatically transfer existing non-SOA architecture into SOA-based architecture.

Now, what is the difference between a message bus and a service bus?

Before we dig into the differences let me give you a word of caution. Even though the main differences between a message bus and a service bus will be listed as follows, they may not be very obvious in the first read. Hence, I recommend the reader to go through the subsequent chapters and samples too, and get a feel of how we do things in the "ESB way", and then revisit the given differences.

More easily said, than understood! OK, let us worry less (I didn't say "let us worry not"!), understand more.

Service description and service discovery are two key elements to attain higher levels of SOA maturity. Only if something is described, can it be discovered. This is where a service registry is going to add value, there you can register a service description so that some other interested party can discover it.

Let us now come back to our message bus. We earlier saw that message bus applications interact with each other with the help of request-response queues. If you have ever worked in any messaging solution (like JMS) before, then you will appreciate the fact that queues are addressed by themselves, which will not give you any further information about the underlying service. Information on the operations available, or the messaging patterns to expect, or the schema for the types exchanged is never available at the queue-level. In other words, services are not described in a message bus.

What is available is just the messaging counterparts for the put and get primitives so that messages can be sent to and received from the message bus. So consumer or client applications should have pre-awareness of the format of the messages exchanged. This implies everything has to be known before hand—rather, static binding or compile-time binding becomes the norm.

Let us now consider the service bus. We said earlier that many ESB solutions are based on WSDL technologies, and they use XML formats for message translation and transformation. This by itself gives us lot of opportunities for service description and discovery. All the (minimum) details about the service will be available from the WSDL. Hence, message exchange patterns and message formats are readily available. Moreover, the consumer or client applications can discover a matching service from the ESB, given a set of messaging capabilities (WSDL capabilities) they are looking for. I used the term matching service because in an ideal ESB architecture the consumer is looking for capabilities which match their abstract service expectations (interface). It is the role of the ESB to route the requests to any concrete service implementation behind the bus which matches the requested interface.

The next big difference is the type of approach used in each architecture. In service bus architecture we used a standard-based approach. When services are WSDL-based, it brings a time tested and well adopted industry standard to integration. This has a big pay off when compared to traditional Message Oriented Middleware (MOM), because in the message bus architecture the adapters provide connectivity using proprietary bus APIs and application APIs. So, whereas in the pre-ESB world, we have been using CORBA IDL (Interface Definition Language), or Tuxedo FML (Field Manipulation Language), or COM/DCOM Microsoft IDL, and CICS common Area (COMMAREA) as the service definition language, we now use WSDL as the interface in standard-based ESB architectures.

Having looked at a few of the similarities and differences between service bus and message bus, it is time now to look at realities which exist in the industry today. We agreed that an ESB can do lot many things, and for that matter a message bus can too. We then talked about the differences a service bus has to offer.

How mature is the technology today to address these differences? Have we started practical implementations of service bus in a big way yet? The answer to this question is neither yes nor no. The reason is that necessity runs before standards. Rather, when you agree that you need description and discovery along with other features for a service bus-based architectures, the question is, will the existing standards like Universal Description Discovery and Integration (UDDI) alone will help to achieve this? Maybe we need a simple and standard way to specify a pair of request-reply queues along with a HTTP URL (mechanism to specify HTTP URL is already available) in the WSDL itself. This way a consumer or client application can interact in the 'MOM way' through the ESB. Maybe we also need a simple and, again, a standard way to find and invoke a concrete service at the ESB, matching an abstract service interface.

These standards are evolving and are at different stages of adoption. Similar is the case of support for these capabilities across multiple vendors' solutions. Hence, the tail piece is that it is time now to gather all our experience in message bus based architectures, and leverage it to define and demonstrate service bus-based architecture. So, how do we decide that we need an ESB-based Architecture? Read on the next section and you will come to know.

Let us go back to the point-to-point integration scenario and assume that we have four separate in-house systems, one for each of the four separate departments (HR, Accounts, R&D, and Sales). The operational problem the business faces is to interoperate these systems. In the point-to-point integration, we need to have independent channels of connection between each pair of systems. These channels are static, strongly typed, and without much intelligence for dynamic routing or transformation. The advantage is that it is rather easy to build the solution.

As shown in the next figure, if there are six systems (nodes) to be interconnected, we need at least thirty separate channels for both forward and reverse transport. If we add one more node to the IT portfolio, the number of channels to be defined goes up from thirty to forty two. This means, as time passes and as more and more systems and applications are added to the organization, the number of interconnections or channels to be defined between these systems rises exponentially, in the order of two.

We can generalize the number of channels (Nc) required for complete interconnection for n separate systems as:

Nc = n2 - n

This number is still manageable for small organizations with a small number of systems, but experience has shown that this is going to be a nightmare for mid-sized and large-sized organizations. It is customary for such organizations to have more than fifty or hundred systems. For complete interoperability in a hundred system scenario, we need to define around ten thousand separate channels! Leave alone the definition, how do we maintain such a huge network of interconnection?

Perhaps, every system in an organization needn't interoperate with every other system. Again, experience has shown that only half of them need to interoperate fully, thus bringing down the figure to five thousand. What this means is, we have to build five thousand adapters to define channels to interoperate for these systems. Still this number is not manageable.

In a perfect world, we can somehow build five thousand separate adapters to interconnect a hundred systems by fifty percent, and then hope to relax. But the real scenario is far from perfect. The imperfection arises out of constant innovation and evolution, and this forces change even to system interfaces. An int data type in C++ is of two bytes whereas the same int in Java is of four bytes. We have already taken care of this data impedance by building adapters. But, what if on one fine morning, the C++ application starts spitting float instead of int? The existing adapter is going to fail, and we need to rework on the adapter in this channel.

The good news is, within an organization this can be controlled and managed since we have access to both the ends of the adapter. The scenario worsens when the interface changes in a B2B scenario where systems between two separate organizations interact. As we know already, this is going to happen. So, we have to build agile interfaces.

Adding new systems is an after effect of business expansion, but there are multiple risks associated with it. First, we need to integrate new systems with the existing systems. Secondly, many a time, new systems introduce data redundancy. This is because the same data might get entered into the network through multiple interfaces. Similar is the problem of the same data duplicated at different parts of the organization. This contradicts the requirements of Straight-Through Processing (STP). STP aims to enter transactional data only once into the system. This data can flow within or outside the organization. To achieve STP, there should be mechanism for seamless data flow, also in a flexible manner.

Data redundancy can be managed if and only if there is a mechanism to consolidate data in the form of Common Information Model (CIM). For example, the NGOSS's Shared Information/Data (SID) model from Telecom Management Forum is the industry's only true standard for development and deployment of easy to integrate, flexible, easy to manage OSS/BSS components. The SID provides an information or data reference model and a common information or data vocabulary from a business and systems perspective. Not every domain or organization has access to a readymade information model similar to this.

In such a scenario, ESB can play a vital role by providing the edge of the system view by wrapping and exposing available network resources. Thus, ESB provides hooks for a "leave-and-layer" architecture which means that instead of altering existing systems to provide a standards-based services interface they are wrapped with a layer that provides the services interface.

Code and component reuse is a familiar concept amongst designers and developers. We have Gang of Four patterns to prescribe design solutions for recurring problems. But with the invention of SOA, we have what is called the Service Oriented Integration (SOI) which makes use of Enterprise Integration Patterns (EIP). SOI speaks about the best ways of integrating services so that services are not only interoperable but also reusable in the form of aggregating in multiple ways and scenarios. This means, services can be mixed and matched to adapt to multiple protocols and consumer requirements.

In code and component reuse, we try to reduce copy and paste reuse and encourage inheritance, composition, and instance pooling. A similar analogy exists in SOI where services are hosted and pooled for multiple clients through multiple transport channels. ESB does exactly this job in the best way integration world has ever seen. For example, if a financial organization provides a credit history check service, an ESB can facilitate reuse of this service by multiple business processes like a Personal Loan approval process or a Home Mortgage approval process.

Cross cutting concerns like transactions or security or even Service Level Agreement (SLA) management and monitoring are just a few of a set of features apart from business functionality that any enterprise class service ecosystem has to provide. The ESB architectures provides enough hooks to attach similar services onto the bus separate from normal business functionality. This ensures that these cross cutting functionality can be applied to all services and also enable us to manage, monitor, and control them centrally.

It is time now to understand "Service" in ESB. Why don't we call it Enterprise Component Bus?

Services are exposed functionalities which are clearly defined, self contained, and which will not depend on the state and context of peer services. As shown in the following figure, the service has a service description (similar to WSDL) and it is this description which is exposed. It is never necessary to expose the implementation details, and in fact, we shouldn't expose them. It is this hidden nature of the implementation which helps us to replace services with similar services. Thus, service providers expose the service description and service consumers find them.

Once the service description is found service consumers can bind to it. Moreover, it is during the time of actual binding, we need more information like transport protocols. As we know, we can handle with transport details with the help of protocol adapters. Intelligent adapters are required if we need to route messages based on content or context. These adapters are of a different kind like splitters and content based routers. ESB is all about a run-time platform for these kinds of adapters and routers, which are specifically designed for hosting and exposing services.

Almost all OOP languages support interface-driven development and hence this is not a new concept. Separating interface from implementation abstracts the Service Consumer from the implementation details from a Service Provider perspective. But, what about abstracting out this interface itself? If the Service Consumer interacts directly with the Service Provider, both these parties are tightly coupled and it may be difficult to change the interface. Instead, a service bus can effectively attach itself to a service registry. This means, a consumer can even be unaware of the service interface.

Now, just in time, the consumer can selectively choose an appropriate interface from a registry, then bind to a suitable implementation through an ESB, and invoke the service at run time. Such kinds of flexibility come at the cost of design time or compile time type checking, which can be error prone during service invocation using conventional tools. An ESB with its set of tools and design primitives helps developers to effectively address this.

The key value of aggregation, here, is not so much in being able to provide core services, which developers have to build anyway. It is to facilitate an arbitrary composition of these services in a declarative fashion, so as to define and publish more and more composite services. Business Process Management (BPM) tools can be integrated over ESB to leverage service aggregation and service collaboration. This facilitate reuse of basic or core (or fine grained) services at the business process-level. So, granularity of services is important which will also decide the level of reusability.

Coarse Grained or Composite Services consume Fine Grained Services. Applications which consume Coarse Grained Services are not exposed to the Fine Grained Services that they use. Composite Services can be assembled from Coarse Grained as well as Fine Grained Services.

To make the concept clear, let us take the example of provisioning a new VOIP Service for a new customer. This is a Composite Service which in turn calls multiple Coarse Grained Services such as validateOrder, createOrVerifyCustomer, and checkProductAvailability. Now, createOrVerifyCustomer, the Coarse Grained Service in turn call multiple Fine Grained Services like validateCustomer, createCustomer, createBillingAddress, and createMailingAddress.

As is shown in the above figure, a Composite Services is an aggregate of other Composite Services and/or Coarse Grained Services. Similarly, Coarse Grained Services are again aggregates of other Fine Grained Services. Whereas, Fine Grained Services implement minimum functionality and are not aggregates of other services. ESB here acts as a shared messaging layer for connecting applications and other services throughout the enterprise, thus providing a workbench for service aggregation and composition.

Services can be designed but if left alone they will sit idle. The idea behind true SOA is not just service definition or implementation, but also service enablement. By service enablement, we mean enabling, configuring, tuning, and attaching QOS attributes to services, so that they are accessible, as per SLA irrespective of formats or transport protocols.

Using today's techniques like Aspect Oriented Programming (AOP), Annotations and byte code instrumentation, service enablement can be done without any dependency on the services framework. An ESB does exactly this. It is to be noted that a JBI-based ESB will do this API-less. It is not completely API-less, in the sense that there is always a dependency on an XML configuration or annotation. The differentiating factor is that this dependency is externalized so that they are not cemented to the service implementation.

Service consolidation is an important step to eliminate redundant information and data. As a part of service consolidation, we may have to first consolidate IT infrastructure and data assets in the network. There should be clear ownership for these two entities, so that we know who or which applications are changing which sectors of the data assets. Once this is done, the next step is to do application portfolio analysis and do direct mapping between services and data operations (business-level CRUD).

A point to be noted here is that I am not talking about exposing simple Create, Read, Update, and Delete operations to be exposed as services, since they are not (coarse grained) services at all. They are just functions to alter the constants of data model and if we expose them as services, we are trying to expose our internal data modeling, which is going to impact internal agility. Instead, what we have to expose is our coarse grained business operations (processes) which carry out useful business functions on behalf of client applications. When we do that, we can map which services need to be invoked to finish an online retailing experience, and which other services are responsible for realizing the retail booking in the back office systems.

Thus services are named, labeled, and all the more indexed. And you look at the index for a particular service similar to the way you look at a book index, to find a particular subject. Thus, we effectively pool services and consolidate unused capacity formerly spread over many connected systems and servers.

Message bus facilitates service sharing as compared to service reuse. In fact service sharing is one of the key aspects of SOA.

In service reuse, the same service is hosted in different deployment environments due to various reasons. Even though the component or the code behind the service is reused here, there is significant overhead in terms of deployment, management, and maintenance. The Service reuse scenario is as shown in the following figure:

Service sharing is, in contrast, the true sharing of a service itself. This means the service hosting environment is configured for different scenarios and consumers access the same service instance through multiple channels or formats. SOA web services can be leveraged to achieve this, and this is further enabled using a message bus as shown in the following figure:

Microsoft SQL Server has the ability to query remote database objects as if they were local to the server. Linked servers are useful in deployments, where the data can be split by functional area into databases with very little coupling as a means of scale out strategy. This means that a scaled out database can look like a single large database to an application. Similar to this, ESB can be seen as a means to link services. Binding to remote end points is synonymous to linking to remote databases—an application code can interact with the linked services as if they were local to the invoking code.

By service virtualization, I mean dividing the organization's information assets into "Virtual Services" without regard to the transport or binding protocol of the actual implementation components. Corporate network assets (information) can reside in multiple places like in a centralized mainframe or distributed across multiple systems implemented using DCOM, RMI, IIOP, remoting, or SOAP protocols.

Service wrapping and service aggregation are two methods by which we can virtualize them to be exposed in a consistent way to external or internal clients. Thus, virtualization is closely related to "black box" reuse of services by providing a different interface at the same abstraction-level. The ESB abstracts behind the scene addressing, translations, and aggregation.

Well-defined interfaces facilitate development of interacting software systems not only in different LOBs, but also at different times, thus providing a suitable abstraction for any consumers of the service. But abstraction has a negative side too—software systems, components, or services designed for one interface will not work with those designed for another interface. This, in fact, is an interoperability issue which can be a concern, especially in a world of networked computers where the trend is to move data and information freely.

Service virtualization is a way to circumvent this issue by mapping the visible interface of a service onto the interface and resource of an underlying, possibly different, real service.

This figure depicts the concept of Service Virtualization, where we make use of core ESB features like Aggregation, Routing, and Translation to republish the existing service assets. These are explained in detail in subsequent chapters.

We, integration people, need to learn lessons from the way network architects organize their network channels. When using fiber, network architects have three network topologies. For the fibre channel option, we have three network topologies via any from the following list:

In the fabric topology, there are one or more hardware switches in addition to ports on each device to network. Comparatively, in ESB we have intelligent adapters. Multiple clients can hit the ESB fabric, and intelligent adapters at the ESB edge are responsible for dynamic binding of client requests to appropriate concrete service implementations. Thus, ESB services fabric functions as the virtualization layer. In other words, the ESB provides infrastructure capabilities and application plug-in points for an SOA fabric for the whole enterprise.