The Trivadis Integration Architecture Blueprint applies a clear and simple naming to each of the individual layers. However, in the context of integration, a wide range of different definitions and terms are used, which we will explain in this chapter.

A2A, B2B, and B2C

Nowadays, business information systems in the majority of organizations consist of an application and system landscape, which has grown gradually over time. The increasing use of standard software (packaged applications) means that information silos will continue to exist. IT, however, should provide end-to-end support for business processes. This support cannot, and must not, stop at the boundaries of new or existing applications. For this reason, integration mechanisms are needed, which bring together individual island solutions to form a functioning whole. This happens not only at the level of an individual enterprise or organization, but also across different enterprises, and between enterprises and their customers. At an organizational level, a distinction is made between A2A, B2B, and B2C integration (Pape 2006). This distinction is shown in the image below. Each type of integration places specific requirements on the methods, technologies, products, and tools used to carry out the integration tasks. For example, the security requirements of B2B and B2C integrations are different from those of an A2A integration.

Modern concepts such as the Extended Enterprise integration across organizational boundaries, (Konsynski 1993) and the Virtual Enterprise (Hardwick, Bolton 1997) can be described using a combination of the different integration terms.

Integration projects are generally broken down into information portals, shared data integration, and shared function integration. Portals integrate applications at a user interface level. Shared data integration involves implementing integration architectures at a data level, and shared function integration at a function level.

The challenge represented by semantic integration is based on the following problem:

This means that the data structure and data information (its meaning) are often not the same thing and, therefore, have to be interpreted (Inmon, Nesavich 2008).

The following example will help to make this clearer:

A date, such as "7 August 1973," forms part of the data. It is not clear whether this information is a text string or in a date format. It may even be in another format and will have to be calculated on the basis of reference data before runtime. This information is of no relevance to the user.

However, it might be important to know what this date refers to, in other words, its semantic meaning in its reference context. Is it a customer's birthday, or the date on which a record was created? This example can even be more complex.

Another example that can be interpreted differently in different contexts is the term Caesar, for instance. Depending on the context, it could be the name of a pet or the name of pet food, a well-known salad, a gambling casino, or the name of a Roman emperor.

It is clear that data without a frame of reference is lacking any semantic information, causing the data to be ambiguous and possibly useless. Ontologically-oriented interfaces, as well as adaptive interfaces, can help to create such semantic reference and will become increasingly important in the field of autonomous B2B or B2C marketplaces in the future.

One semantic integration approach is, for example, the use of model-based semantic repositories (Casanave 2007). These repositories store and maintain implementation and integration designs for applications and processes (Yuan et al. 2006). They access existing vocabularies and reference models, which enable a standardized modeling process to be used. Vocabularies create a semantic coupling between data and specific business processes, and it is through these vocabularies that the services and applications involved are supplied with semantic information in the surrounding technical context. The primary objective of future architectures must be to maintain the glossary and the vocabularies, in order to create a common language and, therefore, a common understanding of all the systems and partners involved. Semantic gaps must be avoided or bridged wherever possible, for example transforming input and output data by using canonical models and standardized formats for business documents. These models and formats can be predefined for different industries as reference models [EDI (FIPS 1993), RosettaNet (Damodaran 2004), and so on]. Transformation rules can be generated and stored on the basis of reference models, in the form of data cards and transformation cards. In the future, there will be a greater focus on the declarative description (what?) and less emphasis on describing the concrete software logic (how?) when defining integration architectures. In other words, the work involved in integration projects will move away from implementation, and towards a conceptual description in the form of a generative approach, where the necessary runtime logic is generated automatically.

The term Enterprise Application Integration (EAI) has become popular with the increased importance of integration, and with more extensive integration projects. EAI is not a product or a specific integration framework, but can be defined as a combination of processes, software, standards, and hardware that allow for the end-to-end integration of several enterprise systems, and enable them to appear as a single system (Lam, Shankararaman 2007).

From a business perspective, EAI can be seen as the competitive advantage that a company acquires when all its applications are integrated into one consistent information system. From a technical perspective, EAI is a process in which heterogeneous applications, functions, and data are integrated, in order to allow the shared use of data and the integration of business processes across all applications. The aim is to achieve this level of integration without major changes to the existing applications and databases, by using efficient methods that are cost and time effective.

In EAI, the focus is primarily on the technical integration of an application and system landscape. Middleware products are used as the integration tools, but, wherever possible, the design and implementation of the applications are left unchanged. Adapters enable information and data to be moved across the technologically heterogeneous structures and boundaries. The service concept is lacking, as well as the reduction of complexity and avoidance of redundancy offered by open standards. The service concept and the standardization only came later with the emergence of service-oriented architectures (SOA), which highlighted the importance of focusing on the functional levels within a company, and its business processes.

Nowadays, software products which support EAI are often capable of providing the technical basis for infrastructure components within an SOA. As they also support the relevant interfaces of an SOA, they can be used as the controlling instance for the orchestration, and help to bring heterogeneous subsystems together to form a whole. Depending on its strategic definition, EAI can be seen as a preliminary stage of SOA, or as a concept that competes with SOA.

SOA is now moving the concept of integration into a new dimension. Alongside the classic "horizontal" integration, involving the integration of applications and systems in the context of an EAI, which is also of importance in an SOA, SOA also focuses more closely on a "vertical" integration of the representation of business processes at an IT level (Fröschle, Reinheimer 2007).

SOAs are already a characteristic feature of the application landscape. It is advisable when implementing new solutions to ensure that they are SOA-compliant, even if there are no immediate plans to introduce an integration architecture, or an orchestration layer. This allows the transition to an SOA to be made in small, controllable steps, in parallel with the existing architecture and on the basis of the existing integration infrastructure.

Message queues were introduced in the 1970s as a mechanism for synchronizing processes (Brinch Hansen 1970). Message queues allow for persistent messages and, therefore, for asynchronous communication and the guaranteed delivery of messages. Messaging decouples the producer and the consumer with the only common denominator being the queue.

The most important properties of messaging, quality attributes of messaging, are shown in the following table:

Publish/subscribe represents an evolution of messaging (Quema et al. 2002). A subscriber indicates, in a suitable form, its interest in a specific message or message type. The persistent queue guarantees secure delivery. The publisher simply puts its message in the message queue, and the queue distributes the message itself. This allows for many-to-many messaging:

The most important properties of publish/subscribe, quality attributes of publish/subscribe, are listed in the following table:

A message broker is a central component, which is responsible for the secure delivery of messages (Linthicum 1999). The broker has logical ports for receiving and sending messages. It transports messages between the sender and the subscriber, and transforms them where necessary.

The most important tasks of a message broker, as shown in the preceding diagram, are implementing a hub-and-spoke architecture, the routing, and the transformation of messages.

The most important properties of a message broker, quality attributes of a message broker, are listed in the following table:

A messaging infrastructure provides mechanisms for sending, routing, and converting data, between different applications running on different operating systems with different technologies, as shown in the following diagram (Eugster et al. 2003):

A messaging infrastructure involves the following parties/components:

An Enterprise Service Bus is an infrastructure that can be used to implement an EAI. The primary role of the ESB is to decouple client applications from services, as shown in the following diagram (Chappell 2004):

The encapsulation of services by the ESB means that the client application does not need to know anything about the location of the services, or the different communication protocols used to call them. The ESB enables the shared, enterprise-wide, and even intra-enterprise use of services and separate business processes from the relevant service implementations (Lee et al. 2003).

The core functions of an ESB

The major SOA vendors now offer specific Enterprise Service Bus products, which provide a series of core functions in one or another form, shown in the following diagram:

The structure of an ESB

The following diagram shows the basic structure of an ESB in a vendor-neutral way:

The naming for the single components used by the different vendors of SOA products will vary from those shown in the above diagram, but the products provide the following functions as a minimum (Craggs 2003):

• Routing and messaging as base services

• A communication bus, which enables a wide variety of systems to be integrated using predefined adapters

• Transformation and mapping services for a range of different conversions and transformations

• Mechanisms for executing processes and rules

• Monitoring functions for a selection of components

• Development tools for modeling processes, mapping rules, and message transfers

• A series of standardized interfaces, such as JMS (Java Messaging Specification (Hapner et al. 2002)), JCA (Java Connector Architecture (JCASpec 2003)), and SOAP/HTTP

In most cases the technological realization of EAI systems is done through what is commonly termed middleware. Middleware is also described as a communication infrastructure. It allows communication between software components, regardless of the programming language in which the components were created, the protocols used for communication, and the platform on which the components are running (Thomson 1997). A distinction is made between the different types of middleware according to the methods of communication used, and the base technology.

Routing schemes

Information can be routed in different ways within a network. Depending on the type of routing used, routing schemes can be broken down into the following four categories:

• Unicast (1:1 relationship)

• Broadcast (all destinations)

• Multicast (1:N)

• Anycast (1:N, most accessible)

Unicast

The unicast routing scheme sends data packages to a single destination. There is a 1:1 relationship between the network address and the network end point:

Broadcast

The broadcast routing scheme sends data packets in parallel to all the possible destinations in the network. If there is no support for this process, the data packets can be sent serially to all possible destinations. This produces the same results, but the performance is reduced. There is a 1:N relationship between the network address and the network end point.

Multicast

The multicast routing scheme sends data packets to a specific selection of destinations. The destination set is a subset of all the possible destinations. There is a 1:N relationship between the network address and the network end point:

Anycast

The anycast routing scheme distributes information to the destination computer which is nearest, or most accessible. There is a 1:N relationship between the network address and the network end point, but only one end point is addressed at any given time for the purpose of routing the information.

A point-to-point architecture is a collection of independent systems which are connected through a network. All the systems have equal rights, and can both use and provide services (Lublinsky 2002). This architecture can be found in many organizations, where application islands that have grown through time have been connected directly to each other.

As shown in the above diagram, in this architecture, there is no central database—each system has its own data storage.

New systems are connected directly with the existing ones, which over time leads to a highly complex set of interfaces. A point-to-point architecture with n applications can in theory have n*(n-1)/2 interfaces.

It is easy to imagine how complex, error-prone, and difficult it can be to maintain such an architecture as more and more applications are added. Expanding the system is costly and, as the number of interfaces grows, operation becomes increasingly time consuming and expensive. A SWOT analysis is shown in the following table:

Hub-and-spoke architecture represents a further stage in the evolution of application and system integration, as shown in the following diagram (Gilfix 2003):

Its objective is to minimize the growing interface complexity by using a central integration platform to exchange messages between the systems. The central integration platform can transform messages, route them from one application to the next, and change the content of the messages. This architecture is often used for complex data distribution mechanisms. A SWOT analysis is shown in the following table:

In a pipeline architecture, independent systems along the value-added chain are integrated using a message bus, as in the following figure. The implementation of this architecture corresponds to that of the hub-and-spoke architecture, as the corresponding middleware products are normally installed and operated on central servers. The bus capability results in the distribution of the interfaces to the central bus throughout the communication network, which generally also gives applications local access to a bus interface (Ambriola, Tortora 1993).

Similarly to the hub-and-spoke architecture, this architecture also keeps interface problems to a minimum. The use of appropriate middleware components allows the communication between the systems to be standardized. The bus system is responsible for message distribution. The transformation and routing rules are stored in a central repository. Depending on the middleware product in use, business functions and rules can also be represented. A SWOT analysis is shown in the following table:

This form of architecture is ideal for:

The core of a service-oriented architecture, and the main distinction between this form of architecture and those described earlier, is the fact that business processes and applications are no longer coded as complex program structures, but are orchestrated as independent, distributed service calls.

An ESB is used as the central integration component for service calls. It has similar properties to those of the integration platform in hub-and-spoke architecture, or of the bus in pipeline architecture. A SWOT analysis is shown in the following table

Direct connection represents the simplest type of interaction between two applications and is based on a 1: N topology, in other words, an individual point-to-point connection. It allows a pair of applications within an organization to communicate directly. Interactions between the source and the target applications can be as complex as necessary. Additional connection rules are defined for more complex point-to-point connections. Examples of connection rules include data mapping rules, security rules, and availability rules.

The direct connection pattern can be broken down into the following logical components:

The advantages and disadvantages of the direct connection pattern are shown in the following table:

The broker pattern is based on the direct connection pattern, and extends it to a 1: N topology. It allows an individual request from a source application to be routed to several target applications, which reduces the number of 1:1 connections required. The connection rules take the form of broker rules. This allows the distribution rules to be kept separate from the application logic (Separation of Concerns principle or SoC). The broker is also responsible for the composition and decomposition of interactions. The broker pattern uses the direct connection pattern for the connection between the applications. The broker pattern forms the base for the publish/subscribe message flow:

The broker pattern can be broken down into the following logical components:

The advantages and disadvantages of the broker pattern are shown in the following table:

The router pattern is a variant of the broker pattern with several potential target applications, in which the message is always routed to only one target application. The router decides which target application will receive the interaction. While the broker pattern supports 1:N connections, the router pattern only allows 1:1 connections, as the router rules determine the target application in each case.

The router pattern as shown in the diagram can be broken down into the following logical components:

The advantages and disadvantages of the router pattern are shown in the following table:

The federation pattern is a simple data integration pattern that provides access to different data sources, and gives the calling application the impression that these sources are a single, logical data source. This is achieved as follows:

This is illustrated in the following diagram:

The federation pattern as shown in this diagram can be broken down into the following logical building blocks:

The federation pattern supports structured and unstructured data, together with read-only and read/write accesses to the underlying data sources. Read/write accesses should be limited, wherever possible, to a single data source, as otherwise a two-phase commit is needed, which can be difficult in distributed databases.

The population pattern has a very simple model. It gathers data from one or more data sources, processes the data in an appropriate way, and applies it to a target database. In its simplest form, the population pattern is based on the read dataset-process data-write dataset model. This corresponds to the classic ETL (Extract, Transform, and Load process.

This is illustrated in the following diagram:

The population pattern can be broken down into the following logical components:

The synchronization pattern (also known as the replication pattern) enables bidirectional update flows of data in a multi-copy database environment. The "two-way" synchronization aspect of this pattern is what distinguishes it from the "one-way" capabilities provided by the population pattern.

This is illustrated in the following diagram:

The synchronization pattern shown in this diagram can be broken down into the following logical components:

Multi-step synchronization

There is one variant of the synchronization pattern: the multi-step variant. The multi-step variant of the two-way synchronization pattern makes use of one instance of the population pattern, with its gather, process, and apply functions, for each of the two synchronization directions. An additional "reconcile" function is placed between the two data flows, and guarantees that there are no conflicts in the updates. If the opportunities for conflicts are minimal, this pattern can be constructed from existing population components. However, a specialized solution should be used for more complex situations.

The following diagram illustrates the "reuse" of the population pattern, once for each direction with the additional "reconcile" component in the middle.

The process integration pattern extends the 1: N topology of the broker pattern seen in EAI. It simplifies the serial execution of business services, which are provided by the target applications, and therefore enables the orchestration of serial business processes, based on the interaction of the source application. The serial sequence is defined using process rules, which allows for decoupling from the process logic (flow logic and the domain logic) of the individual application. The rules define not only the control and data flow, but also the permitted call rules for each target application. Interim results (process data) are stored in individual results databases.

The process integration pattern can be broken down into three building blocks:

The advantages and disadvantages of the process integration pattern are shown in the following table:

The workflow integration pattern represents an extension of the process integration pattern, as illustrated in the following diagram:

It extends the capability of simple serial process orchestration to include support for user interaction during the execution of individual process steps. As a result, it supports a classic workflow.

According to a study by Gartner (Gartner 2006), the success of companies such as Dell and Google is due to the fact that these organizations are able to identify market factors or market events in the global marketplace at an early stage, and follow them up consistently and quickly. Both examples are very close to the picture drawn by Gartner of an ideal organization: the real-time enterprise (RTE) . An RTE is characterized by its highly-automated business processes and the shortest possible process runtimes (Nussdorfer, Martin 2003).

While SOA concepts within IT structures form the basis for the automation of business processes, the second step, which relates to the ideal image of the RTE, involves processing more fine-grained information about changes in the state of these business processes. The complexity of these state changes is increasing noticeably, in the same way that the number of reaction interfaces in the business processes is. This is where the significance of EDA lies, because the observable changes in the state of the business processes can be modeled as events. Classic integration architectures, such as OLTP (Online Transaction Processing) or OLAP (Online Analytical Processing) are no longer able to meet the requirements for rapid and consistent action on the basis of event analyses (Zeidler 2007).

The above diagram illustrates the symbiosis between EDA and SOA, which is often referred to as SOA 2.0 (Carter 2007) or Next Generation SOA (Luckham 2002). An SOA or an SOA domain provides the technical services, independently of consumers. These services can be combined or orchestrated, and they form the building blocks for the business processes. If a service of this kind triggers an event, for example because of a change in its state, an orthogonal EDA extension can activate a new SOA domain. As a result, a service becomes an event-producing building block in an EDA. In contrast to the typical producer/consumer patterns of an SOA, the EDA largely uses a publish/subscribe mechanism. An event processor processes the

events as they occur, and publishes the processed results via an event channel, which triggers the services of other SOA domains. Various types of event processor are used depending on the type of event processing required. These include Complex Event Processors (CEP), for example, which are described later in this chapter.

The SOA domains (to be integrated) should ideally be defined in such a way that they represent reusable services which can be used several times in business process chains of any length. The principle of loose coupling for the formation of such flexible business process chains is of decisive importance in the EDA, in the same way as it is in the SOA.

The second aspect of integration that we want to highlight is of a more technical nature. It concerns the possibilities for event processing within an EDA concept, as shown in the following figure:

Event-processing technologies have been in day-to-day use in many industries for several years. Examples include algorithmic trading in stock markets and Radio Frequency Identification (RFID) in road charging systems.

There are three fundamental types of event processing:

Complex Event Processing (CEP)

The third form of event processing is Complex Event Processing, which is part of ESP. In CEP there is a strong focus on identifying patterns in a large number of events and their (message) contents, which may be distributed across different data streams.

The CEP funnel model is illustrated in the following figure:

The CEP funnel model illustrates the process of compressing (large) volumes of events to produce compressed information. The source of the events includes business events (Viehmann 2008). One of the classic uses of CEP is in tracing credit card fraud. In the case of two transactions using the same credit card, which were made within a short period of time, in locations a long distance apart, this type of geographical and chronological pattern indicating the possibility of fraud, can easily be applied to the model. Systems of this kind, which correlate thousands of events in order to filter out the few cases of fraud or misbehavior, are more and more frequently in use.

Grid computing is the term used to describe all the methods that combine the computing power of a number of computers in a network, in a way that enables the (parallel) solution of compute-intensive problems (distributed computing), in addition to the simple exchange of data. Every computer in the grid is equal to all the others. Grids can exceed the capacity and the computing power of today's super computers at considerably lower cost, and are also highly scalable. The computing power of the grid can be increased by adding computers to the grid network, or combining grids to create meta grids.

The following diagram illustrates the basic model of grid computing, with the network of computers forming the grid in the middle:

The main tasks of grids are:

Grids have the following features which allow more sophisticated Service Level Agreements (SLA) to be set up:

Grids can be broken down into data grids, in-memory data grids, domain entity grids, and domain object grids on the basis of their primary functionality.

Grid technology can be used in a variety of different ways in architectures:

As a result of the need for complex processing of large and very large volumes of data (for example, in the field of XML, importing large files with format transformations, and so on.), new distributed storage architectures with parallel application access functions have been developed in recent years.

A range of different cross-platform products and solutions is available, also known as "extreme transaction processing" or XTP. The term was coined by the Gartner Group, and describes a style of architecture which aims to allow for secure, highly scalable and high-performance transactions across distributed environments on commodity hardware and software.

Solutions of this kind are likely to play an increasingly important role in service-oriented and event-driven architectures in the future. Interoperability is a driving force behind XTP.

Distributed caching mechanisms and grid technologies with simple access APIs form the basis for easy, successful implementation (in contrast to the complex products widely used in scientific environments in the past). Although distributed cache products already play a major role in "high-end transaction processing" (an expression coined by Forrester Research), their position in the emerging Information-as-a-Service (IaaS) market is expected to become more prominent.

New strategies for business priority have been introduced by financial service providers in recent years. Banks are attempting to go beyond the limits of their existing hardware resources and develop increasingly high-performance applications, without having to invest in an exponential increase of their hardware and energy costs.

The growth of XTP in areas such as fraud detection, risk computation, and stock trade resolution is pushing existing systems to their performance limits. New systems which should implement this challenging functionality require new architecture paradigms.

It is clear that SOA, coupled with EDA and XTP, represents the future for financial service infrastructures as a means of achieving the goal of running complex computations with very large volumes of data, under real-time conditions. XTP belongs to a special class of applications (extreme transaction processing platforms) that need to process, aggregate, and correlate large volumes of data while providing high performance and high throughput. Typically, these processes produce large numbers of individual events that must be processed in the form of highly volatile data. XTP-style applications ensure that transactions and computations take place in the application's memory, and do not rely on complex remote accesses to backend services, in order to avoid communication latency (low latency computation). This allows for extremely fast response rates while still maintaining the transactional integrity of the data.

The SOA grid (next generation, grid-enabled SOA) is a conceptual variant of the XTPP (Extreme Transaction Processing Platform). It provides state-aware, continuous availability for service infrastructures, application data, and process logic. It is based on an architecture that combines horizontally scalable, database-independent, middle-tier data caching with intelligent parallelization, and brings together process logic and cache data for low latency (data and process affinity). This enables the implementation of newer, simpler, and more efficient models for highly scalable, service-oriented applications that can take full advantage of the possibilities of event-driven architectures.

XTP and CEP are comparable, in that they both consume and correlate large amounts of event data to produce meaningful results.

Often, however, the amount of event data that needs to be captured and processed far exceeds the capacity of conventional storage mechanisms ("there just isn't a disk that can spin fast enough"). In these cases, the data can be stored in a grid. CEP engines can be distributed across this data and can access it in parallel. Analyses can be carried out, and business event patterns can be identified and analyzed in real-time. These patterns can then be processed further and evaluated using Business Activity Monitoring (BAM).

Solid State Disk (SSD) technology is developing at high speed. Data capacities are increasing rapidly and compared with conventional drives, the I/O rates are phenomenal. Until now, the price/performance ratio per gigabyte of storage has been the major obstacle to widespread use. It is currently a factor of 12 of the cost of a normal server disk, per gigabyte of storage. The major benefit for data centers is the very low energy consumption, which is significantly less than that of conventional disks.

Because of their low energy requirements, high performance, low latency, and the expectation of falling costs, SSDs are an attractive solution in blades or dense racks. One interesting question concerns the influence which SSDs may have on data grid technology.

Disk-based XTP systems can benefit from the introduction of an SSD drive However, SSDs currently have a much lower storage capacity (128 GB versus 1 TB) than conventional disks. Nevertheless, this is more than the capacity of standard main memory, and SSDs are also less costly per gigabyte than memory. The capacity of SSDs is lower than that of conventional disks by a factor of 10, and higher than the capacity of memory by a factor of 8.

SSDs bridge the gap between memory-based and disk-based XTP architectures. SSD-based architectures are slightly slower than memory-based systems, but significantly faster than the fastest disk-based systems. The obvious solution is, therefore, to provide a hierarchical storage architecture in XTP systems, where the most volatile data is stored in memory, data accessed less often is stored on SSDs, and conventional disk-based storage is used for long-term persistent data. It also seems reasonable to store memory overflows from memory-based caching on SSDs.