The main objective of information technology is to provide support for business operations. IT has successfully automated various functions such as payroll, general ledger, and invoices through the introduction of application systems. Although this has been very valuable for companies, there has also been an understanding that automation of such activities is not all that IT can provide. Therefore, information systems have tried to cover more and more functions. As a result, ERP (Enterprise Resource Planning), CRM (Customer Relationship Management), SCM (Supply Chain Management), and similar systems have emerged.
Through the introduction of these systems, companies have started to realize that the ultimate objective would be to automate business processes – in other words, to develop applications that would provide support at each and every step of a business process, from its beginning until its completion.
Although fulfilling this objective might sound simple, it is not! There are at least two major challenges:
Each company has its unique business processes (and application systems should be designed around the business processes, not vice versa).
Business processes are not constant; they change with time. Every change in the business process has to be reflected in the enterprise systems. This requires the enterprise systems to be highly flexible, so that they can be modified quickly and efficiently.
Fulfilling both requirements requires a highly flexible IT architecture, which would allow changes to be made to the software quickly and efficiently. Business processes are also required to relate more closely to the application systems, which has not been the case so far. Usually, business processes have been modeled "on paper", resulting in nice pictures. However, there has been a semantic gap between those pictures of business processes and the actual application systems, and changes in the processes have not produced clear dependencies to the changes required in application systems.
Service Oriented Architecture (SOA) has emerged as a solution to these problems. In this book, we will show you how we can use SOA along with Business Process Management (BPM) to solve these and other related challenges. We will look at its complete life cycle, starting with the business process modeling and ending with the application that implements such processes. We will see that SOA introduces 'new approach' technologies and languages such as BPMN (Business Process Modeling Notation), BPEL (Business Process Execution Language), ESB (Enterprise Service Bus), services, rule engines, registries/repositories, and others, to fulfill the objectives.
In this chapter, we will look at business processes and their relevance to IT, application systems, and SOA in particular. We will:
Explain why we have to care about business processes
Discuss how business processes emerge
Think about the relationship between business processes and IT
Discuss the importance of IT flexibility
Explain why we need SOA
Introduce the SOA approach to business processes and explain the major benefits of the SOA approach
Explain the role of a SOA competency centre
Discuss SOA's inception
See an overview of the SOA forces and their significance for IT departments
Explain the changes in the development approach, required by SOA
See an overview of the technical aspects of SOA
Briefly introduce BPMN, BPEL, services, interfaces, messages, synchronicity, loose coupling, and quality of service
Discuss the role of the ESB, Registry and Repository, Rules Engines, Business Activity Monitoring, and User Interactions
Explain how SOA, BPMN, and BPEL fit together
Business processes are essential for every company—large or small. Companies rely on business processes. Let us look at what business processes are.
Note
A business process is a set of coordinated activities that are performed either by humans or by tools with an objective to realize a certain business result.
A business process consists of a set of coordinated activities that accomplish a particular business goal. The order of these activities and the efficiency of those who perform the activities determine the overall performance of a business process. It is very much in the interest of every company to have business processes that are efficient and include only necessary activities, because this will allow them to work faster.
As business processes define the order of work, they are related directly to the efficiency and effectiveness of a company. The better the business processes are defined, the more efficiently a company can operate. In today's competitive market, the efficiency of a company is a key criterion for success, because in addition to innovation, operating efficiency is key to improving the company's competitive advantage in the market.
Knowing and understanding the details of business processes is important, because this gives us the opportunity to identify the bottlenecks and optimize business processes. Optimizing business processes make them more efficient, our customers happier, will reduce the workload on the employees, and will reduce the utilization of resources.
Managing business processes is, therefore, very important. Each company should know how its business processes are defined, who is involved in the various related activities, and how long it takes to execute a certain process.
We have already mentioned that the objective of IT is to support and automate business activities using application systems. On the one hand, this reduces the load on the employees, and on the other, it guarantees that the activities and tasks are performed efficiently as compared to manual tasks and activities.
The focus, therefore, has to be on the process itself. The level to which a business process is optimized is directly related to efficiency of such a process. Today, having highly-optimized business processes is one of the most important priorities of companies.
Note
Optimized business processes are increasingly important for companies as they provide the company with a competitive advantage. With SOA, companies can optimize business processes easily, with less effort, and in a shorter time than with previous approaches.
However, it's not only the competitive advantage that matters; companies also need to react to changes in the global market, to new opportunities, and to threats from other companies. They react with modifications to their business processes.
The more efficient business processes a company has, the more efficient its operations will be. This will allow the company to have a competitive advantage over other companies and possibly become a leader in a certain area. Many of you might say that this is trivial to understand. Indeed, it is. But if we put emphasis on these topics, you will see that optimizing business processes is not easy, and not all companies define a systematic approach to business process optimization. Optimizing processes is also related to information systems. Each change in the process requires changes in application systems.
Note
Often, business processes are classified into internal or private business processes, and public or global business processes. Internal business processes are related to a single company. Global business processes connect two or more companies. Both types of processes are important and can be modeled and automated by SOA.
Before we dig into these topics, let us first look at some examples of business processes. Some examples of common business processes are supply, marketing and sales, procurement, and so on. Such processes take place in almost every company. They are called support processes.
However, the most important thing for each company is its "core" business processes—those processes that are directly related to the core business activities of the company. For example, a marketing company that manages advertisement boards relies on efficient processes that start with an order for advertising, and end when the advertisement is published on boards throughout the country or abroad. The faster the company can realize the orders, and the faster it can react to new requirements from customers, the better it is.
Note
Business processes are always specific to a company. This is particularly true for core business processes. Companies, therefore, cannot buy IT support for these processes in the market, but have to custom develop it to address their specific needs.
A company that produces car seats requires an efficient production process. Such a process can be automated, starting with the cutting of leather, continuing with the gluing of seat pillows, assembly of the seat carrying construction, and so on. In addition to production tasks, we can also automate other related tasks. Production can also be linked to the supply, which will open up new opportunities for supply chain management and help reduce stock levels. It is obvious that we are limited only by our own creativity.
A clothing company's core processes are design, production, and sale of clothes. The faster the turnaround between design and production, the more flexibility the company has to react to customers' wishes, and the better it can adapt to new opportunities. The better the connection between production and sales, the better the company can adjust its production to actual demand in the market.
A stockbroker company provides services related to buying and selling stocks. It also provides capital management services. Using the Internet, the company can develop applications that will allow customers to observe price changes in real-time and place orders online. This will change the processes dramatically and provide new opportunities for services that such companies offer their clients. At the same time, it will allow tighter integration with clients and closer observation of their needs. IT can help business processes change in order to get the maximum value from new technologies.
Note
SOA introduces new horizons in business processes optimization, where the only limit is our creativity. Business processes can provide an important competitive advantage.
In some industries, best practices related to business processes have been gathered and published. Such business process frameworks can be used for various reasons:
To standardize business processes in an industry
To ease integration between different companies from the same industry/sector
To use them as benchmarks to compare our own processes
To follow and improve our processes
Some experts are of the opinion that the best practices represent the average state in the industry. If our company is benefiting from such processes, it means that our processes are no better than average. Usually, companies that are above average keep their business processes confidential, because they know that their business processes reflect their true competitive advantage.
In the telecommunication sector, a well-known business process framework is Enhanced Telecom Operations Map (eTOM). eTOM defines best practices process frameworks for different aspects related to telecommunication business, such as:
Customer relationship management
Marketing fulfillment
Order handling
Problem handling
Customer SLA/QoS management
Invoice management
Service management
Service configuration and activation
Problem management
Resource management and operations
Resource provisioning
Partner relationship management
Marketing and offer management
Sales development
Service development and retirement
Supply chain development and management
Human resources management
Financial and asset management
We could find many more examples. However, the fact is that business processes are always specific to a single company, and are quite complex. A birds-eye view of the processes (as provided above) only tells half the truth. We can get a complete understanding of a business process only when we look at the details. It's in these details that the complexity hides.
When a company is established, its business processes are often not defined in a systematic way. Rather, they arise in a spontaneous way. It is the employees who often define the various activities and tasks. As the company grows, more and more people get involved in the processes, and processes become more and more complex. Processes in a large company differ from the processes in a small company.
In the real world, however, no single person can give a complete overview of an organization's business processes. The knowledge about how a process works is usually in the heads of employees and very often each person only knows about his or her own part of the process. The management does not motivate employees to think about the process, or even to think of optimizing it. Therefore, processes remain unchanged and are far from optimal.
If we agree with this, we can very simply conclude that if there is no single person in the company who could give an overview of the whole business process, then how can we know whether the company's operations are optimal. The management definitely wants to have the answers for the following questions:
Are the tasks and activities of a process organized in an optimal way?
Which tasks and activities require the most time to complete?
How are the activities and tasks distributed among employees?
How efficient are the employees?
We need to understand the business process in order to answer these questions. First, we should understand how business processes work. To do this, the usual approach has been business process modeling, which uses graphical (visual) languages to represent process flows, roles, and related documents. Business process modeling is not something very new. It has been around for quite a while and is a matured and well‑understood discipline.
We can model business processes in a variety of visual languages; the most widely used are EPC (Event Process Chain), eEPC (Extended Event Process Chain), UML Activity Diagrams, and recently BPMN.
We use business process models to understand the processes. This gives us the opportunity to modify and improve them, and to optimize them. Business process optimization and re-engineering are very important and it is up to our imagination to improve processes and integrate them to gain synergic benefits, and use other approaches to optimize them. We will not go into the details of business process optimization right now, because we will come back to this topic later in this chapter, and will discuss it again in the next chapter.
However, we will emphasize that business process re-engineering and optimization are related to several important topics that should not be overlooked. We will mention just three here:
Changing business processes requires changing the way people work. And people do not like change. Therefore, in order to be successful, we need to be careful how we apply the changes to the real world, and how we motivate the employees to change their way of working. Otherwise, a theoretical process, even if highly optimized, will not work in the real world.
Changing business processes does not mean changing only the behavior of the employees, but requires changing the IT support and related application systems. This topic is of particular interest for us, and we will look at it in detail in the next section.
Finally, changing the business processes only once is not the key to long-lasting success. If a company wants to have long-lasting success, it should develop an environment where business processes can be continuously optimized. This is a particularly difficult task, because continuous change in business processes also requires continuous change in the way employees work, and in the way IT supports the business.
We have defined business processes as a set of coordinated activities. We have also seen that most often employees perform these activities. Usually employees use applications to support their work. Sometimes, these applications fully support some activities, and the employee's intervention is not required.
With each new application that is introduced into a company, more activities become supported by IT. In other words, business processes are highly dependent on these applications, and vice versa. In addition, applications gradually start to impersonate business processes.
Note
Usually, business processes are tightly coupled with applications. Companies rely on IT applications in order to function. IT not only provides support for business operations, but has actually become an essential part of every business.
Today, businesses depend on IT. Can you imagine a business without IT? Can you imagine how any major company would operate, if IT does not function for one day?
Up to here, everything looks good. Nevertheless, we have not thought about the fact that business processes change over time. If the business processes are constant, we can afford to tightly couple them with application software. The fact, however, is that business processes are not constant and need to change. They need to be flexible. Therefore, IT also has to be flexible. In the next section, we will look at IT flexibility.
If each change in a business process requires a corresponding change in the IT system and in one or more applications, then the crucial question is: How quickly can we modify the applications?
The fact is that the time needed to modify the applications is crucial. In the eyes of the managers, this is 'lost' time, because the company needs to wait until the IT system is modified, before it can start using the new processes. Moreover, new processes might be better equipped to offer new or modified products or services. Therefore, the management will always put IT under pressure to do the modifications as quickly as possible—to minimize the IT gap.
To get a better understanding, let us look at a few examples. If an insurance company wants to offer a new insurance product, it has to upgrade the IT system before it can launch this new product in the market. The insurance company has to modify all the applications related to offering/ordering insurance products. This includes the applications on the office counters, the applications on the notebooks of insurance agents, the web site where insurance products can be bought online, in the call center for phone orders, and so on. Next, the applications for invoicing have to be modified. Various reports have to be modified too, in order to obtain information on how well the new product sells, who buys it, and how satisfied the customers are.
If a telecommunication operator introduces a new service or modifies an existing service, does this require changes to the IT applications? Yes, it does. It requires changes similar to those discussed earlier. In a telecommunication company however, such changes might also require modification in the software that runs the network. Today, there are many businesses where the IT support has infiltrated so deep into the company operations, that not only do support services rely on the application systems, but the whole core business operation has become dependent on the software.
Therefore, it is quite understandable that the management requires its IT systems to be flexible.
On the other hand, we know that IT systems in companies are usually very complex. We also know that complex systems require time to change. We can easily see that we are talking about two contradictory forces. One is the requirement from the management that changes should happen as quickly as possible. The other is the requirement from engineers, who require time to change complex systems.
In the real world, the request from management is usually of higher priority. This is why engineers are forced to change applications in a short time. Therefore, they do not have enough time to plan and design the changes. Even worse, they often need to apply modifications to software under pressure of time. In such conditions, they might be able to implement the changes, but changes may be done in a manner that negatively influences the overall software architecture and the entire information systems.
When such changes happen repeatedly, the overall software architecture becomes less robust. The changes get more difficult to implement, and hence require more time. This becomes a vicious circle. At a certain point, the architecture might even get so fragile that changes can compromise the integrity of the whole system. The dependencies and relations between the various software parts become almost unmanageable, and it takes so long to apply changes that it cannot fulfill the business requirements any more.
This is related to other impact factors, including:
The heterogeneous architecture of a typical information system
The usage of traditional software life cycles, which have not anticipated change
A typical information system today consists of a mix of heterogeneous application systems that have been developed over time. The application systems in a company are usually a mixture of:
Self-developed solutions
Custom-built, but outsourced solutions
Commercial systems such as ERP, CRM, SCM, and similar solutions
These systems use different architectural styles (client/server, multi-tier), different technologies, and languages (C++, Java, C#, Visual Basic, COBOL, and so on). It is quite unlikely that this mix of different systems was designed in a unified manner—it just grew, and will continue to grow. The fact is that companies rely on these systems and cannot afford to turn them off overnight.
Over time, some integration was achieved between the application systems, but it was not properly designed. This resulted in point-to-point integrations and the use of several different integration middleware products, including RPC, message brokers, distributed object models, proprietary integration servers, and in the recent years, web services.
Point-to-point integrations are very problematic, because as the number of involved systems starts to grow, the number of connections starts to grow exponentially. As point-to-point integrations are tightly-coupled integrations, they are very difficult to maintain. Changes in one system provoke a 'domino effect', whereby changes have to be applied in all related systems. Often, the complexity of maintaining the integrations becomes very high, and so does the cost.
To get a feeling for the number of necessary connections, let us presume that several applications have to be connected to each other. We will count only unidirectional connections, that is, if application A has to be connected to B, and B to A, then these are counted as two connections. With fifty applications, we would need an unbelievable 2,450 connections if we integrate applications on a point-to-point basis. To be honest, in the real world, we will probably not need to connect each application with the others. However, this does not change the fact that we will have to manage and maintain a large number of individual connections.
These issues are manifested while dealing with:
Combinations of monolithic, client/server, and multi-tier applications
A mix of procedural and object oriented solutions
A mix of programming languages
Different types of database management systems (relational, hierarchical, and object)
Different middleware solutions for communication (message oriented middleware, object request brokers, remote procedure calls, web services, and so on)
Multiple information transmission models, including publish/subscribe, request/reply, and conversational models
Different transaction and security management middleware
Different ways of sharing data
Possible usage of EDI, XML, and other proprietary formats for data exchange
All of these issues make applying changes to information systems even more difficult and time consuming, thus increasing the IT gap even further.
Existing applications have most likely been developed using traditional software life cycles. These consist of various stages, including:
Requirements specification
Analysis and design
Implementation
Testing
Deployment
Depending on the development process, these stages are either sequential, partially parallel (Waterfall model), or iterative-incremental (Rational Unified Process). No matter which development process is used, they all are based on the following two facts:
The requirements are specified in advance. These requirements should be specified as precisely as possible, because the architectural design depends on the requirements. Real-world experience, however, shows that in most of the cases, it is difficult to define the requirements in advance. It is also very common that the requirements change, even throughout the development.
Traditional software development processes were not designed for continuous change. Rather, they relied on certain precise requirements. Modifications to the requirements need to be run through the whole development cycle again (because requirement gathering is usually the first stage), which is very time consuming. This is not aligned with business expectations, which require quick changes.
With this, we do not want to say that requirement specifications are not necessary. In contrary, it is very important to specify the requirements. The key is to develop a software architecture that will anticipate change and be flexible to change. Only such an architecture can fulfill the requirements of the next generation of information systems—information systems that will provide end-to-end support for business processes. Existing software architectures have not fulfilled these expectations.
Service Oriented Architecture (SOA) is the architecture of the next generation of information systems. It provides answers to the problems identified in the existing software architecture. SOA enables the development of applications that are more flexible, and more adaptable than applications built using traditional architectures. SOA applications are, therefore, much easier to modify and adapt.
SOA also enables better alignment between the business processes and the applications. As we will see later in this chapter, SOA minimizes the semantic gap between the business process models, and the actual application software. SOA achieves this with the introduction of new technologies and languages, most importantly, BPEL.
All of this is in line with our primary objectives:
To minimize the IT gap, and the time required to modify the information system in response to the changes in business processes
To make the information system more flexible and adaptable to change through a loosely-coupled approach
To provide end-to-end support for business processes
Finally, to make the company more agile, more flexible, and to allow it to adapt more easily various forces, such as competitive pressures, new opportunities, changes in the global market, and so on
The fact is that in the past there have been several attempts to make software architectures more flexible. There have also been several attempts to align business process modeling with software development (and vice versa). However, the fact is that the problems continue to exist even now.
To be able to answer this question, we will first look at the business processes, and how they are managed today. We have already identified that:
Business processes in a company usually emerge in a nonsystematic way
Business processes are dynamic and change over time
Often there is no single person who has all of the details of all of the business processes in a company
The classical approach to BPM foresees the cataloguing of all business processes and drawing the exact specifications of each business process. The output of such projects are "nice pictures" of processes. We have used the phrase "nice pictures" because capturing a business process can only result in a snapshot of a process at the time when the snapshot was taken. However, business processes are dynamic and change over time. Therefore, such pictures of processes cannot correspond to the real processes for more than a limited period.
Larger companies usually face two additional problems:
Modeling business processes (specified as one-time projects), can take quite a while. It can happen that even before we complete the project, some process specifications might be out-of-date (this is particularly true for processes that have been modeled at the beginning of the project).
Business processes are also usually quite complex. While specifying them, the most difficult part is to provide in-depth specifications. The complexity usually hides in the details. Therefore, we might get high-level process specifications. However, such specifications may be very simplistic, and may not include important details. This could influence our understanding of the processes. Very often, process specifications do not include exceptional scenarios, and do not foresee how to react to such scenarios.
The following figure provides an example of a process specification:
Even if we succeed in modeling business processes in detail, and in a relatively short span of time, there is another huge problem we have to face with the traditional approach – the business process activities and tasks are performed by the employees and the software. There is a huge semantic gap between what goes on in the information systems and the process model diagrams. This semantic gap exists because it is very difficult to relate the changes in the processes with the changes required in the information systems, and vice versa. Therefore, it is also very difficult and time-consuming to maintain the business process models in-sync with the actual software. This is very time-consuming because it has to be done manually, and is therefore not done in real-world environments, where time is valuable.
Part of the problem is hidden by the fact that enterprise systems are still developed at a relatively low level. Even if we use modern software platforms, such as Java EE or .NET, we are still required to develop in languages such as Java and C#. These are object-oriented languages, which use concepts such as classes, objects, inheritance, delegation, and so on. These concepts are not synchronized with business processes. Java, C#, and similar languages are still relatively low-level, compared to business processes. Therefore, we sometimes refer to them as 'programming-in-the-small'.
SOA has introduced several new approaches to software development that we will look at, in the next section.
The SOA approach differs considerably from traditional approaches, although this may not be obvious at first sight.
SOA introduces technologies and languages that reduce the semantic gap between the business processes (pictures) and the actual applications (code). Particularly important here are BPMN, which is used for modeling business processes, and BPEL, which is used for the execution of business processes. With these two technologies, plus some additional ones (which we will mention later in this chapter), SOA provides:
A language—BPEL—for direct execution of business processes
Round-trip mapping between the process models in BPMN, and their executable representation in BPEL
With this, SOA considerably reduces the semantic gap between the business processes and application systems. BPMN enables us to draw the representation of a business process, which is then mapped into the executable BPEL code, and executed directly on the SOA platform.
The other major difference in SOA, as compared to the traditional BPM, is the approach to business process automation. SOA takes a path different from the traditional approach which was based on the "big bang" approach, where processes were first modeled, then optimized, and finally the application software was rewritten to support the new processes. SOA has learned from the major mistakes of the traditional BPM approach:
Modeling of business processes has taken too long and has not been done in enough detail. Therefore, the business models have not been accurate enough, were usually without exceptional scenarios, and have been a little out of date.
The optimization has been done "in the heads" of the analysts. Although optimizations looked good on the model, there has been no proof that they would work in the real world.
As there has been no evidence that the optimizations are acceptable, the modifications in the software, which had to be done in order to support the optimizations, have not been justified enough.
Finally, the introduction of new optimized processes, together with the modified application software, has often represented a huge change in the operations procedures. This has resulted in changes in the behavior of employees. Such changes, particularly if they have been huge, have often met with resistance from at least a few employees, which ensured that the fragile model did not work anymore.
These are the three major lessons SOA has learned:
Business processes are so complex that we could approach them all over.
Business processes are too valuable to a company to be merely optimized "on paper" (even if there has been support from tools that enable simulation).
Changes in business processes are not simple and particular attention has to be paid to the fact that changes in business processes require changes in people's behavior.
Note
People do not like to change their behavior. Changes to business processes require that people change their behavior. Therefore, the human aspect of process optimization must never be underestimated.
The SOA approach to business process automation therefore relies on the process-by-process approach:
First, we identify the business process that we would like to automate. Here, we focus on the business value of the process, the visibility, and other factors. We also asses the real value of process automation and possible optimizations.
Then, we model the process. Here we use a visual notation. In this book, we will explain why it makes sense to use BPMN while modeling business processes for SOA. The process modeling for SOA has to be done in detail. It is important that we model the process in detail so that we identify individual activities that are atomic from the perspective of execution. It is also important that we model the exceptional scenarios.
Note
Exceptional scenarios define how the process behaves when something goes wrong. In the real world, business processes can go wrong. Therefore, it is important to model processes so that they can react to exceptional situations and recover appropriately. Modeling exceptional scenarios may be even more complex than modeling the regular process flow.
Next, we automate the process as it is, without any major modifications. In SOA, this requires mapping the BPMN process model into the executable representation in BPEL, and connecting the BPEL process with partner links, that is, with services. In other words, it is required that we relate process activities to the various services. This step is best carried out if we have a portfolio of existing services that are re-usable and in conformance to other SOA characteristics (which we will describe later in this section). If we do not have existing services, we will also need to implement the services, where we have three options:
To implement new services
To expose the business logic from existing applications
To use user tasks to delegate the activities to employees, and possibly automate these tasks in the future
Once we automate the process, we will take up process optimization.
The step-by-step approach to optimization is much more efficient and friendly to the people involved in the processes. We have already mentioned that people do not like to change their behavior. Therefore, it is much wiser to implement changes in phases. This is sometimes called the evolutionary approach to business process optimization.
The SOA approach has another important advantage. As we have automated the process (as described in bullet three), we can obtain some measurements about the different process activities, and how long in average they need to execute. Such quantitative metrics, which are calculated automatically by modern SOA platforms, can provide valuable information that can be used to decide where to start process optimization: we can focus on activities where we can gain the largest improvements. Gathering quantitative data about process activities is called Business Activity Monitoring (BAM).
Note
BAM, which is a part of modern SOA platforms, provides valuable quantitative data about process execution, which can then be used to decide on the point from which process optimization should start.
We can see that the SOA approach to business process automation is iterative and incremental, as shown on the figure below:
When done properly, the usual duration of one circle (iteration) is three to four months. In other words, the SOA approach can produce new automated processes or optimized versions of already automated processes in approximately three- to four-month intervals. This is very important, because the SOA approach delivers results in relatively short intervals. This way, the whole company will recognize that IT delivers useful results. This can improve the position of IT, particularly if IT has not been efficient enough in the past.
There are other important benefits to the SOA approach:
It allows step-by-step identification of important business processes, and the in-advance assessment of the benefits of process automation and optimization.
It allows the in-advance assessment of costs and benefits, and ROI (Return on Investment) calculation. It also allows continuous monitoring of total costs throughout the process.
Owing to the reduced semantic gap between the business and IT, the communication of requests to IT becomes more precise and efficient, thus minimizing misunderstandings.
The systematic approach to process optimization requires less adoption for the employees, who therefore, becomes more amenable to changes.
The real-time monitoring of process performance (through BAM) provides valuable insights into the efficiency of the current processes, and helps in identifying points of optimization.
With SOA, IT becomes more flexible and can respond faster to changes in business. This makes the whole company more agile, which improves its competitive position, reduces its response time to market changes and reduces competitive threats. It also allows the company to react faster to new opportunities.
We have seen that SOA focuses on business processes. The business processes are the content of the business. This means that SOA shifts the focus of IT from technology to business content. This is good, because the technology should be seen merely as an enabler. The less we are aware of the technology itself, the better it serves its purpose.
In the traditional IT approaches that are in common use today, the focus has been more on technologies than on content, as shown in the following figure:
The Improvements in the performance of computers have allowed us to think about software development at higher levels of abstraction. This has started with the shift from assembler to higher -level programming languages and has since then continued with the introduction of new layers, which have simplified the development and made it less complex.
SOA continues this journey with an important addition. It raises the level of abstraction in the direction of the business content, and away from the technologies. Although some technology-oriented IT people may not like this scenario, businesses may find it useful as they can now focus more on their business' content. The following figure shows how SOA has changed the focus of IT:
SOA, therefore, puts focus on the content of business.
Note
Who in the company is responsible for the business content? Is it the IT department? Usually not! SOA shifts the focus of IT from technology to business. SOA is not the domain of the IT department alone.
This brings us to the conclusion that SOA is not the domain of the IT department alone. As the focus is on business processes, it is important for SOA to outgrow the boundaries of IT departments and involve other departments as well. For success, we should involve the whole company, starting with the management.
SOA is about business content, business processes, aligning IT with the business, and optimizing the business processes.
For an SOA project to succeed, it is very important to gain management's support. Without support, it is unlikely that we will be able to implement all of the required changes. Particularly problematic are the changes relating to the behavior of employees. Often, such changes are possible only if approved by the top management.
The other important reason why management support is necessary is that IT alone does not have the required knowledge about how business processes work in a company. It has to obtain this knowledge from the employees, who have to provide information on how the processes work.
Finally, SOA is also about business process optimization. The IT department usually does not have the authority to decide about changes in business processes. Therefore, process owners have to be involved in the SOA project. For this, orders have to come from the management.
We have seen that because of its focus on business content, SOA requires the involvement of not only IT, but also the other departments in a company. We have already mentioned that management and process owners have to be involved. Employees with knowledge about the business processes, the so-called key users, should also be involved.
Nevertheless, as the SOA projects usually arise from the IT departments, we could face some organizational barriers that prevent the people mentioned below from working together efficiently. Therefore, SOA will most likely require organizational changes.
The solution to these challenges can be a new organizational unit, called the SOA Competency Center. The SOA Competency Center should involve the following people:
SOA project leader
Business process analyst
Process owner
Key user
Technical architect
Process developer
Service developer
Integration and test expert
Representative for SOA governance
So far, we have seen that SOA is a comprehensive project that includes:
Business aspects
Technical aspects
Organization aspects
The most important business aspects, based on an SOA survey that the author has conducted among 283 IT departments of large and medium-sized companies, are:
Faster adaptability to changes and better adaptation of business process
Improved efficiency of business processes
Better alignment with IT and business requirements
This results of the survey are shown in the following figure:
SOA is a long-term project, and it is very important that it is seen as such. In other words, SOA is a long-term development of the overall IT architecture. To be successful, we have to plan the project carefully:
We have to set the objectives. We have to identify the goals of SOA. It is important that we articulate the objectives very precisely. Just saying that we would like to improve the efficiency of business processes is not adequate. We have to identify which processes we would like to improve, why, when, and by how much. Only when we have a deep understanding of this, will we be able to move forward.
We have to identify the risks. There are many risks involved with the SOA project, starting with the organizational aspects, selection of processes, technology-related risks, and so on.
We have to take the necessary organizational steps and also educate the SOA team members at the same time. Here it is very important to understand that SOA introduces many changes to all aspects of application development. Team members have to understand these changes. They also have to understand the new technologies and languages.
We have to select an appropriate SOA platform. Major vendors today offer SOA platforms that differ in several important aspects. Careful selection is therefore necessary. We have to take into account specific aspects of the environment, existing systems, and existing knowledge in order to make a good decision.
An SOA project is usually started with a pilot project, which should be done with the help of external SOA experts. Within the SOA pilot, several aspects can be addressed. The most important aspect is probably that our team has to get used to the round-trip development of business process modeling, and their transition into executable BPEL processes. In other words, the SOA team has to feel comfortable with the composite approach to development.
We have seen that SOA aligns the business and IT aspects. Listed here are the five most important business aspects:
Business models
Business processes
Organizational structures
Workforce
Business functions such as marketing, finance, and so on
The following are the most important IT aspects:
IT architecture
Application systems
Security
Transactions
Databases
Hardware infrastructure
There are four major categories of forces that influence business process automation using IT:
Business aspects such as agility, competition, new opportunities, customer demands, and customer contact
Organizational aspects such as the need for optimization, improvements in efficiency, and cost reduction
Increased complexity, integration demands, and standards
Introduction of new technologies, such as Web 2.0, new devices, and architecture
This is illustrated in the following figure:
Besides having business value, SOA also introduces important benefits to the IT departments:
IT departments are under constant pressure due to frequent changes. SOA makes such continuous changes easy, and reduces the negative effects of changes.
Duplication of data across different databases and systems is quite common in existing systems. SOA fosters consolidation of data and introduces master data management solutions based on SOA concepts, services, and loose coupling.
In existing applications, we usually face duplicated functionalities. SOA fosters consolidation of such duplicated functionalities. Using services, we can expose composed functionalities.
When talking about business processes, companies often have variants of business processes that differ only in details. SOA enables support for such variants and their modifications with a common base process.
With the diversity of devices, it becomes more and more important to enable access to applications and data through different channels (PCs, palmtops, cell phones, voice, and so on). SOA enables access to processes through different channels.
IT departments often do not develop everything in-house. With the existing approaches, it is quite difficult to separate the roles of the external partners (outsourcing partners) and in-house development. Too often, the outsourcing partner has gained control over the application it has developed, and the IT department (and the company as a whole) has become dependent on the partner. SOA enables easier separation of responsibilities, whereby services can be outsourced while their integration into business processes stays in-house. In this way, IT departments retain control over the most valuable know-how a company has: the business processes know-how.
Finally, SOA enables the development of service networks, which in turn enable the development of virtual value chains, not only within the company, but also between companies. This can open completely new possibilities in how IT can be used to optimize the business.
SOA has also learned from the experiences of the existing software development methods. These methods were based on the specification of requirements, and had foreseen the analysis, design, implementation, testing, deployment, and maybe some other phases. However, the core assumption was that requirements had to be specified as precisely as possible. Changing the requirements has always been seen as something that is undesirable.
The fact today, however, is that changes are expected.
SOA has considered change from the start. Therefore, it has introduced some important modifications to the development approach. Instead of the classic approach shown here:
SOA introduces a modified approach, which consists of the following phases:
We can see that the phases are quite different. Instead of the analysis, the SOA approach foresees modeling, which refers to business process modeling. This way, the development is aligned better with the actual needs of the business. This is the first advantage of the SOA approach.
The second phase in the SOA approach is composition. Composition refers to the way business processes are developed. Instead of traditional implementation in a programming language, such as Java or C#, in the SOA approach, we have to develop business processes so that we can re-use services and compose (orchestrate) them into processes. This approach works best when we already have a portfolio of services such as:
Services from existing applications, where we expose business logic as services
Services that are bought, or for which development is outsourced, to external companies
Services developed in-house
In all three scenarios, we have to follow certain guidelines. The most important one is that we develop services that are re-usable. This is not an easy task and we will talk more on this later in this section, when we describe the technical characteristics of SOA.
Re-usable services are very important for SOA, because they represent "big" building blocks, which contain business logic. Developing applications (processes) with such existing building blocks is much faster compared to the traditional approach in Java or C# (even if we have some libraries available). Therefore, the SOA approach to development is sometimes called programming-in-the-large. Composition of services (using BPEL) has been designed for such a purpose.
The third phase of SOA development approach is testing. Testing SOA applications refers to testing of the process and related services. However, we re-use services that have already been tested. Therefore, the effort required for testing is also reduced, as services do not have to be unit-tested again. Although, processes may still have to be tested (integration testing), the overall effort for testing is reduced.
Finally, we come to the monitoring phase. This phase refers to run-time monitoring of the process performance and includes:
Monitoring of business activities, say BAM, which provides valuable information about the performance and efficiency of business processes, and can serve to identify future optimization points.
Monitoring of QoA aspects of processes and services, such as response time, security, availability, and so on. This is often related to the definition of a SLA (Service Level Agreement) for processes and services.
Leading SOA platforms from vendors such as Oracle, IBM, Microsoft, and others support both these aspects of process monitoring.
The changes in the development approach described above considerably reduce the overall complexity of the development. According to some estimates, SOA reduces the complexity by approximately 50%. This is very important because the increased complexity of application systems has been a significant source of the problems related to the long response time, needed for application modifications.
The development of composite applications is one of the major contributions of SOA, which brings us to the technical aspects of SOA. In this section, we will look at the technical background of SOA.
To be able to develop composite applications, that is, to compose business processes out of services, we need technologies that will allow us to develop services. The obvious answer to this are 'web services', as they are best aligned with the concepts of SOA. With this, we want to emphasize that SOA is not bound to specific technologies. What matters are the concepts. The most important thing from the perspective of a developer is the service description, the WSDL (Web Services Description Language).
We also need a language and a technology to perform the composition of services into processes, and an environment in which to execute the processes. You may argue that we could use well-known programming languages such as Java or C# for this. It turns out that the process composition differs somewhat from traditional programming. With composition, we merge services into larger services and processes. In other words, we represent the high-level state transition logic of a process. Using programming languages such as Java, C#, and so on for these purposes is likely to result in inflexible solutions, particularly because there is no clear separation between the process flow and the business logic, and these should not be tightly coupled. However, we continue to use traditional programming languages to develop services.
In addition to these facts, the composition of business processes has other specific requirements such as support for many process instances, long-running processes, compensation, fault handling, correlation, parallel flows, complex dependencies, asynchronous invocations, and so on. All of these make it reasonable to use a specific language and a specific environment for business processes.
In SOA, the language used is BPEL and the environments for executing processes are called process servers (sometimes called BPEL engines). Process servers provide additional valuable features, such as an overview of the running processes, completed processes, faulted processes, and so on. They provide a complete history of process execution and insight into process activities.
BPEL is an executable language. We are emphasizing this because, in order to execute BPEL processes, we have to specify details such as interfaces, messages, variables, types, and so on. These details are usually not needed at the time when we model the business processes. Therefore, for modeling, we will use a modeling notation, the BPMN. BPMN has been designed explicitly for process executions. Therefore, it provides round-trip mapping from BPMN to BPEL.
This brings us to the simplified SOA architecture, as shown in the following figure:
Let us now look briefly at the different building blocks. First we will look at the BPMN, then at the BPEL, and finally at the services.
BPMN is a graphical notation used for business process modeling. We use BPMN to draw business process diagrams. These diagrams present the activities and tasks of a process and their relations. The diagrams use flowchart concepts to represent the logic of business processes.
BPMN is a visual language, and uses a set of graphical elements. Activities are represented as rectangles, and decisions as diamonds. BPMN successfully joins the simplicity of the diagrams with expressive power, which allows BPMN to be used for complex processes and specification of details.
We will take a closer look at the syntax of BPMN in the next chapter.
BPEL has been adopted as the de facto standard for developing executable business processes. The main goal of BPEL is to standardize process automation between services.
Note
With BPEL, we can describe business processes in two distinct ways: we can either specify the exact details of business processes, or specify the public message exchange between parties. The former processes are called executable business processes and can be executed by a process server. The latter processes are called abstract processes. They do not include the internal details of process flows, and are not executable. In the real world, BPEL has been used predominantly for executable processes.
Executable business processes are processes that comprise a set of existing services and specify the exact algorithm of activities, along with the input and output messages. With BPEL, we can define business processes that make use of services, and business processes that externalize their functionality as services. When we define an executable business process in BPEL, we actually define a new service that is a composition of existing services.
Within enterprises, BPEL is used to standardize enterprise application integration, and extend the integration to previously isolated systems among business partners. Between enterprises, BPEL enables easier and more effective integration with business partners. Further, BPEL encourages enterprises to define their business processes, which in turn leads to business process optimization, re-engineering, and selection of the most appropriate processes, thus further optimizing the organization. Definitions of business processes described in BPEL do not influence existing systems, thus stimulating upgrades. BPEL is the key technology in environments where functionalities already are, or will be, exposed via services.
BPEL represents a convergence of two early workflow languages, WSFL (Web Services Flow Language) by IBM, and XLANG by Microsoft. IBM, BEA, and Microsoft jointly developed the first version of BPEL in August 2002. Since then, several other partners have joined. Version 1.1 was adopted in March 2003. In April 2003, BPEL was submitted to OASIS (Organization for the Advancement of Structured Information Standards) for standardization purposes, where the WSBPEL TC (Web Services Business Process Execution Language Technical Committee) has been formed. In 2007, WS-BPEL version 2.0 was adopted.
BPEL uses an XML-based vocabulary to specify and describe business processes, and is based on WSDL, XML Schema, and XPath specifications. Familiarity with these specifications will be helpful while learning BPEL.
With BPEL, we can define both simple and complex business processes. BPEL offers constructs such as loops, branches, variables, assignments, and so on, which allow us to define business processes in an algorithmic way. The most important BPEL constructs are related to the invocation of services. BPEL makes it easy to invoke operations of services either synchronously or asynchronously. We can invoke operations either in sequence, or in parallel. We can also wait for callbacks. BPEL provides a rich vocabulary for fault handling, which is very important, as robust business processes need to react to failures in a smart way. BPEL also provides support for long-running process and compensation, and allows us to undo partial work done by a process that has not terminated successfully. The following are the most important features that BPEL provides:
Describing the logic of business processes through composition of services
Composing larger business processes by combining smaller processes and services
Handling synchronous and asynchronous (often long-running) operation invocations on services, and managing callbacks that occur later
Invoking service operations in sequence, or in parallel
Selectively compensating completed activities in case of failures
Maintaining multiple long-running transactional activities, which are also interruptible
Resuming interrupted or failed activities to minimize re-work
Routing incoming messages to the appropriate processes and activities
Correlating requests within and across business processes
Scheduling activities based on the execution time and defining their order of execution
Executing activities in parallel and defining how parallel flows merge based on synchronization conditions
SOA and BPEL approaches rely on services. We compose services into processes. To be able to take advantage of this approach, we need services. Services are discrete self-contained building blocks of SOA that contain business logic. Services expose business logic through well-defined interfaces.
Services provide business functionalities such as applications for business travel, applications for loans, and so on. This differs considerably from technology-oriented functionalities such as retrieving or updating a table in a database. Services in SOA must provide business value, hide implementation details, and must be autonomous. Therefore, in SOA we usually organize services in several layers. In the top-most layer, we have business services.
Note
Business services should represent activities of a business process. At the same time, they should be generic, so that they can be used in different business processes.
Such business services cannot be developed in an ad-hoc manner. They require a lot of planning and designing. The success of SOA relies on services and the ability to define these services at the right level of abstraction, to make them re-usable, and to use them in different processes. From this perspective, we could say that SOA is the development of the overall IT architecture.
Business servers are usually composed of lower-level services. At the bottom, we have technical services. For example, these can be stored procedures, exposed from the database, or functions or methods of an existing application, and so on.
Defining the right services and positioning in the right layers requires some planning and designing. It also requires us to be familiar with good practices and patterns. This should lead us to the architecture as shown in the following figure:
Note
From the technical perspective, processes and services do not differ. They both expose interfaces, which are described in WSDL. Therefore, we can develop composed services using different approaches (BPEL, programming languages, and so on). We can also re-use services and/or processes. As they look similar from outside, they give us the opportunity to develop multiple layers of services and processes, from the simplest services to the complex end-to-end processes.
It is also important to know how service interfaces are defined. The interface of a service defines a set of public operation signatures. The interface is a contract between the service provider and the service consumer. Interface is separated from the implementation, is self-describing, and is platform-independent. In order to define business services, we need to focus on the correct granulation of operations. SOA services are best modeled with coarse granulation.
Service operations are defined as a set of messages. Messages specify the data to be exchanged and describe it in a platform-and language-independent way, using schemas. Services exchange only data, which differs considerably from object oriented and component approaches, where behavior (implementation code) can also be exchanged. Operations should also be idempotent (operations are idempotent if repeated invocations have the same effect as one invocation).
Service consumers can use synchronous or asynchronous communication modes to invoke operations of services. In synchronous mode, a service operation responds to the service consumer after the processing is complete. The service consumer has to wait for its completion. Usually, we use synchronous mode when operations complete processing after a short period of time. In asynchronous mode, a service operation does not respond to the consumer, although it may return an acknowledgement so that the consumer knows the operation has been invoked successfully. If a response is needed, a callback from the service to consumer is used. In such a scenario, message correlation is needed.
Loose coupling of services is achieved through self-describing interfaces, coarse granulation, the exchange of data structures, and support for synchronous and asynchronous communication modes. Loosely-coupled services are services that expose only the necessary dependencies and reduce all kinds of artificial dependencies. This is particularly important where services are subject to frequent changes. When a service is modified, minimal dependencies ensure that changes made to other services are minimal. Such an approach improves robustness, makes systems more resilient to changes, and promotes re-use of services.
For developing truly re-usable services, it is also important that we care about attributes, such as availability, performance, security, and so on. These are called quality of service attributes. Quality of service attributes are important in large information systems. In web services, quality of service attributes are covered by WS-* specifications such as WS-Security, WS-Addressing, WS-Coordination, and so on.
The scenario described above with BPMN, BPEL, and services covers only the most essential parts of SOA. For a full-blown SOA, other important parts are required. These include:
Enterprise Service Bus
Registry and repository
Rules Engine
BAM
User interactions (human workflow)
SOA is also related to the following:
Unification of presentation layer, which is related to portals
Security and identity management
These layers are shown in the following figure:
In the coming sections, we will describe the building blocks of a full-blown SOA architecture, starting with the Enterprise Service Bus.
The Enterprise Service Bus (ESB) addresses the communication between the services and the processes. We have already mentioned that for services, we can use various technologies. In most cases however, we will use web services. Web services communicate using the SOAP protocol, which can be mapped to one of the transport protocols, most likely to HTTP. HTTP is an unreliable protocol, and is therefore not well-suited to mission-critical applications.
ESB addresses this problem and provides a reliable mechanism for communication between services. In addition, it offers protocol mapping capabilities. This means that ESB can translate different protocols, such as SOAP to JMS (Java Message Service) or vice versa. This is important because in enterprise information systems, we usually have some components already developed in different technologies. We might even have a kind of middleware, such as a message broker. ESB helps to connect and integrate these systems.
ESB can also enable reliable message delivery and provide other value-added features, which are very useful in mission critical systems. Usually, ESB also addresses aspects of security and transaction management. It allows us to configure secure access to the services and processes, and to monitor the access. It also allows the definition of services that should participate in transactions. Some ESBs even allow us to choose between ACID (WS-AtomicTransactions) and compensating transactions (WS-BusinessActivity).
Services in SOA are described in WSDL, and use input and output messages, which are formatted using XML Schema. In the ideal world, all services would use the same schema definitions. In the real world however, this is not the case. Therefore, we often have to transform the XML payload from the output schema of one service to the input schema of another service. For such transformations, we often use XSLT, XPath, or XQuery. ESB provides the ability to do such transformation transparently, on the bus, without having to modify existing services or processes. This capability to transform XML payloads is particularly valuable when new versions of services or processes are deployed. New versions might change the interface or the message schemas somehow. With ESB, we can mask these differences without having to modify all related clients.
Related to transformations is message enhancement. Sometimes, it is not enough just to transform the XML payload. We may need to add or even alter certain information in order to make the message compliant with the service definition. Such alterations may can be simple, such as altering the representation of date, or perform more complex operations such as calculating data, translating types, and so on.
ESB can also serve as a message router. Sometimes, we might want to route messages to certain services. Such routing is usually done based on some deterministic business rules, or can be based on message content, user information (which user created the message, or which user is the receiver of a message), or the time when the message was created. Routing messages on the bus gives us the advantage of doing this in the Good ESBs are integrated with Rules Engines, and allow us to use the rules from the engine for routing.
Routing is related to service mapping. ESBs can also be used to dynamically map a service implementation to the corresponding WSDL interface. On ESBs, this can be done in run time, which gives us greater flexibility over which implementation we will use. If this feature is used with care, it can improve overall flexibility.
In future, we believe that SOA will introduce another important concept: business events. This will extend the way services and processes interoperate. Instead of invoking services by calling their operations, we will simply trigger a business event. Such events will be consumed by those services and processes that have subscribed to the event. ESB will play a central role in this scenario, because it will monitor, route, and manage events in the same way as service invocations.
A very important aspect of SOA is re-use. Re-use at the service level means that when we compose executable processes (in BPEL), we re-use existing services as far as possible. Successful re-use is not easy to achieve and requires at least the following:
The developers are motivated to develop re-usable services. Developing a re-usable service initially takes longer than developing a single-purpose service. Re-use pays off in the second step, when the service is actually re-used. The more it is re-used, the more justifiable this initial longer development becomes.
Developers also have to be motivated to search for services and to identify the most appropriate services to use. They may also need to modify the behavior of the service slightly. In such a case, they need to know who else uses the services, and the boundaries within which they can make changes.
To achieve maximum re-use and keep the architecture sound, it is useful to have some sort of governance in place. Governance procedures require information on who uses a service, how many times a service has been re-used, what level of re-use is achieved in processes, and so on.
Another very important aspect is the deployment process. A professional deployment has three environments:
development
test
production environment
When an application is ready for production use, it is deployed into the production environment. Controlled deployment of processes requires that a process starts using the production version of services instead of the development or test version. If references to services (partner links) are hard-coded into the BPEL processes, such migration can be very painful, as it might, in the worst case scenario, require manual changes to URL addresses. When deploying a service, we might want to know which processes use this service, because we might want to retest those processes.
Addressing all of the aspects mentioned above is very difficult if we have no list of the available services, or when processes and services are tightly coupled (for example, when a process uses the direct URL of a service in the partner link).
Registries and repositories address these aspects. They are used to register services in a central location. Once registered, we can search and locate appropriate services at design time and at run-time. The more metadata we include for a service, the better the search capabilities that the registry and the repository can provide.
The rule of thumb is that once we have more than 50 services, we will desperately start needing the registry and the repository. However, sometimes it might be wise to introduce it from the beginning, because once developers get used to a certain development process, it will be very difficult to change their behavior.
Note
The current notion about the role of registries and repositories in SOA has changed considerably. A few years ago, we believed that a relatively simple registry—UDDI would cover all our needs. Today, we have identified that a registry alone is not powerful enough for SOA, because in many cases it makes sense to store service metadata as well (WSDL interfaces, XSD schemas, and many more). Therefore, today we talk about registries and repositories.
A powerful registry and repository should have the following features:
Classification capabilities to categorize and classify services and processes based on one or more classification schemas. This simplifies queries and enables easier location of the most appropriate services for re-use.
Governance functions to enable the definition of proprietary service/process lifecycles together with the conditions to go from one stage of the life cycle to another. Stage transition can trigger automatic actions such as execution of validators.
Access control to define who can perform which operations against the registry and repository, and for which registered services/processes. Such access control could be based on XACML (eXtensible Access Control Markup Language).
User, programming, and administration interfaces.
Business rules are an important part of each enterprise. Experience has shown that business rules change; sometimes very often, sometimes not that often. Today, business rules are coded into different applications and are tightly-coupled with the implementation of an application system. Changing business rules is therefore very difficult, and requires modifications to applications. Each modification requires testing and deployment, which makes things even more complicated. The major challenge today, however, is to identify the applications into which the same business rule is coded, and to modify all applications where this rule has been used. If we forget some applications, this could lead to unpredictable results, which we do not want to happen.
Business rules are very common in business processes. The idea of SOA has been to extract these rules from the executable code (from BPEL, Java, C#, and so on) and to store them in a central place, where:
Business rules can be re-used from different processes, services, and applications
User friendly interfaces exist, which enable us to change and modify business rules
Often, business processes manage business rules. Therefore, putting business rules into a central place, the Rules Engine, is appropriate because the same rule is used very often in more than one process. This might not be obvious at the start, when we will have just a few processes automated, but it will become important when the number of processes start to grow. However, similar as it is to the registry and repository, it makes sense to introduce business rules early. This is because if rules engines are introduced after we have implemented several processes, we will have to:
Change the way development team works
Extract the rules from the existing processes and services, which will require significant changes to these processes and services
With the introduction of Rules Engines, SOA now addresses the particular challenge of changes in business rules.
Business Activity Monitoring (BAM) helps in answering the question "Where to optimize business processes? Once a process is automated end-to-end, we can use the SOA platform to provide quantitative data about the performance of the process.
A BAM solution gathers information about the time required to complete different activities of a process. It can then provide reports with average values for each activity. Having these numbers can be very helpful, because we can use them to identify which activities take the longest time to execute. With this, we can identify those parts of a process, where we will realize the largest benefits from optimization.
BAM solutions can provide other useful information. They can show us how many process instances are active at a specific time, and how long, on an average, it takes to complete a process. They can also show us which users (employees) have started how many process instances, and so on.
Please note that BAM is not just related to automatic activities (those implemented by services). It can be related to human activities as well. In such a case, we can use BAM to observe the productivity of the employees.
This brings us to the next important part of SOA—the human activities or user tasks: user interactions.
User interactions in business processes are very common. It is true that SOA's objective is to automate as many activities as possible. However, this cannot be done overnight. Therefore, we will need to model certain activities of a process as user interactions—human tasks.
User interactions in business processes can be simple, such as approving certain tasks or decisions, or complex, such as delegation, renewal, escalation, nomination, or chained execution. Task approval is the simplest and probably the most common user interaction. For example, in a business process, say for opening a new account, a user interaction might be required to decide whether the user is allowed to open the account. If the situation is more complex, a business process might require several users to make approvals, either in sequence, or in parallel. In sequential scenarios, the next user often wants to see the decision made by the previous user. Sometimes, particularly in parallel user interactions, users are not allowed to see the decisions taken by other users. This improves the decision's potential. Sometimes, a user does not even know which other users are involved, or even whether any other users are involved at all.
A common scenario for involving more than one user is workflow with escalation. Escalation is typically used in situations where an activity doesn't fulfill a time constraint. In such a case, a notification is sent to one or more users. Escalations can be chained, going first to the first-line employees and advancing to senior staff if the activity is not fulfilled.
Sometimes it is difficult or impossible to define in advance which user should perform an interaction. In this case, a supervisor might manually assign the task to an employees. A group of users or a decision-support system can also assign a task.
In other scenarios, a business process may require a single user to perform several steps that can be defined in advance, or during the execution of the process instance. Even the simplest processes might require that one workflow is continued with another workflow.
User interactions are not limited only to approvals. They may also include data entries or process management issues such as process initiation, suspension, and exception management. This is particularly true for long-running business processes where, for example, user exception handling can prevent costly process termination and related compensation for those activities that have already been successfully completed.
As a best practice for human workflows, it is usually not wise to associate human interactions directly with specific users. It is better to connect tasks to roles and then associate those roles with the individual users. This gives business processes greater flexibility, allowing any user with a certain role to interact with the process, and allowing changes to users and roles to be made dynamically.
To interleave user interactions with BPEL processes, we can use a workflow service, that interacts with BPEL processes using standard WSDL interfaces, as any other service. This way, the BPEL process can assign user tasks and wait for responses by invoking the workflow service using the same syntax as any other service. The BPEL process can also perform more complex operations such as updating, completing, renewing, routing, and escalating tasks.
After the BPEL process has assigned tasks to users, users can act on the tasks by using the appropriate applications. The applications communicate with the workflow service by using WSDL interfaces or another API (such as Java) to acquire the list of tasks for selected users, render the appropriate user interfaces, and return the results to the workflow service, which then forwards them to the BPEL process. User applications can also perform other tasks such as reassigning, escalating, routing, suspending, resuming, and withdrawing tasks. Finally, the workflow service may supports other communication channels such as email and SMS, as shown in the following figure:
In the coming years, it is likely that we will see further development of user interactions in business processes with the objective of standardizing the explicit inclusion of human tasks in BPEL processes. This standardization effort is called the BPEL4People specification, which was originally developed jointly by IBM and SAP.
The most important extensions introduced in BPEL4People are people activities and people links. A people activity is a new BPEL activity used to define user interactions, in other words, tasks that a user has to perform. For each people activity, the BPEL server must create work items and distribute them to users eligible to execute them. People activities can have input and output variables and can specify deadlines.
To specify the implementation of people activities, BPEL4People introduced tasks. Tasks specify actions that users must perform. Tasks can have descriptions, priorities, deadlines, and other properties. To assign tasks to users, we need a client application that provides a user interface and interacts with the tasks. The interface can query available tasks, claim and revoke them, and complete or fail them.
To associate people activities and their related tasks with users or groups of users, BPEL4People introduced people links. People links are somewhat similar to partner links -they associate users with one or more people activities. People links are usually associated with generic human roles such as process initiators, process stakeholders, owners, and administrators.
The actual users that are associated with people activities can be determined at design, deployment, or run time. BPEL4People anticipates the use of directories such as LDAP, (lightweight Directory Access Protocol) to select users. However, it does not define the query language used to select users. Rather, it foresees the use of LDAP filters, SQL, XQuery, or other methods.
SOA provides the technical architecture to develop end-to-end support for business processes. SOA achieves this objective by exposing an organization's IT assets as re-usable business services that can be composed into processes on the one hand, and can integrate and communicate more easily on the other hand.
From the bottom-up perspective, SOA is integration architecture. It provides technologies and approaches for the systematic integration of existing applications and the development of new solutions. With SOA, software architects develop a high-level integration architecture that uses common concepts to share data, information, and business logic between applications in a controlled, transactional manner using a service bus or other supporting technologies, such as rules engines, registries and repositories. SOA is based on typed communication with messages that confirm to common schemas. In new-generation SOA, business events have been introduced, that provide an alternative approach to the realization of one of the most important goals of SOA—loose coupling. Loose coupling is an approach where different software services and components share the lowest common denominator of dependencies. This makes application architecture more robust and resistant to changes. This will allow applications, components, and services to evolve and change with, or without minimal effects on other applications, components, and services.
SOA is also the architecture for designing, automating, and optimizing business processes. The objective of SOA is to provide end-to-end automation of business processes. Business processes in SOA are based on the composition of services and processes using programming-in-the-large technologies, most importantly BPEL. BPEL business processes enable fast development, and are flexible and easy to change and modify in the future, as shown in the following figure:
Another important feature of SOA is that it minimizes the semantic gap between process models and executable code. It achieves this objective by providing a common language to business analysts, IT architects, and developers. BPMN has become the new notation for business process modeling. An automatic round-trip mapping between BPMN and BPEL has also been developed. In this context, we should not forget the importance of re-use, which is the key to fast development of new solutions, and minimized testing efforts (due to re-use of existing artifacts), as shown in the following figure:
Companies that use SOA to develop end-to-end automation of business processes will benefit in terms of agility, flexibility, resilience, quality of service, better collaboration between business and IT, and better chances to develop new innovative business models.
Note
Innovation is the key objective of SOA and end-to-end business process automation. SOA will open up new opportunities for the development of new and innovative business models. Therefore, SOA will become a new key competency factor and an important tool for raising the competitive advantage of companies.
Let us look at some of the benefits of this approach.
SOA can improve the agility of the whole company by enabling the development and adaption of business processes quickly and efficiently using the composition of services, that is, through programming-in-the-large. As SOA services are designed for re-use and integration, the design, development, and test efforts are considerably reduced. SOA promotes re-use, which leads to increased standardization and compliance at the enterprise level. SOA also minimizes the business-to-IT gap, with the introduction of round-trip mapping between business process models (BPMN) and executable processes (BPEL).
SOA is concerned with the development of loosely-coupled information architecture, which shields business processes and services from changes, and functions independently of versions, locations, or technical details of applications systems. SOA also enables easier and less painful migration from legacy systems, consolidation of duplicated resources, master data management, multi-channel access to applications, and the flexibility to develop variants of processes from the same base. Loose coupling enables IT assets to develop and evolve without the limitations imposed by interdependencies and point-to-point integrations.
SOA introduces a new dimension to application development with a major consequence – that of better aligning business with IT. SOA raises the level of abstraction from technologies to business services. SOA introduces business vocabulary to the IT, which simplifies the connection between IT and business people, and enables them to better understand each other. Above all, SOA talks about applications in terms of business processes. Therefore, it does not require complex mapping of requirements to actual software representation. Further, SOA provides the ability to achieve two-way mapping between the business process model and the executable processes (and services). This guarantees better alignment between business and IT in the long run—even after several maintenance and upgrade cycles.
As SOA is related to BPM, it will encourage companies to think about business processes and achieve better collaboration between business analysts and IT. Business will discover that the SOA architecture is agile enough to adapt to requirements quickly, and IT will better understand the needs of the business. This will lead to business process optimizations, improvements, and to overall process excellence, which will have important impacts on the overall efficiency of the company.
SOA has become the new key competency factor across companies. It has achieved this by combining the business requirements with IT capabilities. SOA uses technologies, which enable modeling, execution, configuration, and adaptability of business processes using service composition. These technologies enable better alignment of business and IT, which improves the efficiency of IT, and enables improvement of IT services on one hand, and cost savings on the other. These cost savings can be used for development of new innovative services and products, which will improve the efficiency of the whole company.
SOA also allows easier outsourcing of services, and opens up opportunities for new business models in which the business processes of a company can be exposed to customers and suppliers to achieve much tighter integration with business partners.
This brings us to the overall picture of SOA architecture, which connects the following technologies:
BPMN for business process modeling
BPEL for business process execution
Services, which represent business logic at various levels of abstraction, and are the basic building blocks of SOA
The ESB for managing the communication between processes and services over the bus
The registry and repository for registering services, and processes for re-use and governance
The rules engine, which is a central place for definition of business rules
Human interaction using user tasks, which is used for human workflow in business processes
BAM, which is used to monitor activities and processes and to learn about process performance
In addition, SOA is typically related to the following:
A presentation layer, which could be a portal providing a unified user experience for all integrated applications
Security and identity management, together with single-sign-on, which becomes even more important with the introduction of SOA
The complete SOA architecture is shown in following the figure:
The technologies described above are part of SOA platforms, that are offered by major software vendors. Here are some major SOA vendors, which offer complete solutions:
Oracle SOA Suite, BPA Suite, BAM, and related products including BEA (recently acquired by Oracle), AquaLogic and WebLogic
IBM WebSphere Process Server, Business Modeler, Integration Developer, Process Monitor, ESB, Registry&Repository, and related products
Microsoft WCF (Windows Communication Foundation), WF (Workflow Foundation), BizTalk, and the forthcoming Microsoft Oslo
Software AG webMethods
SAP NetWeaver SOA Middleware
We can find several other companies in the market that offer separate products such as BPEL engines, ESBs, rules engines, and so on. We have not listed them here, as this information can be obtained on the Internet.
An alternative to commercial vendors is an open source implementations. SOA can be developed around open source software. One of the most popular is JBoss Enterprise SOA Platform. We can also use a mix of products such as the Glassfish application server, Open ESB, BPEL engine (such as ActiveBPEL, Apache Agila or Bexee), Drools rules engines, and many more.
In this chapter, we looked at the relation between SOA and business processes. We saw that SOA and BPM are closely related. SOA provides the technology platform for the implementation of business processes, and the development of applications that provide end-to-end support for business processes. SOA is an architecture that has introduced several important new concepts into application development. One of the most important concepts is the composition of services into business processes. With this, SOA has provided an architecture that is flexible enough to accommodate business needs related to agility, adaptability, along with other aspects related to the optimization of operations and improvement of business process efficiency.
We have seen that we have to look at SOA from three different perspectives: business, technical, and organizational. Only if we address SOA from all the three can we minimize the risks and maximize the benefits of SOA inception. The benefits are: improved flexibility, better alignment of IT and business, faster and simplified application development with reduced complexity, and most importantly, end-to-end automation of business processes, and a reduced semantic gap between business and IT. Risks are related to various organizational, technology, and business issues, which we also discussed in this chapter.
We learned that SOA is a long-term project connected with business processes and BPM. This is important because it means that we have to understand business processes in order to be able to use SOA for their implementation. Therefore, in the next chapter, we will look at how we should manage and model business processes for SOA.
