This chapter gives an overview of the BPM technology and its core component, the workflow engine. We will analyze the different types of BPMS and the flexibility they provide in terms of workgroup collaboration and system orchestration. A brief outline of the topics covered in this chapter is as follows:
Business Process Orientation
What is a BPMS?
Types of BPM Systems
Components of a BPM solution
Open-source BPMS
Today's marketplace is more global and competitive than ever. Big players are trying to attract new clients along with maintaining the existing ones (thereby making more profit for shareholders), and smaller players are entering the markets with innovative products. To add to this, customers are demanding better prices and service and governments are continuously changing regulations.
Businesses redefine themselves in order to adapt to this environment or to create new market niches for exploitation. Technology plays an important role in realizing these changes and adaptations. With the appropriate technology and tools, businesses can reduce costs, raise margins, and make the most of the existing information to understand the clients better and thus create new markets based on this data.
Businesses are focusing on internal efficiency, which aims at realizing the goals of making profit, reducing costs, creating and maintaining customers, and negotiating with suppliers in faster, cheaper, and improved ways.
With the appropriate use of technology, businesses can benefit in many ways such as greater revenue, bigger margins, automated processes, improved decision making, and so on.
Every business has a set of connected activities and functions driven by business rules such as sales, account receivables, R & D, and so on. The daily operation of such essential processes costs a big chunk of revenue from shareholders. Business Process Management (BPM) technology promises continuous enhancement of the business processes.
To diminish these costs and maximize the profit generated, each business process must be efficient and smooth. The more visible these processes are, the more we can modify them to align with the organizational strategy.
Every corporate strategy and company mission wants to treat the customer in the best possible way it can, but are the customer service processes aligned with the strategy? Is the bureaucracy of my service as minimal as possible? Are my business processes impeding the efforts of employees to excel at customer support? Am I generating long-lasting bonds between my customers and my company?
These questions can only be answered by knowing the processes, the actors, the context, etc. Today's business executives are held accountable for the performance of the processes they manage. Accountability needs objective measures, and how can we measure what we don't know?
In the early nineties, a trend called Business Process Reengineering (BPR) emerged as a panacea to make business process more efficient. A common buzzword of those days was downsizing. BPR failed its goal; downsizing was associated with layoffs and more work for the employees and not with the ultimate efficiency goal. Reengineering proposed that the existing processes be discarded and new ones be created from scratch. This was a big and ambitious approach, a big bang change for the organization.
At the beginning of 2000, the term BPM was coined to refer to a new management approach, a holistic way to produce an agile and adaptive organization based on the enhancement of business processes. BPM unlike BPR is based in an incremental approach, improving the process step by step, with enough evidence to support this move. BPM is an old approach to manage processes, which has been renewed by using technology as the improvement tool.
With the advent of IT as a strategic partner and change enabler of the business, Business Process Management Systems (BPMS) appeared to support the BPM way of doing things.
Both BPR and BPM are based on the rational analysis of a business process from every possible dimension, destructuring it into its parts and relations, stripping the unproductive activities. BPR didn't take technology tools into account, but BPM advocates technology and feedback as the drivers of business process efficiency. BPM tries to automate as many activities as possible. Iterating as many times as needed, obtaining feedback, and making incremental changes to the process is the BPM mantra.
Efficiency is not the only goal of BPM. Companies are trying to gain a competitive edge over their competitors by perfecting their processes. A better process can win a customer; for example, in customer service a better process should reduce time to market and eliminate unnecessary internal bureaucracy.
The first BPM process improvement efforts started a few years ago with the replacement of the BPR methodology, forcing companies to rethink their strategy in operational efficiency and placing processes in a strategic place.
The first effect was creating a Business Process Improvement (BPI) area in every process-led organization in order to take care of process design, enhancement, and monitoring. Technology is a key change enabler in BPI, so it's not uncommon to see IT departments very involved in BPI projects.
BPM efforts generally take more than an iteration to get the process right. So you must analyze the existing process, add the improvement, and then take feedback from the real‑world operation. Then, you go back to the scratchboard, analyze the feedback (both numeric and from people), and add some other improvements until Key Process Indicators (KPI) are met. KPI are metrics that ensure that a process is running in its appropriate level of service, and are used to measure its Service Level Agreements (SLA).
Business functions can agree a service level with other business functions within the same company or with business partners and customers. The goal of BPM is to make the process exceed or maintain its current SLA.
Every effort to change a business process must take people into the equation. People have a hard time adapting to new roles, functions, and procedures and can really subvert the effort of BPM. BPR considered people as numbers, replaceable material, and this was a key factor for its failure.
Today's marketplace is more global and competitive than ever. Big players are trying to attract new clients along with maintaining the existing ones (thereby making more profit for shareholders), and smaller players are entering the markets with innovative products. To add to this, customers are demanding better prices and service and governments are continuously changing regulations.
Businesses redefine themselves in order to adapt to this environment or to create new market niches for exploitation. Technology plays an important role in realizing these changes and adaptations. With the appropriate technology and tools, businesses can reduce costs, raise margins, and make the most of the existing information to understand the clients better and thus create new markets based on this data.
Businesses are focusing on internal efficiency, which aims at realizing the goals of making profit, reducing costs, creating and maintaining customers, and negotiating with suppliers in faster, cheaper, and improved ways.
With the appropriate use of technology, businesses can benefit in many ways such as greater revenue, bigger margins, automated processes, improved decision making, and so on.
Every business has a set of connected activities and functions driven by business rules such as sales, account receivables, R & D, and so on. The daily operation of such essential processes costs a big chunk of revenue from shareholders. Business Process Management (BPM) technology promises continuous enhancement of the business processes.
To diminish these costs and maximize the profit generated, each business process must be efficient and smooth. The more visible these processes are, the more we can modify them to align with the organizational strategy.
Every corporate strategy and company mission wants to treat the customer in the best possible way it can, but are the customer service processes aligned with the strategy? Is the bureaucracy of my service as minimal as possible? Are my business processes impeding the efforts of employees to excel at customer support? Am I generating long-lasting bonds between my customers and my company?
These questions can only be answered by knowing the processes, the actors, the context, etc. Today's business executives are held accountable for the performance of the processes they manage. Accountability needs objective measures, and how can we measure what we don't know?
In the early nineties, a trend called Business Process Reengineering (BPR) emerged as a panacea to make business process more efficient. A common buzzword of those days was downsizing. BPR failed its goal; downsizing was associated with layoffs and more work for the employees and not with the ultimate efficiency goal. Reengineering proposed that the existing processes be discarded and new ones be created from scratch. This was a big and ambitious approach, a big bang change for the organization.
At the beginning of 2000, the term BPM was coined to refer to a new management approach, a holistic way to produce an agile and adaptive organization based on the enhancement of business processes. BPM unlike BPR is based in an incremental approach, improving the process step by step, with enough evidence to support this move. BPM is an old approach to manage processes, which has been renewed by using technology as the improvement tool.
With the advent of IT as a strategic partner and change enabler of the business, Business Process Management Systems (BPMS) appeared to support the BPM way of doing things.
Both BPR and BPM are based on the rational analysis of a business process from every possible dimension, destructuring it into its parts and relations, stripping the unproductive activities. BPR didn't take technology tools into account, but BPM advocates technology and feedback as the drivers of business process efficiency. BPM tries to automate as many activities as possible. Iterating as many times as needed, obtaining feedback, and making incremental changes to the process is the BPM mantra.
Efficiency is not the only goal of BPM. Companies are trying to gain a competitive edge over their competitors by perfecting their processes. A better process can win a customer; for example, in customer service a better process should reduce time to market and eliminate unnecessary internal bureaucracy.
The first BPM process improvement efforts started a few years ago with the replacement of the BPR methodology, forcing companies to rethink their strategy in operational efficiency and placing processes in a strategic place.
The first effect was creating a Business Process Improvement (BPI) area in every process-led organization in order to take care of process design, enhancement, and monitoring. Technology is a key change enabler in BPI, so it's not uncommon to see IT departments very involved in BPI projects.
BPM efforts generally take more than an iteration to get the process right. So you must analyze the existing process, add the improvement, and then take feedback from the real‑world operation. Then, you go back to the scratchboard, analyze the feedback (both numeric and from people), and add some other improvements until Key Process Indicators (KPI) are met. KPI are metrics that ensure that a process is running in its appropriate level of service, and are used to measure its Service Level Agreements (SLA).
Business functions can agree a service level with other business functions within the same company or with business partners and customers. The goal of BPM is to make the process exceed or maintain its current SLA.
Every effort to change a business process must take people into the equation. People have a hard time adapting to new roles, functions, and procedures and can really subvert the effort of BPM. BPR considered people as numbers, replaceable material, and this was a key factor for its failure.
Business Process Management Systems (BPMS) technology is a tool that implements BPM in the enterprise. BPMS suites are designed to model, execute, and optimize business-process models, helping BPM people and executive-level managers to take the necessary steps for process improvement.
In the BPMS world, modeling, executing, and optimizing a business process is a continuous cycle and is therefore represented by a circle. The following figure is called the BPM lifecycle. It depicts the sequence of steps followed during the implementation of BPM in a process.
The BPM lifecycle begins by modeling the business process (taking an existing process or starting from scratch), followed by testing it, deploying it, and finally monitoring its execution in a production environment.
The first step is process modeling. It includes a phase of analysis if the modeling is being done for an existing process and if the modeling is being done for a new process, then it includes a design phase. Modeling is usually done by a business analyst who identifies and connects the basic building blocks of the process such as activities, roles, data, and so on. The next figure shows a business process modeling environment. It corresponds to OSDesigner, the business process modeler included in OSWorkflow.
As shown in the figure, a process is being created visually, by specifying the steps and the transition among them.
The testing phase consists of two steps, namely, validation and verification. Validation asserts correctness of the solution while verification checks conformity to the requirements. These activities include several debug iterations and test runs in a development or testing environment.
Once the process model is known to be correct, the analyst or programmer deploys the model in a process engine. This engine parses and identifies the model, and then executes the instructions and actions associated with it. The next step is monitoring the process in place.
During monitoring, we spot the bottlenecks, superfluous steps, and possible automation activities. Monitoring the process is usually a visual activity, through the use of a Business Activity Monitoring (BAM) dashboard. The BAM dashboard, also known as the BAM console, is a visual aggregation of business process information, usually real‑time information. The figure next page shows a typical BAM dashboard, which includes charts and gauges to display the different metrics of a business process.
In the meantime, we must take care of unforeseen environmental changes, like laws and regulations that have an impact on the process and the company. Regulations usually demand businesses to have a very good visibility of their processes, as well as their inputs and outputs.
The optimization can be done manually from the feedback obtained in production or can be simulated from artificial or historical load. The optimization must be modeled again, restarting the BPM lifecycle. The BPM cycle can be repeated until the business process performs like a fine-tuned machine.
BPMS also adds traceability and auditing to a business process. When the BPMS workflow engine executes an activity it registers a trace or audit log. This log commonly includes context data, such as the current user, the process instance ID, an activity start and end timestamp, and other useful data.
Traceability supports two goals—operational and strategic. The operational goal helps users in searching for information about the process and exposes the current activity and state of the system. The strategic goal enables the users to know where the bottleneck of a process is, and where they must tune the process for maximum efficiency. The following screenshot shows sample traceability data from a business process.
The traceability data in the screenshot shows the transition from step to step, the user that caused the transition, the time, and the current status, as well as other context information.
Traceability data is a goldmine for Business Intelligence as it includes a lot of hidden patterns about usage and operation of the process.
Through the use of data-mining software, you can uncover these patterns to understand your business, customers, and suppliers in a better way.
Every business process has two types of tasks—manual and automated. Manual tasks require essential intervention of a human being, whereas automated tasks are done by computers or machines. A number of BPMS tools are suited to the human aspect, some support only the automated level, and others can implement end-to-end processes without hassle. These are commonly called people to people, system to system, and people to system BPMS.
First, we will talk about the system to system BPMS, so let's see how they were born in order to understand them well.
Some BPMS suites are descendants of Enterprise Application Integration (EAI) tools. EAI began in the mid-nineties to break the barriers placed in the way of integration of legacy systems. Legacy systems were separated into vertical silos and the integration of information and functionality was difficult, costly, and error prone.
Every application that needed to share or gain information from another system had to program a custom interface or connector for integration, thereby taking away time, money, and most importantly agility and flexibility to change.
EAI alleviated this problem by using proprietary connectors to legacy systems and having a central broker of information. This central broker had the ability to talk to several heterogeneous legacy systems (such as ERP, CRM, custom applications, and so on) and made the overall integration process much easier.
Integration became more affordable and faster, but the need for custom connectors made the solution expensive and provider dependent. These custom connectors used the proprietary language of the systems they were integrating, which made them expensive. There was no universal integration protocol to talk to every partner system.
The answer to a universal integration protocol came from standards like HTTP and XML. XML allows applications to share information without taking lower-level details such as the architecture of each system into account.
HTTP, originally designed for web navigation purposes, fit the bill of common transport. XML and HTTP, when used in combination to share information and functionality are called Web Services. The following figure shows an EAI broker sending and receiving information from different applications initially not suited to talk to each other.
The use of Web Services technology by all applications to share information and functionality with each other has made the integration process much easier and standard based. This has brought down the integration costs and allowed companies to embrace business changes faster.
An architecture based on the integration of services is called a Service-Oriented Architecture (SOA).
Getting back to the different tasks of a business process, the automated activities are usually scattered across several systems, some core, some satellite, some legacy, and some new ones. The use of several disparate user interfaces and applications by an end user to perform a task such as an airline reservation is prone to error, lacks currency of the information, has a high training cost, and causes loses in productivity.
The main concept of system orchestration is to facilitate the business activity of the end user by eliminating the need for several separate information systems to realize the business process.
The orchestration systems query each system to gather information, combine this information, and feed it to other systems. So, to make an airline reservation, the airline clerk uses three to four systems. With orchestration this is reduced to a unique application. This scenario is very common in big companies where legacy systems are responsible for some tasks.
This has several benefits such as better information accuracy, reducing the information lag, and abstracting the end users from needing to know which systems have the functionality required. The following figure shows a sample orchestrator dynamic interaction between information systems.
The orchestration enables us to have higher-level services and composite applications.
Higher-level services use existing infrastructure such as a service for giving customer details and another for customer credit details, and combine the information into one service.
Composite applications allow the user to use only one user interface to query several sources of information and services without knowing the rest of the applications.
Some BPM tools implement only the system orchestrator component and talk only the Web Services standard, while others also have custom connectors to share information with legacy systems.
System orchestration can be done with systems inside an enterprise or across industry partners or in companies using heterogeneous applications. The OSWorkflow workflow engine can be used as a system orchestrator to automate system interactions. We will talk about this in Chapter 7.
We have covered the automated part of a business process. Now, let's take a look at the manual tasks requiring frequent or mandatory human intervention.
In our day-to-day life, we often find ourselves coordinating activities with other coworkers. Business processes tend to require a lot of coordination and organization of people's work, and it grows exponentially with the number of people involved. It gets even more complex with a broad geographic distribution as is the case with multinational companies.
The successful completion of a business process depends on the coordination of the people.
BPMS technology can facilitate workgroup collaboration through sequencing of activities, reminders, alerts, escalations, automatic approvals, etc. This automates the operational and tedious task of coordination and enables collaboration between different company departments.
For human interaction, people-to-people BPMS suites provide a user interface (web, fat client, or email) for the users to interact with the business processes and collaborate with each other. Collaboration is sharing knowledge about a process or a work item, increasing productivity, and increasing communication between coworkers.
This user interface guides the user through the process, manages the user roles and to-do tasks, gives search capabilities, and other useful features.
The usability of this user interface is a critical success factor in the implementation of a human-oriented BPMS. The following figure shows a sample BPMS user interface suited for human interaction.
An important thing to notice from the screenshot is the user-friendliness of the interface. This interface orders the processes into folders with user-defined criteria and serves as a unique front end for all business processes. This means that the user interacts with all the business processes through this GUI thereby making the experience more consistent. Moreover, it is easier to use and so significantly less user training is needed.
After knowing the different types of BPMS, you'll wonder about the components that make up a BPMS. The main components of a full-fledged BPM solution are as follows:
Process Modeler
Workflow Engine
Business Rule Engine
Graphical User interface (for human-oriented BPMS)
Integration connectors (for system-oriented BPMS)
BAM and BI tools
The following figure shows the layers of the solution with the workflow engine and the business rule engine as the main components on which the others are based.
The Workflow Management Coalition (WfMC) is a non-profit organization constituted by users and vendors, dedicated to disseminating best practices about Workflow technology. The WfMC recommends the following model to architect workflow solutions:
As shown in the figure, the WfMC specifies the Workflow Engine (as the Workflow Enactment Services and API), the Modeler (implicitly in the Process Definition), the BAM dashboard (as the Administration and Monitoring Tools), the GUI (in the client application block), and the integration connectors (via the Invoked Applications).
Let's examine each component in detail. The process modeler found in both human‑oriented and system-oriented BPMS is usually a visual application that can be used by a business analyst and optionally by an end user. The modeler allows the user to define the process and describe the activities, decisions, rules, and so on.
The output generated by the modeler is a process definition that is executable by the workflow engine. The process definition is the set of activities, decisions, roles, etc. and the relationships between them.
The workflow engine is the core component of a BPM solution. It's in charge of initiating, maintaining state, and transitioning the business process, while managing security and roles.
The workflow engine takes the process definition as an input, executes it, and while the process is alive, it stores the process data into a traceability and auditing log. Additionally, it provides both synchronous and asynchronous execution of tasks. These tasks can be internal tasks such as state transition and external tasks such as integration connector invocations.
The workflow engine acts as a system orchestrator in system-oriented BPMS.
Rules for business processes are another facet of BPM solutions. Most companies have complex business rules in every process scattered through several legacy systems and these rules change frequently due to internal and external factors.
Efforts must be made to permit these rules to change rapidly, and be modified by anyone without technical skills, such as an end user. Externalization, definition, administration, and traceability of these rules are common features of rules engines.
The workflow engine calls business rules from inside the business process. Sometimes the business rules are embedded in the process definition and sometimes they are placed in an external application. There's a third scenario when the rules reside inside the business rule engine; this is the best place to foster reusability and uniqueness of the business rules.
For human-oriented BPMS, the critical component is the GUI. The GUI is the view of the business process allowing the users to interact with other components. Usability of this GUI is critical to the success of a BPM solution implementation.
On the other hand, integration connectors are a standard feature for system orchestration and EAI-oriented BPMS. These connectors handle the connection, mapping, chatting, validation, and error checking of interfaces to new and legacy systems. It's common to see an integration engine managing the connection pipeline, the input of a system is the output of a previous system, with its data mapped and transformed.
For the BPM circle to be fulfilled, users need to monitor the process, analyze it, and get feedback for improvement. BAM and BI tools enable real-time monitoring, alert activation, reporting, and statistics collection for the process.
BI allows the BPMS to exploit historical and real-time data; some suites even give forecasts from historical data. The BI component analyzes the traceability and auditing database to discover bottlenecks, gather statistics, and optimize the process.
For more information about the WfMC visit http://www.wfmc.org/.
In the first section of the chapter, we saw the BPM lifecycle—the usual methodology to implement BPM in a single process or whole enterprise. The BPMS components align with this methodology by giving a tool for each part of the cycle. The next figure shows the mapping between components and the BPM lifecycle.
The modeling phase has the process modeler, the test and deployment (execution) phases uses the workflow engine, and finally the monitoring (optimization) phase is supported by the BAM.
This book discusses creating a complete open-source BPM solution stack using OSWorkflow, Pentaho, Quartz, JBoss Rules, and Esper, which are mature open-source solutions built with 100% Java code. The following figure shows the layers of our solution using open-source products.
OSWorkflow covers the process modeler and workflow engine complemented by Quartz to make asynchronous tasks possible. JBoss Rules is the rule engine used to easily define and centralize rules. Pentaho and Esper make up the BAM. Pentaho deals with the reporting and analysis and Esper with active event notifications enabling proactive monitoring.
This chapter serves as an introduction to BPM technology, its goals and vision, exposing much of its history and terminologies. With this background, you can add value in architecting and developing enterprise BPMS or simple workflows.
In the next chapters we will cover in detail the open-source stack of products used to build completely free BPM solutions.
This book has been broken into eight chapters, with the first one introducing the BPM technology and its context, chapters two to four covering the OSWorkflow concepts from basic to advanced uses, and chapters five to eight showing how to integrate different open‑source products to enhance the OSWorkflow functionality. These products are JBoss Rules, OpenSymphony's Quartz, Codehaus's Esper, and finally the Pentaho reporting engine.
Each chapter will present a part of the BPMS solution.
