Beginning Your SOA Journey
Many organizations start their journey towards SOA through some sort of grass roots effort. Unfortunately, these efforts normally result in what's known as JBOS (Just a Bunch of Services). Typically, a project that had previously used some form of distributed component technology, such as Enterprise Java Beans, has now chosen to use XML or SOAP and HTTP, instead. The issue with this approach is that the service boundary that establishes the consumer and provider relationship really doesn't exist when one team is responsible for both the consumer and the provider.
Eventually, the organization will encounter a situation where the development of the service and development of the consumer takes place in a separate project. This could be due to there being more than one consumer, a B2B scenario where services are developed for consumption by partner companies, a large program that involves many independently managed projects, or simply a decision that the organization makes as it learns more about SOA. In our example, this was exactly the case. There was a program that encompassed three separate projects, two that involved development of service consumers, and one that handled the service development. The two consumers were the front-end for the auto insurance system and the front-end for the home insurance system. Spencer's project was responsible for creating a new service that provided an abstraction layer in front of the data systems for both applications.
Key Project Roles
The nice thing about projects and programs is that they have an explicit hierarchy. If a developer has a question or concern, they work with the project architect. The project architect may take things to the project manager, and the project manager may take things to the sponsor. If it's a program, then there's likely a hierarchy of architects and project managers, but everything bubbles its way up to the top. Everyone working on the project understands the objectives, the scope, the milestones, and the deadlines. This explicit hierarchy is the first, and often only, source of governance within the project. Within the project we have one piece of the governance puzzle: people. The challenge, however, is that the people only have authority within the project. If your SOA adoption efforts are broader than that single project or program, you'll likely run into problems.
In our Advasco example, Spencer ran into exactly this problem. Initially, Spencer only had to deal with project managers that were within the overall program. These project managers knew that the desired outcome was a shared, accurate, complete view of the customer, and it would be achieved through usage of the new service. As a result, they worked together with Spencer to ensure that outcome would be reached. When Spencer went outside of the program, however, his position of authority did not go with him. When he met with Ryan, he had no perceived or explicit authority. Even though the company had recognized a need to improve its image with its customers, the scope of that effort within IT was limited to the home and auto insurance areas. Therefore, for Ryan, service reuse was not on his list of desired outcomes, and regardless of how good Spencer made it sound, it was not something that he was willing to risk for the outcomes that he did desire.
The Service Contract
In this example, we clearly had a service provider, Spencer and his team, and two service consumers, the auto insurance application whose development efforts were managed by Jennifer and the home insurance application managed by Mark. A key aspect of this example is that these three efforts were independently managed, even though all being under a common program. A service should be independent of all consumers and this begins at the time that version one is developed, not at the time version one goes into production. In our example, imagine if the service development was under the management of either Mark's project or Jennifer's project. If a conflict arose, whose project would win out? Clearly, the project manager that oversees the service development effort has the upper hand, and will likely make decisions that will benefit their own project first. By separating out the service development as an independently managed effort, both of the two consumers are now equal, as they should be.
When we have the notion of a service consumer and a service provider, we need an explicit representation of the relationship between them, and that relationship is a service contract. This is no different than how we deal with services in the real world. If you hire a crew to replace the roof on your home, the first step is for you and the construction crew to sign a contract that governs the work. It provides governance by establishing policies. These include the hours that work will take place, the time in which the work will be done, the payment schedule and conditions, the behavior of the crew in the event of bad weather, and so on. In the world of SOA, the service contract is the collection of policies that govern the interaction between a service consumer and a service provider. That contract states the messages that will be exchanged, the URIs to be utilized, and more.
In this article, the initial focus was on two factors: the functional interface and the delivery schedule of the implementation. This is where most organizations start as the functional interface and a working implementation of that interface are clearly the minimum mandatory elements. It is no surprise that terminology like contract-first quickly sprung up as the varying technical approaches started to gain in popularity.
This article also addressed another key element of service contracts. In the real world, a contract is a binding agreement between two parties. Using our earlier roofing example, if the contractor replaces the roof on your house as well as the roof on your neighbor's house, he would have one contract with you and one contract with your neighbor. While you and your neighbor are both receiving the same service (a new roof) the terms and conditions around that service are likely to be different. You may both have been presented a standard contract to begin with, but from that point on, each one of you may have made your own adjustments or additions. The same approach needs to hold true for technology services. You may choose to expose a subset of operations to one consumer, while another consumer may have access to all operations.
In our example, Spencer's efforts initially fell short. He had initial conversations with Jennifer, but the only thing that came out of it was some agreement on when things had to go live. There was no discussion of the service interface, no discussion of the delivery schedule of milestone releases, or anything else. Given that this service was only going to be consumed internally, the development of the interface should have been a joint effort of both the consumer and provider. Spencer's team would bring domain knowledge from the provider's side, Jennifer's team would bring domain knowledge from the consumer's side, and together they would establish a service interface that was amenable to both. Instead, Spencer's team developed the initial service interface in a vacuum, creating something that may have met their needs, but did not meet the needs of the consumer. Meeting the needs of the consumer is the most important aspect of providing a service.
The second mistake that Spencer made was that he did not establish a formal definition of the handoffs that would be required between his team and Jennifer's team. In providing Jennifer's team an endpoint that could be used during development, he thought he was doing the right thing, but then when that endpoint changed out from underneath them, since, after all, it was under development, it had an impact on the trust between Jennifer's team and his team.
This particular situation can be a challenging one for many organizations, because the basic design of their environments often assumes that everything required for a project is under the control of the project team, and can therefore be promoted through the environments in lock-step. Now, when service consumers and the service provider are being developed according to their own timelines, instability can be introduced.
The appropriate way to handle this situation is to make explicit those policies that govern the interaction between the service consumers and the service provider during the design and development phase, rather than dealing with situations that arise on an ad hoc basis. The service provider has the responsibility for delivering a stable version of the service at various points throughout the project, and deploying it onto a stable platform that only changes according to the policies within the contracts enacted with the consumer teams.
For example, suppose both the service consumer and service provider are taking an iterative approach to the development of their solutions. In order to allow the service consumer adequate time for testing and feedback, the service provider may only promote a subset of their iteration builds to an integration environment for use by the service consumer. The service consumer would be required to provide feedback within a specified amount of time in order to have the fix included in a subsequent integration release. This is shown in the following figure:
In short, within a single project, iterative, agile development can certainly take precedence. Across projects, however, the handoffs should be formalized and explicitly specified as part of the service contract, especially when two or more consumers are involved.
Adding SOA to Traditional Project Governance
While the big change for the organization is learning how to manage the consumer-provider relationship, we can't forget about traditional project governance. Today, your organization may make use of architecture reviews, design reviews, code reviews, and operational readiness reviews as part of the software development process. These reviews already embody the three components of governance: people, policies, and process. These reviews more than likely involve resources from outside the project that the enterprise has positioned as authorities. An architecture review may involve enterprise architects, more senior architects, or other architectural peers. A design review may involve architects or senior developers. A code review may involve senior developer or other development peers. The operational readiness review probably involves members of the operational areas to ensure that a handoff to the support teams will be successful.
In order for these reviews to be successful, the policies that need to be followed for architecture, design, coding, and deployment need to be known to the project team. In the absence of documented policies that encourage the desired behavior, these reviews tend to be a show of power by the review team, where it is simply an exercise in trying to find something about the project to make the statement that they know better than the project team. Meanwhile, the project team plays a guessing game trying to determine what the review team wants to see, usually winding up wrong. In short, without documented policies, the review tends to be a lose-lose situation for all involved.
The process is the part that can vary. Some organizations choose to utilize a formal review process where an hour or two of time is scheduled with the reviewing body, the team prepares a formal presentation, and the review takes place. However, it doesn't need to be this heavyweight. Any of these reviews could also be done in a more informal manner, with a single meeting between a recognized authority for the review being done and the project architect or technical lead. If there is a formal technical hierarchy in the organization, the process may simply be part of the normal conversation that a project architect has with their architecture manager on a regular basis. Finally, an organization can even choose to have no review process, and simply trust that the decision makers on the project have awareness of the policies that must be followed.
So how does SOA change the current governance model for projects? Presuming your existing governance model is working, the only thing that SOA introduces is additional policies. If your existing governance model isn't working, consider making a change.
You now have a project that is building a new artifact, the service. While we discussed the importance of involving potential consumers in the definition of the service interface, the enterprise also has a role. Policies that are normally enforced by an external review board are typically associated with ensuring consistency across projects. When building services, the areas for consistency are:
- The technologies used for the service implementation
- The technologies used for communication between the consumer and the provider
- The representation of the information that is transferred between the consumers and the provider
Service Implementation Technologies
The first area that an enterprise may strive for consistency is in the technologies used to build services, also known as service platform technologies. There's a good chance that an organization may already have some standardization in this area, such as a single Java EE application server. Even if they do, there is still room for standardization.
At a minimum, the organization will need a general purpose application server and an associated development framework, such as a Java EE application server or Windows Server and the Microsoft .NET framework. Both of these platforms provide libraries for many different types of service communication technologies, as well as a robust library of open-source frameworks either as alternative or as extensions for other purposes. Depending on your organization, you may have one or many of these platforms. A general principle that organizations try to use is to not have two tools for the same job. That being said, if an organization has a federated IT department, whether due to past acquisitions, geographic needs, or other reasons, each of these separate IT groups may have their own standards.
In addition to the general purpose application server, another common service platform is the automated process platform, frequently associated with the use of BPEL technologies. This is a new breed of development platform tailored towards the orchestration of other services. Typically, it involves a graphical modeler, providing a drag-and-drop metaphor for connecting services in an orchestrated sequence, such as shown in the following figure:
Depending on the product chosen, it may include some out-of-the-box adapters for doing common activities, such as retrieving records from a relational database, publishing messages to an enterprise messaging systems, or sending an email message.
The recommended approach for determining the appropriate number of service platforms is to first determine the service types that an organization may need to provide. A service type is a class of service with a specific set of capabilities that may lend itself to specific technologies. Common types for which you may consider having specific platforms include:
- Composite services
- Automated (Orchestrated) processes
- Integration services
- Presentation services
- Management services
- Information services
- Content subscription services
- General business services
Composite services are, as the name implies, services that are built from other services. This typically involves combining the output of several services and combining it to be delivered to a consumer through a single service.
Automated processes, as was discussed earlier, are about the orchestration of a collection of services to perform a higher level function. Frequently, composite services can be thought of as a subset of the overall space of automated processes.
Integration services are services that are geared towards communication with third-party systems, such as SAP and Oracle. These are frequently associated with Enterprise Application Integration, or EAI, technologies. While EAI, as a category, has lost favor, there is still plenty of need for integration technologies in communicating with these complex application platforms.
Presentation services are services that provide user interface functionality, whether simply content in a presentation-friendly format, or fully featured presentation components suitable for composition in a portal context.
Management services are services that are geared towards the management of IT systems, typically leveraging technologies like SNMP (Simple Network Management Protocol) and JMX (Java Management Extensions) for communication.
Information services are services that are intended for data access and manipulation. These are typically marketed as Data Service Platforms or Enterprise Information Integration products.
Content subscription services, such as news feeds, are intended to provide content to consumers on a subscription basis using technologies such as RSS (Really Simple Syndication) and Atom.
General business services is the final service type, and is intended to be the catch-all for other services that don't cleanly fit into any of the previous categories. These would normally be implemented using a general purpose application server platform.
Service Communication Technologies
The next set of technologies that an organization may standardize on is service communication technologies. While much has been written on how adopting web services is not the same as adopting SOA, one or more technologies must still be chosen for service communication. This may vary by the service type involved. While in the past, organizations may have leveraged distributed component technologies such as CORBA or Enterprise Java Beans, today most organizations are leveraging either Web Services technologies, some less-restrictive form rooted in XML and HTTP (also known as POX—Plain Old XML), or REST.
In the past, a very important aspect of choosing a communication technology was interoperability. Most distributed communication technologies were platform-specific, focused on communication within a platform rather than across platforms. This posed a significant problem when an enterprise had multiple platforms in the environment, which today is the norm for any large organization. This is a challenge that is still being fought today, with each technology getting closer and closer to achieving out-of-the-box interoperability in a very heterogeneous environment.
In the case of web services, there is an industry organization whose specific charter is to establish best practices for WS-I (Web Services Interoperability). As of 2008, there are well over 60 specifications that are related to web services, so determining the right way for using each of them can be a challenging exercise. WS-I has defined a series of profiles (starting with the Basic Profile) that provide best practices for how to use various subsets of the web services specifications. As a best practice on its own, if you choose to leverage web services technologies as part of your technology portfolio for SOA adoption, you should ensure that all platforms involved are compliant with the WS-I Basic Profile.
Another factor that must be considered when choosing service communication technologies is security. At the time of writing, security is still a large differentiator between some of the popular approaches today, and also a source of potential interoperability problems, even when there are no functional interoperability issues. For example, within web services, the WS-Security specification defines a framework for placing security tokens within a SOAP message. Building on that framework, there are now several standard profiles for common security tokens, including a basic username and password combination, X.509 certificates, SAML (Security Assertion Markup Language) assertions, and Kerberos tickets. So, while two parties may both support web services, if a provider decides to only accept security tokens via the Kerberos profile, while the consumer is only capable of sending SAML assertions, we still may have interoperability issues.
Overall, organizations are in a far better state with the options available to them now than ten years ago. While there are still interoperability challenges, most of them can be overcome through a variety of means. The role of governance in this is to ensure that communication technologies are used consistently and appropriately, not to ensure that the organization only chooses one communication technology.
Service Interface Specification
Perhaps the most important aspect of enterprise governance on service technologies is in the interface specification. Regardless of whether an organization leverages web services, REST, XML over HTTP, Enterprise Java Beans, or virtually any other communication technology, information needs to be transferred between a service consumer and a service provider. While choosing a single communication technology can resolve interoperability problems when transferring information between a single consumer and single provider, there is still risk for inconsistency in the information when a service consumer talks with multiple service providers.
In our example, Spencer had the challenge of providing a Customer service that met the needs of both the auto insurance system and the home insurance system. This proved to be a challenge because each of these systems had their own definition of what a Customer is. In designing the original interface, Spencer's team focused inward, basing their decisions on the service development team's understanding of the service implementation. The path of least resistance for this team was to simply base the service interface on that knowledge, working with people within their own project. In other words, the implementation model of the information bubbled its way out to the service interface as shown in the following figure:
This is a bottom-up approach, where the representation of the information begins with how it is stored in the information store (for example, relational database) and is expressed in a storage model. The information is used within the service implementation in some form, that representation is known as the implementation model. Finally, that information is exposed via the service interface, as the interface model. The challenge that is faced is that the easiest path for a development team is to simply let these models bubble up. It took many years for the industry to realize that a model that was well-suited for storing information (for example, a relational data model) may not be well-suited for processing information, and thus a new model was needed (for example, an object model). Likewise, it is also true that the implementation model is not well-suited for transmitting information to another processing node, because their processing needs (and possibly their platform for implementation) is going to be different.
An alternative approach is to take a very consumer-oriented view. Instead of allowing the internal service implementation to drive the interface model, the model is determined through analysis of the service consumers, as shown in the following figure:
At first glance, this may seem like the more service-oriented way of approaching the interface design, since after all, services are about consumption. Unfortunately, not all consumers are created equal. In our example, Spencer had a challenge in meeting the needs of both Mark's team and Jennifer's team because each of them had their own model, and there were inconsistencies between them.
An approach that Spencer could have taken would have been to have his team meet with Mark's team and Jennifer's team to balance each of their respective interests in coming up with a service interface in which all parties agree. While this may have prevented some of Spencer's problems, it still is not without its challenges. Suppose that a few months later, Advasco chooses to partner with a new company for billing services. Clearly, these billing services need Customer information, but it is likely that the new company has their own definition of what a Customer is. So, the situation we now have is shown in the following figure:
In this situation, we have two service consumers, 1 and 2, and two service providers, A and B. Both services A and B require some representation of Customer information as part of their service interface. Unfortunately, if these representations are inconsistent, the burden of transforming and mapping between them fall back to each individual consumer. Every consumer that needs to use both services A and B winds up having to implement that transformation and mapping within their own code. As the number of services that a particular consumer interacts with grows, the situation gets more and more complicated.
Using a Canonical Model
The approach to solving this dilemma is to embrace the notion of a canonical model. A canonical model is an information model that is independent from any particular service consumer or service provider. When adopting a canonical model, service interfaces are defined according to it. In this way, a consumer or a provider only needs to know how to map to the canonical model, rather than having to know how to map to any of the possible interface models of the independent service providers. The picture now looks like this:
The hidden challenge behind this approach is the actual development of the canonical model itself. Performing an analysis of all possible consumers and all possible providers that deal with a particular piece of information, and then trying to find common ground across all of them can be a painful and time-consuming exercise. Many organizations choose to avoid this altogether and base their canonical model on an industry standard for their line of business. In some cases, this is exactly the right thing to do, especially if the services being designed are going to be exposed externally.
While the construction of a canonical model is beyond the scope of this article, there is no doubt that there is a need for an information model independent of any consumer or provider that establishes policies for message interfaces. In the absence of such a model, the path of least resistance is to push the implementation models of individual consumers or provider out into service interfaces, which leads to a far worse situation of point-to-point integrations, and many-to-many mappings between them, contained within each service consumer. At the same time, if the development of an enterprise model is a painful and time-consuming process that frequently ends in failure, what is an organization to do?
To establish some best practices, we must remember that the goal of governance here is to ensure consistency among service interface definitions, not to ensure that the enterprise creates a canonical model. Consistency can be achieved following these best practices:
- Consult all potential service consumers: In our example, Spencer eventually consulted the two consumers that were associated with his program, but as we found out later in the article, there were other potential customers, such as Ryan, as well. Some human judgment is always needed, as a project can quickly get into analysis paralysis.
- Consult service owners that also use similar information: In addition to consulting potential service consumers, you should also contact other service providers that may also utilize the same information. With good use of a registry or repository tool, other service providers should be easily found.
- In the absence of an existing standard, leverage an industry standard if one exists: Many times, groups with vested interests have a difficult time reaching consensus and as a result, they bring in a professional mediator. In the case of service interfaces, an industry standard can act as the external mediator, providing a model that is independent of any of the models at hand, yet having the backing of an industry coalition that focuses in the particular domain at hand.
- If the services are going to be exposed externally, leverage an industry standard if one exists: It is difficult enough to get internal groups to agree on messaging standards, doing so with other organizations can be even more of a challenge. If an organization pushes its own model on its partners, it can be perceived as domineering. It also runs the risk of alienating partners if other competitors offer similar services based on standards. If a standard exists, it should be leveraged. If one doesn't exist, an organization should think carefully about driving a process to establish one rather than trying to make their own model the de facto standard.
- Define your plan for change: Regardless of how many potential consumers and providers are consulted, it is likely that, at some point in the future, your interface will change. Rather than starting with the assumption that your interface will be defined once and never be modified, start with the assumption that it will change, and walk through the steps necessary to ensure that the change happens in a smoothly managed fashion. Decide how existing consumers will be notified, how long previous versions will be supported, and at what timeframes the new interfaces will get rolled out. We discussed how the service contract needs to explicitly specify the interactions between service consumer and service provider during service development, but it must also specify the interactions that will occur after the development of the initial version, including the development of subsequent versions.
Web Services, POX over HTTP, and REST
Any conversation about service interface technologies would not be complete without a discussion about the differences between the three most popular interface technologies today, which are web services, POX over HTTP, and REST. We've mentioned web services quite a bit in this article, so let's begin there. Web services provide the most explicit service interface of the three approaches, through the use of a WSDL (Web Services Description Language) file. The WSDL file describes how to interact with a service by specifying the URLs to which messages should be directed, as well as schemas for the messages themselves, using XML Schema. A service is comprised of one or more operations, and each operation can specify messages associated with its usage including a request message, a response message, and fault messages. At least from a functional perspective, the message formats are very explicit.
REST, which many people mistakenly confuse with POX, stands for Representational State Transfer, and is based on the work of Dr. Roy Fielding. REST views the world not as a collection of services, but rather a collection of resources. Each resource is uniquely addressable via a URI, and all resources can be interacted with according to a universal interface. When using it with HTTP, the universal interface is generally accepted as the HTTP operation of POST, GET, PUT, and DELETE. When using REST, a query for information would always be done via a HTTP GET operation, while when using POX over HTTP, it may be tunneled through HTTP POST, and when using Web Services, it is most certainly tunneled through HTTP POST. The key difference for this discussion, however, is that REST does not make the information model explicitly known at design time via the use of an interface specification, like a WSDL file.
Between these three approaches, the single biggest differentiator from the point of view of interface technologies is the use of an explicit interface definition that includes an information model with Web Services and the distinct absence of one for either POX or REST. The one common truth, however, is that all three of these technologies can be used to share information between systems. If the information being shared is done so using different representations, it places more complexity with the consumer to deal with those inconsistencies. Regardless of whether the information model is explicitly specified or not, there is still a role for governance in ensuring that appropriate consistency is maintained and the associated complexity in maintaining that consistency is appropriately managed.
In this article, we have covered key project roles, Service Contract, adding SOA to Traditional Project Governance, Service Communication Technologies, WS-I Compliance, Service Interface Specification, Web Services, POX over HTTP, and REST.
Beyond project governance, the role of enterprise governance was introduced and how the desired behavior for the organization is consistency. While a single standard is one way of achieving consistency, it is not the only way, nor may it be the right way, depending on the structure of your organization. Adopting SOA does not establish the need for enterprise governance; it simply inserts new policies into the mix of things that must be governed. The three key policy areas are service implementation technologies, which are the platforms and frameworks used to build services, service communication technologies, which are the technologies used to allow communication between service consumers and service providers, and service interface technologies, which are the information models that are used within service messages. Of these three, service interface technologies pose the most challenging problem, as inconsistent information models create point-to-point integration scenarios and increase the amount of complexity in service consumers. Adopting a canonical information model, or allowing one to grow organically, can manage this complexity effectively.
If you have read this article you may be interested to view :