As a part of informative conversation, we have Adil — IT Manager for Home Insurance Systems, Elena — Chief Architect, Jennifer — Project Manager for Auto Insurance Systems, Mark — Project Manager for Home Insurance Systems and Service Manager for Customer Information Service, Mike — IT Manager for Insurance Products, Raj — Technical Lead and Member of SOA Center of Excellence, Ramesh — Solution Architect for Annuity Systems, Spencer — Member of Enterprise Architecture Team and finally Alexandra — Spencer's wife as the key characters.
Every organization's journey to SOA adoption must begin somewhere. Some organizations may take a very top-down approach based upon direction from the Chief Information Officer (CIO) or other senior IT leader, while other organizations may begin with a grass-roots effort from within the IT organization, often times during a single project.
Beginning the SOA Journey
Spencer walked in through the main doors of Advasco, a leading financial conglomerate, on Monday morning knowing it was going to be a busy day. He was part of the Enterprise Architecture team at Advasco, and immediately headed for his weekly meeting with his boss, Elena, the Chief Architect. "Come on in, Spencer," she said. "As you know, we've been given a big challenge over the next few months."
Late last week, the head of the sales and marketing for the insurance division announced that they needed to improve the way they interacted with their customers. Advasco began as a typical financial services company, but had recently expanded into the insurance area through acquisition. They began by acquiring a company that provided homeowner's policies in several Midwest states. Over the next few years, Advasco had acquired several other regional insurance companies. This resulted in an increase in the number of insurance products that it offered, as well as turning Advasco into a nationwide provider. Unfortunately, Advasco was struggling to increase the number of insurance products per customer. Analysis of the situation had determined that while the sales staff of the original organizations had been combined, each of the different insurance products relied on different applications for customer management. As a result, it was far more difficult for the sales agents to know what insurance products any given customer had. After discussing it with Mike, the IT manager supporting insurance products, IT was given the task of providing the sales agents and marketing staff for the insurance division with a single view of the customer.
Elena then said to him, "I had a meeting with Mike to discuss their new initiative to provide a single view of the customer to the sales agents and marketing staff. While he has some great developers in his area, he asked me if Enterprise Architecture could provide some architectural guidance to their effort. Given the excellent integration work you did when Advasco acquired our first company in the insurance area, I think you'd do a great job on this effort."
"It certainly sounds like an exciting project," said Spencer, "I'd love to help out. Just last week, I met with my own insurance agent and saw first hand the frustration he had in trying to see the different insurance products that I have for my family."
"I'm glad to hear that," said Elena, "I'd like you to meet with Mike today and start coming up with an architecture for the effort. I know you've been reading about SOA. Perhaps this effort can serve as a good pilot for some of the techniques."
Spencer agreed. He left Elena's office and went to his desk to set up a meeting with Mike. This was going to be an exciting effort. This initiative was highly visible within the organization, since Advasco's customer approval rating had been taking a beating over the past two years.
In addition, Elena knew that Spencer had been researching SOA. In his reading, he felt that SOA had great potential to change the way that Advasco built applications. This effort would provide an opportunity to try out some of the technologies associated with it.
Later that afternoon, Spencer met with Mike to go through the existing applications. Mike said, "Unfortunately, the situation is a mess. Right now, the application that handles our auto insurance business is completely independent from the application that handles our home insurance business. The same thing is true for the life insurance business. They each have their own databases, requiring our agents to enter all of the customer information in multiple times. It creates a nightmare for our billing department, especially when trying to compute discounts for multiple policy holders. These applications have all been built using different technologies, including COBOL, VB.NET, and Java."
Spencer said, "Well, let's take a managed approach to this effort. Which are the two insurance lines where we most frequently see repeat customers?"
Mike replied, "The most common case is for a customer to hold both an auto insurance policy and a homeowner's policy. It's an easy way to get a multiple policy discount."
"Well, why don't we start with those two systems and see what we can do. I've been reading about SOA and I think it could provide the right approach for this effort."
"I'll trust you on this one Spencer. Elena spoke very highly of you in our meeting last week. As long as you don't think adopting new approaches will impact the timelines, I'm okay with it. The insurance sales and marketing group is under a ton of pressure to get our customer approval rating back up where it should be as quickly as possible."
The next day, Spencer met with the managers responsible for the auto insurance systems and the home insurance systems. Spencer kicked off the meeting, "We're here to discuss how we can make things better for our sales staff. Right now, they have to deal with two separate applications. As a result, sometimes they don't know when they're dealing with someone who is already a customer. Other times, we wind up with inconsistent records across the two systems, or have problems keeping records up-to-date when a customer moves."
Tim, the manager for the auto insurance systems immediately jumped into the conversation, "If we could get the home insurance system to use our customer database, our problems would go away."
Adil, the manager for the home insurance systems, responded to Tim, "We've spent the last 15 years evolving our application and database. It would be much more expensive for us to try to move all our data into your system."
Spencer could sense the tension in the room. Both of these managers had invested many years in their systems, and neither one wanted to relinquish any amount of control. "I don't think consolidating the data will work with the timelines we've been given. What I'd like to do is to create a new customer information service that will provide an abstraction layer in front of both of your databases. You'll both need to modify your applications to use the service rather than going directly to the database, but the service will ensure that both systems remain in sync. In addition, you'll have access to additional information about the customers in each other's systems that you can now incorporate into your applications. Then, at a later time, we can pursue consolidating the databases into a single one. With the service in place, you won't need to make any changes to the front end of your applications when that occurs."
Adil said, "How are you going to make this work? My system leverages a Java front-end talking to our mainframe, while Tim's system is based completely on Microsoft technologies?"
Spencer responded, "I think this a great opportunity to leverage web services technology. It claims to provide interoperability across these platforms, let's give it a try."
Tim said, "Who's going to write this service?"
Spencer suggested, "Since we need to incorporate information from both of your systems, I suggest that we form a team with a developer from each of your groups to design and build the new service."
Adil and Tim agreed, and told Spencer that they would let him know what developers they would contribute to the effort.
Spencer went back to his desk and knew that he had a real challenge on his hands. While the managers involved had committed developers to the effort, he also sensed that there was some hesitancy about the effort. They knew that changes were needed, but it was clear that neither one wanted to give up the control they currently had over their systems. He was hoping that the right developers would get assigned. He knew that many of them had complained about the redundancy that existed across the various applications, but the scope and timeline of their projects prevented them from doing anything about it. This effort was beginning with the right scope, so as long they met timelines, there was a good chance to make it happen.
Over the next few weeks, Spencer's team, including the developers from Adil and Tim's organizations, worked hard to define the new Customer service that both applications would use. The developers were very familiar with the current data models used by each application, and worked together to define the data models and schemas for the new service. While these developers had some knowledge of how the existing applications manipulated the data, they worked solely with each other in defining the functional interface of the service. In the end, the service interface contained some elements of Customer data that was specific to one of the two applications, but they felt that information could be safely ignored by the other application.
The First Milestone
Soon afterwards, Spencer met with Jennifer, the project manager for the auto insurance application, to discuss their schedule. Spencer said, "Hi Jennifer. I wanted to discuss our delivery schedule with you so we can ensure that you can integrate the new Customer service as part of your effort. I know you're making some additional changes to the application besides the migration to the service."
Jennifer took a glance at her project plan and told Spencer, "We're currently planning on going live on October 6th. Our performance tests are planned for September 2nd, and user testing will begin on July 28th."
Spencer said, "Okay, we'll plan on targeting those same dates for our service. Don't hesitate to contact me if you need anything else."
Jennifer didn't. Just two weeks later she sent him an email that said, "Spencer, my developers want to know when they'll have a service they can test against. They've told me that they can't do anymore work until it's available."
Spencer replied, "I'll have one of my developers provide you the URL for our development service right away." He had his team provide it, and didn't hear anything back from Jennifer's team, so he assumed everything was working well. That lasted for about one week when Jennifer came storming into his cubicle.
"What did you do to the service? We were going to demonstrate where we were to one of our users today, and the application crashed when we tried retrieving data," she said. It was clear that it had not been a good morning for her.
"We've been testing the integration with the home insurance data system, and have run into some issues, so our development environment has been up and down all day long as we try to determine what the problem is," he replied.
"Well, you'd better get it fixed soon. I now have a key user who's very nervous about the stability of this new service-based approach, and my developers are simply telling me that they can't do anything about it. I'm going to report this as a serious issue in our Project Management Office (PMO) update on Thursday."
"I'll get right on it, Jennifer. I apologize for this and we'll get it fixed as quickly as we can."
Spencer immediately realized where his mistake was. By giving Jennifer's team the URL for the service in his development environment, he was exposing her application to all of the instability that is normally associated with any development environment. From her perspective, however, she didn't care about his development environment; she needed something that was stable so she could execute her tests and provide demonstrations.
He gathered his developers and told them that they needed to create a stable version of the service that would be separate from their own development efforts that the auto insurance application could use. He suggested that they determine which of their iterations represented a key milestone from the perspective of the auto insurance application. When those iterations were successfully completed, it would be promoted to the stable environment in use by the auto insurance application. They implemented this plan and things got better, at least from Jennifer's perspective.
Unfortunately, Jennifer wasn't the only project manager whose effort was dependent on Spencer's team. Mark was the project manager for the home insurance application, the other consumer of this new service. Like his initial meeting with Jennifer, Spencer met with Mark to find out what his plans were for the home insurance application. Like Jennifer, there were additional changes that Mark was putting into the application in addition to migrating to the new Customer service.
Mark said, "We need to get our application into production by September 15th. There are some new regulations that have been passed, and we need to incorporate them into our application by then or else we can be subject to fines. We plan on beginning active user testing two months before then."
Spencer replied, "Your dates are pretty close to Jennifer's, so I think we'll be okay. We've already set up a stable version of the service for her project, so I'll send the information to you, and your developers can begin testing against it as soon as they'd like."
"That would be great. I know that it's important that we start using your service, so I'm glad that the timelines look good right now. It's very important that our dates are met. I absolutely must deliver no later than September 15th."
The Second Milestone
Spencer passed this information back to his development team and everything was going well until the second milestone release went out. Jennifer's team received the latest interface definition, which included the next set of functionality that her team was going to test. Unfortunately, her developers were not at all happy with the results. Spencer's team had determined the operations to expose in the web service-based solely upon their past knowledge of the applications that would be using it. Unfortunately, some significant changes had occurred to the applications, and their knowledge was out of date. The way in which they had represented the data in the operation was a complete mismatch to the processing model within the auto insurance application. While all the data was there, the way in which the data was organized would require Jennifer's team to completely disassemble the data and reassemble it in the data structures necessary. This had the potential of impacting her team's delivery schedule. Jennifer discussed this with Spencer, and he agreed to modify the service interface according to what her developers needed. This wound up being a relatively simple change for Spencer's team, so it was pushed out as part of the next iteration, two weeks later. That day was no better.
"Spencer, what happened to the service?" Mark said, with a clear sense of irritation in his voice. "We had integrated our application with the last milestone release, and then you go and change all the operations out from underneath us. What gives?"
"Mark, I'm sorry I didn't let you know about those changes. Jennifer, the project manager for the auto insurance upgrade project, wasn't happy with the last milestone release as her developers would have had to do significant rework in order to accommodate the interface design."
"Spencer, what am I supposed to do now? Your changes will now require significant rework in my application to accommodate these new interface changes, and I can't afford any more slips in my schedule."
"Let me set up a meeting with you and Jennifer so we can find an approach that will work with both of your schedules. I'll get it set up for this afternoon," said Spencer.
That afternoon, Spencer met with Mark and Jennifer to discuss the state of affairs. Neither of them was happy about the situation.
"Spencer, I can't take on any more development tasks right now. My team is already struggling to make the necessary changes for these regulatory changes in the time we have left. I can't afford to have them make changes that we didn't need because your team decided to change the service from underneath us," said Mark.
Jennifer replied, "Mark, your project isn't the only one that's important. I've got just as much pressure on me to get this project delivered, and that original interface would have added about three weeks onto my schedule."
Spencer was regretting not having involved Mark and Jennifer's teams back when the original interface design decisions were made. He had trusted that by having developers who had previously worked on both of those applications they would have known what operations would work best for each, but that clearly wasn't the case. While he didn't want to do it, he knew that he had to make sacrifices within his project in order to meet both Mark and Jennifer's needs. He asked, "Mark, if we put the original operations back in the service interface in our next iteration, would that work for you? It would be available within two weeks. Jennifer, we'd leave the new operations in the interface. While I'd like to only provide this functionality in one way, I realize that in order to meet both of your schedules, this is the only way that can be done."
Mark and Jennifer thought about this option, and they both agreed that they could make this work. All three of them agreed to continue meeting on a regular basis to ensure that there would be no more surprises throughout the rest of the effort. Spencer left the meeting very relieved that he was able to find an approach that would work. He knew that his experience was stronger on the technical side of things, and didn't realize that the far bigger challenges would lie in the coordination and communication efforts.
Over the next three months things went well for these three projects. Spencer, Jennifer, and Mark all met regularly and ensured that any issues that arose were dealt with to the satisfaction of everyone involved. In one of these conversations, Spencer found out that a new project had just been spun up from the annuity department. Even though this project was outside of the original plans, Spencer thought it would be an easy conversation given the success he'd had with Jennifer and Mark. He arranged for a meeting with the project manager, Ryan.
"Hi Ryan, Thanks for agreeing to meet with me. I'm the solution architect for the new Customer service project. The auto insurance team and the home insurance team are both leveraging it, and I thought it might be something you would be interested in. By utilizing it, you'll be well-positioned for providing visibility into the other products that annuity customers have."
Ryan replied, "I've heard a little bit about your effort, Spencer. Is it currently in production?"
Spencer responded, "We're scheduled to go live on September 15th along with the new home insurance system."
Ryan said, "Why don't you send some information about your service to my solution architect, Ramesh?"
"I'll do that," said Spencer. Spencer had worked with Ramesh on a previous project, so he decided to talk to him right away. Spencer walked down to Ramesh's desk, "Hey Ramesh, I just met with your project manager, Ryan. I'm working on a new Customer service project and I thought your annuity system could leverage it."
He went on to give Ramesh the details on the service that was being built, and how it was providing an abstraction layer for the auto insurance and home insurance customer databases, and positioning them for consolidation which would save the company significant costs, in addition to allowing them to gain the visibility they needed to start selling other products to existing customers.
Ramesh said, "This looks really promising, but who's going to take care of putting our database behind your abstraction layer?"
"Our efforts are nearly completed for the home and auto applications. My team has some extra cycles available, and from what I understand, your database is reasonably consistent with the home system, unlike some of the others."
"Well, if you're offering some free resources for the project, I'll at least take a look at your service documentation and meet with Ryan."
Spencer said, "Great! Just let me know if there's anything you need."
A week went by, and Spencer had not heard anything back from Ramesh, so he went to see him again. "Ramesh, did you get a chance to talk to Ryan?"
"Oh! Hi Spencer, sorry I didn't get back to you. I've been busy trying to keep things on track. I did get a chance to talk to Ryan, but he was concerned about taking on the additional scope of moving away from their existing database to the new service."
"Did you tell him that we'll take care of the work necessary within the service to add your database?"
"Trust me, I did. After looking at your documentation, it really didn't look like it would be much work at all for us to integrate in your new service, but he didn't want to hear any of it. Not only was he concerned about the added scope, he also said that he didn't want his project to be dependent on some other team. He said he'd done that in the past, and it only led to delays, angry meetings, and some poor performance reviews for everyone involved."
Spencer decided to go and speak directly to Ryan. While Ramesh indicated that he had talked to Ryan, Spencer still wasn't sure whether he had done a good job in hitting the key selling points for using the service. Spencer was very passionate about SOA, and realized that not everyone shared that same passion. He caught up with Ryan the next day.
"I spoke with Ramesh yesterday, and he said that you told him not to pursue leveraging the service from my team. Is there something I can do that might change your mind?"
"Spencer, I'm sorry, but I just can't take on any additional scope in this project. I appreciate the offer, and the fact that your team was willing to take on the work to integrate our database, but I've had too many bad experiences in the past with creating dependencies on teams outside of my control."
"I really think this service would position you well for the future, since you know that Advasco is really trying to improve our relationship with our customers.
Jennifer and Mark are both in the same situation, and we've been able to deliver everything they've needed without impacting their schedule. Would it help if you talked to them?"
"I really don't have time for this Spencer. Jennifer and Mark were both mandated to use this service thing that you're so excited about, and I know both of them were pretty nervous about it. I am under no such mandate, and I'm not going to take on any risk that I don't have to. Now, if you'll excuse me, I've got a project to manage."
Humbled, Spencer went back to his cube looking like a dog with its tail between its legs. He was frustrated that even though the solution architect for the project had reviewed the service documentation and told Ryan that it wouldn't impact on the project, Ryan refused to budge. Not even the impact of the company's desire to improve its relationship with its customers could change his mind.
Spencer went home that night and talked with his wife, Alexandra, about his day. "I just don't understand why I couldn't convince Ryan to leverage the service. I tried to remove every road block that I could think of, and he still just said 'No' without even looking into the details."
Alexandra responded, "While I know you believe strongly in this, you really were asking him to step outside of his comfort zone. Like he said, Jennifer and Mark didn't have a choice. They were told up front that they had to use your service, so they planned for it from the beginning. From my experience, a project manager's worst nightmare is having dependencies that are outside of his control, and that's exactly what you were asking him to do. I think most of the project managers I know would have said 'No' as well. As much as this may have helped Advasco in the future, I think you need to focus on ensuring that your efforts for Mark and Jennifer are successful. Nothing breeds interest like success."
Spencer thought about what she said and agreed, "I can see your point. I shouldn't think that the whole thing is going to fall apart just because of one project manager. Mark and Jennifer have been very happy for the past two months after our early problems."
The service development effort continued, and on September 8th, a week ahead of schedule, the service went live along with Mark's application. Four weeks later, on October 6th, Jennifer's application was added to the list.
In this article, we walked through a simple example of a company's initial SOA efforts. We saw how project and program structure establishes a recognized set of authorities within the project or program, but how that authority does not go beyond the project or program. We saw how working across project boundaries, even when under a common program, can be challenging when the required interactions are not made explicit. We saw the challenge that a service provider faces when trying to establish an interface that meets the needs of multiple consumers, both from a functional standpoint as well as a scheduling standpoint.
In the next part, we will cover key project roles, Service Contract, adding SOA to Traditional Project Governance, Service Communication Technologies, WS-I Compliance, Service Interface Specification, Web Services, POX over HTTP, and REST.
If you have read this article you may be interested to view :