As companies build and extend their IT information assets, there is a need to integrate applications to exchange data. While a point-to-point application integration using custom code is possible, it does not address large-scale application integrations without duplication of data and the complexity of managing such an integration. BizTalk is a de facto choice for message-based integration for on-premises systems and for services on Azure for both applications and businesses.
In this book, we're going to introduce you to BizTalk Services, Microsoft's new integration middleware hosted on Windows Azure. This book assumes that you already understand the need for integration and the many benefits of using specialist integration software as opposed to building custom point-to-point integration solutions. We will look at why cloud-hosted integration services are so compelling and the scenarios in which these services make most sense. Prior knowledge of BizTalk Server is not expected, though it will help you to quickly understand certain topics within the book. We will assume you are a professional developer, an architect, or a solution designer familiar with the challenges of integrating heterogeneous systems and applications. The book also assumes familiarity of the Microsoft developer toolset, specifically .NET, Visual Studio, and SQL.
In this chapter, we will start to answer some fundamental questions around what BizTalk Services is and how it can help you to build integration solutions. Specifically, the following topics will be discussed:
The business and technical drivers for BizTalk Services on Azure
Concepts and architecture of BizTalk Services
Using BizTalk Services to realize a simple purchase order scenario
First though, some background is necessary. At the Professional Developers Conference (PDC) 2008, Microsoft unveiled Windows Azure—a new operating system designed for the cloud. Over the subsequent years, Windows Azure has become Microsoft's de facto cloud platform covering services, media, websites, mobile applications, and more. While BizTalk Server has been an established product for over ten years (and eight releases), cloud adoption has been driving connected systems across different services and Line-of-Business applications. To meet the growing demand for cloud-based integration, the Microsoft BizTalk team released the first version of BizTalk Services, named Service Bus EAI and EDI Labs Community Technology Preview (CTP) on December 17, 2011. The goal was for customers to be able to sign up in a shared environment and set up simple XML/EDI flows without worrying about installation and maintenance. The capabilities were rich enough to enable simple point-to-point integration scenarios; on April 9th of the following year, the CTP was refreshed, incorporating feedback from customers by providing additional capabilities.
The CTP environment was hosted on a publicly shared service, and therefore had restrictions on running users' custom code. Integration is rarely straightforward, and the ability for developers to write custom code and deploy it as part of their integration solutions was a key requirement. Customers had also needed guarantees around performance and Service Level Agreements (SLAs). Hosting user assemblies as part of the cloud services became a reality by switching to per tenant deployment. Thus, the cloud offering known as BizTalk Services was born on June 3, 2013. Like many Azure services, BizTalk Services is expected to be updated at a regular cadence. The latest update as of this writing was on February 20, 2014. This technology is opening up new integration possibilities; with Microsoft's on-going investments, it will be on par with the capabilities of its on-premises cousin, BizTalk Server, in the near future.
There are many tangible benefits of building solutions on Azure today. A few of these include the ability to scale up/down the platform based on predictable or dynamic shifts in application load and throughput requirements without worrying about the hardware procurement time and setup, the ability to pay as you go with expense incurred as an Operational Expenditure (OpEx) instead of a Capital Expenditure (CapEx), and the increase in reach for certain integration scenarios.
Specific to integration on Azure, there are four factors that drive adoption:
Focus on business operations, not IT: There are business benefits in terms of reduced cost by leveraging platforms running under economy of scale, making it cheaper for customers to obtain them with higher quality of services. The need of the hour is to simplify IT deployment and management and focus on business services instead of configuring software or hardware.
Simplicity of managing externally facing services: Enterprises typically offer services across geographic and organizational boundaries. Many of these services require making changes to the corporate firewall to allow or deny the applications. This process is an IT nightmare for many organizations. With Azure integration services such as B2B, which inherently require external communication, all this could be moved to the cloud and the necessary access and control policies could then be centralized for all internal services. This also enables self-service configuration changes to the application, thereby dramatically improving responsiveness to business changes.
Greenfield cloud applications: The proliferation of mobile devices, such as smartphones and tablets as well as Point-of-Sale (POS) systems, has given rise to services that are inherently cloud based. Think of a POS that transmits daily transaction logs to its backend Line-of-Business systems or an RFID service that transmits information about each item in a shopping cart purchased in a retail store to an inventory application. As new services are being rolled out, organizations want to be able to develop and deploy these services with shorter time-to-market using Azure.
Another factor to consider for the cloud is to leverage existing on-premises investments. Businesses have invested in a variety of on-premises systems, including traditional ERP as well as legacy mainframes and other bespoke systems that literally run the business.
The primary technical goal for BizTalk Services is to reduce the impedance mismatch between source and destination systems that are exchanging information. Such impedance can be at different levels:
Transport protocol impedance: The source might send messages over one transport (say FTP) and the destination may only accept messages over another transport (say POP3). It could also be the case that messages are sent from one LOB to another, for example, one end is sending messages from the Sales Force adapter while the other end is accepting messages via the SAP adapter. BizTalk Services provides the notion of adapters to resolve this impedance.
Application protocol impedance: The source might send EDIFACT messages while the destination may only accept messages in XML. BizTalk Services provides native support to protocols such as X12, EDIFACT, and flat files to resolve this impedance.
Format impedance: The source might send messages in one XML format while the destination may only accept messages in another XML format. BizTalk Services provides transforms to resolve this impedance.
Timing impedance: The source can send messages any time of the day, but the destination only accepts messages between 4 and 7 P.M. The source can send messages twice as fast as the destination can process them.
Size impedance: The source can send messages of any size, but the destination can accept messages of 1 MB at most.
BizTalk Services provides connectivity to Service Bus, batching and debatching to resolve the last two impedances.
The aforementioned drivers have resulted in three core scenarios for BizTalk Services:
Enterprise Application Integration: These are primarily messaging scenarios with flat file or XML-based data that are between two or more applications, atleast one of which is running in the cloud. A good example would be a travel portal connected to the ticketing systems of multiple airlines.
Business-to-Business Integration: These are messaging scenarios with structured flat file/XML between two organizations. An example would be an IT company procuring hardware from vendors such as HP, Dell, or Lenovo.
Connectivity with Hybrid Applications: These are messaging scenarios between Azure and on-premises applications. An example here would be connecting a Salesforce application to a SAP application running in your internal IT environment.
We will look at these scenarios in detail in separate chapters of this book.
The following figure illustrates a basic integration flow from an FTP source to a LOB destination. BizTalk Services, represented by the middle box, is a sequence of processing steps.
Role of BizTalk Services
BizTalk Services introduces several key concepts to facilitate building integration solutions on Azure:
Bridge: A bridge is a unit of processing in BizTalk Services that can address impedance mismatch. It contains three units: one or more source locations (for example, FTP) to read messages from, a pipeline to process the message, and one or more destinations (for example, Queue) to write the processed messages. The pipeline is divided into distinct processing units called stages, each with its own function (for example, a stage in a pipeline can validate a message against a schema). A series of stages represents the bridge pattern or bridge template. Out of the-box, BizTalk Services v1 ships with three templates: XML, EDI, and AS2.
Adapter: An adapter is the transport medium that can send messages (to a destination) or receive messages (from a source) and pass them to the pipeline in a bridge; for example, Line-of-Business adapters such as SAP and Oracle EBS or transport adapters such as FTP and SFTP.
Transform: A transform converts a message from one format into another, aiding structural conversion. Transforms contain operations that can perform commonly used transformations like string operations, loop constructs, list operations, and arithmetic and logical expressions.
Application protocol: A protocol defines the message format and processing semantics such as the requirement to send and correlate acknowledgements of messages.
Route: A route defines the destination endpoint where the message will be sent based on the specified criteria. The route criteria are evaluated based on SQL-92 expression syntax.
Batching: The aggregation of messages based on selection criteria is termed batching. The release (sending) of a batch is governed by size, count, or time, or a combination of these parameters.
Promoted properties: Promoted properties are name-value pairs, where the name is user-defined and the value is derived from the message header, message body, or from the context within the bridge. Promoted properties are commonly used in batching and routing to specify their criteria.
Artifact: An artifact is anything that aids in the processing of the message in the bridge. XML schemas, maps, custom assemblies, and certificates are the artifacts used in XML and EDI bridges. In BizTalk Services, each artifact is stored in the artifact store and is addressable by a unique URL.
Unlike most other Azure services deployments, BizTalk Services provisions dedicated resources for storage and compute instances that are isolated across tenants. This means that no two deployments have anything in common between them. The advantage is that you can write any custom code and be assured that you cannot impact the performance or availability of other deployments. This dedicated deployment also offers isolation of data at the storage level, thus increasing the privacy of data and SLA of the service.
Broadly categorizing, there are three steps to go through before you can have an active usable deployment. The first is to provision the service using standard Azure tools, including the Azure portal to create the service; the second is to deploy the necessary artifacts and configuration outlined in the BizTalk Services concepts section using Visual Studio and the BizTalk Services portal; and the third is sending or receiving the messages.
The architecture of BizTalk Services contains three key components:
Provisioning services: This is a set of Microsoft services that manage the lifecycle of a BizTalk Services deployment as well as monitor its health. It also includes components to bill the end user based on usage of BizTalk Services. The management interface to the service is exposed via the Red Dog Front End (RDFE) public API. The Azure Management Portal or PowerShell scripts from the user go via the RDFE API. Using the service, you can scale up/down your deployment as well as back up and restore deployment across datacenters.
Per tenant BizTalk Services: This is the per tenant deployment that is created in the user's Azure subscription. A BizTalk Services deployment is identified by the deployment name and is accessible using the URL secured by the Access Control Service (ACS). All artifacts such as bridges and schemas are added into the deployment with a URL which is a sub-path of the deployment URL. For example, a bridge is added under
<deployment URL>/default/<bridgeName>. Here default is the namespace name where the artifacts are grouped into.Per tenant dependencies: These dependencies are the Azure services required for tracking, troubleshooting, and security. For example, BizTalk Services provides a tracking store, which is an Azure SQL database where the processing status of messages, along with related properties, are stored as the messages pass through a bridge. The information from the tracking store is shown in the BizTalk Services portal tracking view. Archiving and monitoring is stored in Azure storage blobs and tables. Archived messages are stored in blob containers based on the date of the archive. The storage also contains the Azure table WADLogsTable, where tracing information for a bridge can be obtained. Finally, Access Control Service regulates access to all endpoints in the deployment. During deployment creation, the provisioning service uses the Management Service credentials to programmatically access ACS to create a relying party for the BizTalk Services deployment, add rule groups for Send, Listen, and Manage claims, and create the service identity with the necessary passwords for directly talking to ACS for deployment of the artifacts. The interaction between these components is illustrated in the following figure:
Block diagram of BizTalk Services shared and per tenant services
BizTalk Services provides different user experiences for different personas to facilitate an optimized, task-based approach. The principal personas are:
A developer will typically focus on creating solutions using Visual Studio. BizTalk Services VS 2012 project templates are provided to enable rapid creation of both EAI and EDI solutions. These provide a graphical work surface on which to create and configure bridges to facilitate communication between an enterprise and its trading partners. Additionally, sophisticated tools are provided, including a graphical mapping interface and schema editors.
The Windows Azure Platform Management Portal provides access to create the BizTalk Services deployment and management tasks as well as at-a-glance status information providing details on the overall health of all deployments and accounts. In addition to services deployment, the Windows Azure Platform Management Portal provides an interface for provisioning Azure SQL databases, mobile services, Service Bus entities, and so on.
The Partner Administrator persona uses the BizTalk Services portal for a number of management functions such as the creation and administration of trading partners, configuration of agreements including required transformations, routing and acknowledgements, tracking of messages, and exception processing.
The BizTalk Services portal enables the creation of trading partners and agreements between them. This enables the setting up and management of the protocols used to exchange data (for example, X12 and AS2) and the message formats to use together with transformation and routing capabilities. In this way, trading partners can be onboarded and configured quickly and easily by non-IT personnel without the use of developer tools such as Visual Studio, all through the web-based portal.
In addition, the management portal also provides the ability to set and view tracking data on message flows, including both contextual details (sender, message type, and so on) as well as the message bodies themselves. The ability to archive and export message data is also provided as part of the service.
Additionally, RESTful APIs are implemented to provide full fidelity with the portal, enabling activities to be scripted and deployment to be automated. Additionally, integration with customer systems and tools such as SharePoint for tracking data, visualization, or on-premises storage is also possible using this API.
You will need to consider the BizTalk Services edition required for your production use as well as the environment for test and/or staging purposes. This depends on decision points such as:
Expected message load on the target system
Capabilities that are required now versus 6 months down the line
IT requirements around compliance, security, and DR
The list of capabilities across different editions is outlined in the Windows Azure documentation page at http://www.windowsazure.com/en-us/documentation/articles/biztalk-editions-feature-chart.
Note
Note on BizTalk Services editions and signup
BizTalk Services is currently available in four editions: Developer, Basic, Standard, and Premium, each with varying capabilities and prices. You can sign up for BizTalk Services from the Azure portal. The Developer SKU contains all features needed to try and evaluate without worrying about production readiness. We use the Developer edition for all examples in this book.
BizTalk Services deployment can be created using the Windows Azure Management Portal or using PowerShell. We will use the former in this example.
Certificates are required for communication using SSL, and Access Control Service is used to secure the endpoints of the BizTalk Services deployment. First, you need to know whether you need a custom domain for the BizTalk Services deployment. In the case of test or developer deployments, the answer is mostly no. A BizTalk Services deployment will autogenerate a self-signed certificate with an expiry of close to 5 years. The ACS required for deployment will also be autocreated. Certificate and Access Control Service details are required for sending messages to bridges and agreements and can be retrieved from the Dashboard page post deployment.
You need to create an Azure SQL database for tracking data. It is recommended to use the Business edition with the appropriate size; for test purposes, you can start with the 1 GB Web edition. You also need to pass the storage account credentials to archive message data. It is recommended that you create a new Azure SQL database and Storage account for use with BizTalk Services only.
Now that we have the security and storage details figured out, let us create a BizTalk Services deployment from the Azure Management Portal:
From the Management portal, navigate to New | App Services | BizTalk Service | Custom Create.
Enter a unique name for the deployment, keeping the following values—EDITION: Developer, REGION: East US, TRACKING DATABASE: Create a new SQL Database instance.
In the next page, retain the default database name, choose the SQL server, and enter the server login name and password.
In the next page, choose the storage account for archiving and monitoring information.
Deploy the solution.
The BizTalk Services create wizard from Windows Azure Management Portal
The deployment takes roughly 30 minutes to complete. After completion, you will see the status of the deployment as Active. Navigate to the deployment dashboard page; click on CONNECTION INFORMATION and note down the ACS credentials and download the deployment SSL certificate. The SSL certificate needs to be installed on the client machine where the Visual Studio SDK will be used.
We have one step remaining, and that is to configure the BizTalk Services Management portal to view agreements, bridges, and their tracking data. For this, perform the following steps:
Click on Manage from the Dashboard screen.
This will launch
<mydeployment>.portal.biztalk.windows.net, where the BizTalk Portal is hosted.Some of the fields, such as the user's live ID and deployment name, will be auto-populated.
Enter the ACS Issuer name and ACS Issuer secret noted in the previous step and click on Register.
BizTalk Services Portal registration
Let us put things into action and use the deployment created earlier to address a real-world multichannel sales scenario.
A trader, Northwind, manages an e-commerce website for online customer purchases. They also receive bulk orders from event firms and corporates for their goods. Northwind needs to develop a solution to validate an order and route the request to the right inventory location for delivery of the goods. The incoming request is an XML file with the order details. The request from event firms and corporates is over FTP, while e-commerce website requests are over HTTP. Post processing of the order, if the customer location is inside the US, then the request are forwarded to a relay service at a US address. For all other locations, the order needs to go to the central site and is sent to a Service Bus Queue at IntlAddress with the location as a promoted property.
Before we start, we need to set up the client machine to connect to the deployment created earlier by performing the following steps:
Install the certificate downloaded from the deployment on your client box to the trusted root store. This authenticates any SSL traffic that is between your client and the integration solution on Azure.
Download and install the BizTalk Services SDK (https://go.microsoft.com/fwLink/?LinkID=313230) so the developer project experience lights up in Visual Studio 2012.
Download the BizTalk Services EAI tools' Message Sender and Message Receiver samples from the MSDN Code Gallery available at http://code.msdn.microsoft.com/windowsazure.
We will break down the implementation details into defining the incoming format and creating the bridge, including transports to process incoming messages and the creation of the target endpoints, relay, and Service Bus Queue.
You can create a new BizTalk Services project in Visual Studio 2012.
BizTalk Services project in Visual Studio
From within your project, right-click on the project name, click on Add | New Item, and add a new Flat File Schema.
Add Flat File Schema into BizTalk Services project
Add the following nodes to the schema so that the structure looks as follows:
Flat File Schema structure
For each of the records in the XSD file, make sure that the delimiters are added correctly:
<b:recordInfo structure="delimited" child_delimiter_type="char" child_delimiter="," child_order="postfix" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" sequence_number="4" />
You can validate the schema by running it with an instance file. The Validate Instance command is available by right-clicking on the created schema file in the Solution Explorer. Add the following flat file and XML instances in two separate files, use the Validate Instance command, and verify that the schema validates those instances. For each command run, ensure that the schema properties window has the right Validate Instance Input Type (XML in this case):
OrderId|PaymentType|OrderDate|Code,Qty,Price,|Name,Email,Phone,|Recipient,Number,Street,City,State,Country,Pincode,|
<ns0:Order xmlns:ns0="http://BizTalkServicesOrderSample.OrderFF">
<OrderId>MyOrder</OrderId>
<PaymentType>CreditCard</PaymentType>
<OrderDate>09-08-2013 22:50:00</OrderDate>
<Product>
<Code>100</Code>
<Qty>1</Qty>
<Price>500</Price>
</Product>
<Customer>
<Name>Karthik</Name>
<Email>user@hotmail.com</Email>
<Phone>1-111-1111</Phone>
</Customer>
<ShippingAddress>
<Recipient>Jon</Recipient>
<Number>Building 1</Number>
<Street>One Redmond Way</Street>
<City>Redmond</City>
<State>Washington</State>
<Country>US</Country>
<Pincode>98052</Pincode>
</ShippingAddress>
</ns0:Order>Tip
Downloading the example code
You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
Open the bridge configuration surface (usually the MessageFlowItinerary.bcs file). The Visual Studio Toolbox should show the following entities:
Visual Studio Toolbox for BizTalk Services project
Use the VS toolbox to drag-and-drop FTP Source, Xml One-Way Bridge, One-Way Relay Endpoint, and Queue and connect them using the Connector, as shown in the following figure:
Bridge message flow in BizTalk Services project
Configure the following in the message flow:
Select the FTP server and configure the address, username, and password correctly.
Double-click on the bridge to open the Xml One-Way Bridge configuration:
In the Message Types block, add the
OrderFF.xsdinstance created earlier.
Bridge configuration
In the first Enrich stage, add an XPath Type property reading from
/*[local-name()='Order' and namespace-uri()='http://BizTalkServicesOrderSample.OrderFF']/*[local-name()='ShippingAddress' and namespace-uri()='']/*[local-name()='Country' and namespace-uri()='']and writing to the location as a string. The XPath value can be obtained by opening the schema in VS, clicking on the relevant record, and copying the XPath value from the record properties window.
Promote property configuration in the Enrich stage of the bridge
In the parent
MessageFlowitinerary.bcsview, click on the route link from OrderProcessingBridge to USAddressRelay and set the filter condition as location='US'; for the other link, set the location toUS.
Route properties for the message flow
Edit the Queue
.configfile underMessageFlowitinerary.bcsand update the<tokenProvider>and the<endpoint>details with the Service Bus information.Edit the Relay Service
.configfile underMessageFlowitinerary.bcsand update the<endpoint>details with the Service Bus relay information.Build and deploy the solution.
If the deployment was successful, point your browser to
https://<yourdeployment>/default/OrderProcessingBridge; you should see a 401 HTTP Error code stating a manage claim is required for this operation.
We need to test sending two kinds of messages: one from the corporates and the other coming from the website:
Load the MessageReceiver sample in VS and build the solution. From the output bin folder, run the following in a command prompt window:
MessageReceiver.exe ServiceBusNS owner issuerkey USAddressRelay OneWayRelay
Here,
ServiceBusNSis the namespace where the relay is running andMyRelayTestSvc1is the endpoint information configured in the bridge configuration.Load another MessageReceiver in a new command window.
MessageReceiver.exe ServiceBusNS owner issuerkey IntlAddressQueue Queue
Here,
ServiceBusNSis the namespace where the Queue has been precreated andIntlAddressQueueis the endpoint information configured with the bridge.Load the MessageSender sample in VS and build the solution.
<yourdeployment>is the URL where BizTalk Services was provisioned earlier.MessageSender.exe BizTalkSvcACS owner issuerkey https://<yourdeployment>/default/OrderProcessingBridge instance.xml application/xml
Here,
BizTalkSvcACSis the namespace of the BizTalk Service deployment ACS,ownerandissuerkeyare the ACS credentials of that namespace, andinstance.xmlis theOrderFF.xsdinstance in XML format.The output is observed in the MessageReceiver of the relay.
Edit
instance.xmlwithlocation=EUand run theMessageSendercommand again. This time the output will be observed in the MessageReceiver of the Queue.Drop a flat file in FTP with
location=USand observe the output in the relay service window.Drop a flat file in FTP with
location=EUand observe the output in the message receive queue.
Northwind can now process both flat files and XML orders from either HTTP or FTP endpoints. You can delete the bridge from the BizTalk Services portal Bridge view or by using PowerShell.
In this chapter, we have introduced the basics of BizTalk Services and the concepts, architecture, personas, and tools available to build an integration solution. We also exercised all the concepts learned through a simple order processing scenario with BizTalk Services and Service Bus relay and queues. The example can be further extended to include transforms, routing to other bridges like EDI, custom code, and so on. In the next chapter, we'll look at some of the BizTalk Services capabilities in more detail.
