img
ServiceTechMag.com > Issue LXVI, September 2012 > EAI and EDI in the Cloud: Prospects of Azure Service Bus EAI & EDI - Part II
Manfred Steyer

Manfred Steyer

Biography

Manfred Steyer (www.softwarearchitekt.at) is a freelance trainer and consultant at IT-Visions (www.it-visions.de). He also coordinates the specialist field of "Software Engineering" of the Degree Programme in "Information Technologies and Business Informatics" at CAMPUS 02, University of Applied Sciences (www.campus02.at) in Graz, Austria. In cooperation with Holger Schwichtenberg, he describes the joint use of WCF, WF, WIF, Entity Framework and Azure Service Bus for company applications in his current book "Verteilte Systeme und Services mit .NET 4.0: Konzepte und Lösungen mit WCF 4.0".

Contributions

rss  subscribe to this author

Bookmarks



EAI and EDI in the Cloud: Prospects of Azure Service Bus EAI & EDI – Part II

Published: September 20th, 2012 • Service Technology Magazine Issue LXVI PDF
 

Abstract: This is the second article in a two-part series that discusses how in the future, the EAI features of Azure Service Bus will offer the transformation and routing of messages to integrated systems on a pay-per-use basis while maintaining its usual high availability. Moreover, standards like EDIFACT and X12 will also be supported in the same way as the integration into local systems via relaying. To read the first part of this article series, click here.


Enriching Data in a Bridge

To enrich data in a bridge, actions which extract values from the message and subsequently map them to other values can be configured. These actions are to be performed in the framework of the Property Definitions feature which is offered by the enrich stages of a bridge.

So-called lookup providers are used for the mapping of values. The present CTP only contains one lookup provider which looks up a value in a column of an SQL Azure Table that is to be defined and then delivers the value of another column to the line which has been found in this way. Fig. 9 shows such a table mapping incoming SOAP actions to outgoing SOAP actions.


img

Figure 9 – SQL Azure Table with action mappings


In order to use this table to define the necessary outgoing SOAP action, the incoming SOAP action first must be extracted from the message and assigned to a variable. Therefore, the type Soap and the identifier Action are used as shown in Fig. 10. Action_From has been defined as the name of the target variable, with String being its data type.


img

Figure 10 – Extracting an incoming SOAP action


Once the defined action has been added with a click on the button with the plus symbol, it is defined that the value of the enriched variable is to be used for lookup with a lookup provider, as shown in Fig. 11. Lookup is now used as a type. To configure a lookup provider, the Configure New option must be selected under Identifier. Subsequently, the reference values for the lookup are assigned (see Fig. 12). These reference values are defined in the LookupProviderConfigurations.xml file (cf. Listing 5) and can later be directly modified in this very file.

The target variable is now called Action_To and is a String type variable. This variable must be given when defining the routing action in order for the service bus to use the defined value for the SOAP action when sending the transformed message. Procedures are similar when transforming the SOAP action of the response message.


img

Figure 11 – Using a lookup provider in an enrich stage



img

Figure 12 – Configuring a lookup provider


			
Listing 5
<?xml version="1.0" encoding="utf-8"?>
<ArrayOfLookupProviderConfiguration 
       xmlns:i="http://www.w3.org/2001/XMLSchema-instance" 
       xmlns="http://schemas.microsoft.com/WindowsAzureServiceBus/
	   Bridge">
 
  <LookupProviderConfiguration i:type="SqlTableLookupProvider
  Configuration">
    <ProviderName>ActionLookup</ProviderName>
    <ConnectionString>
      Server=tcp:[...].database.windows.net,1433;Database=Sample;
	  User ID=[...];Password=[...];Trusted_Connection=False;
	  Encrypt=True;
    </ConnectionString>
    <QueryInColumnName>Action_From</QueryInColumnName>
    <QueryOutColumnName>Action_To</QueryOutColumnName>
    <TableName>ActionMapping</TableName>
  </LookupProviderConfiguration>
[...]
 
</ArrayOfLookupProviderConfiguration>
END
			

As alternatives to the Soap type which extracts elements from SOAP headers, the Http and XPath types may also be used for extracting values. Http extracts data from HTTP headers, while the latter extracts data from the message which correspond to an XPath expression which has to be defined. The following instruction which is based on an example from the documentation, for example, would look for an element B which is located directly below an element A, the content of which it would return. Thereby it is assumed that both elements are located in the namespace http://example.org/schema. The asterisks stand for "any element" and are further defined by the expressions given in brackets.

/*[local-name()='A' and namespace-uri()='http://example.org/schema'] /*[local-name()='B' and namespace-uri()='http://example.org/schema']

Enriching Data in Maps

Maps can also be based on lookup providers. Therefore, the Data Lookup map operation (symbol with magnifying glass) is to be used as shown in Fig. 13. The value to be mapped is defined as an input parameter, with the result of the operation being also the result of the lookup. The lookup provider to be used can be entered after a double click on the operation. To create a new lookup provider, its reference values are to be defined in the LookupProviderConfigurations.xml file.


img

Figure 13 – Lookup providers in maps



Integrating Local Systems via the BizTalk LOB Adapter

The BizTalk Server line-of-business systems adapters (LOB adapters) [REF-5] offer direct access to business applications and databases. In combination with relaying they enable Azure Service Bus to communicate with these locally installed applications. Support for SAP, Siebel eBusiness Applications, Oracle eBusiness Suite, Oracle Database and SQL Server is offered "out-of-the-box"; other systems can be integrated by implementing user-defined WCF LOB adapters.

To actually make use of this possibility, first a link to the desired LOB system is to be created in Server Explorer (Fig. 14). Subsequently, access to the LOB system is to be defined and it must be clarified which actions are to be carried out when doing so. Fig. 15, for example, defines that the configured LOB adapter should allow the insertion of data sets in the SQL Server table orders. The name of this operation, which subsequently is to be used as a SOAP action, is TableOp/Insert/dbo/orders. Next, XML schemas for the communication with the LOB adapter are to be generated. This may be done using the Add schemas command which is to be found in the context menu of the added LOB adapter. The messages in the bridge must now be mapped to these schemas using maps. In the case in question, two maps must be created: one for the request with the adapter (Fig. 16) and another one for the successive response containing the number of the lines concerned (Fig. 17).


img

Figure 14 – Adding SQL Server using LOB adapters



img

Figure 15 – Configuring access to SQL Server via LOB adapters



img

Figure 16 – Mapping the request for the SQL Server LOB adapter



img

Figure 17 – Mapping the response for the SQL Server LOB adapter


The maps can be used within a bridge; the LOB system in the bridge configuration is represented by an appropriate element of the Toolbox's On-Premise LOBs category. A Route Action defining the abovementioned identifier as a SOAP action is to be created to connect the bridge with the LOB system. Moreover, a filter is required, although this is a match all filter only. This is demonstrated by the bridge configuration in Fig. 18 which is an extension of the previously mentioned examples. Bridge1 receives orders from client A and transforms these into the format of service B. A filter ensures that, depending on the data in the message, these are either routed to OrderServiceB or to Bridge2. This decision is based on a context variable defined during an enrich stage of Bridge1 using an XPath expression. Once Bridge2 has received the message, it is once again transformed for the LOB adapter and routed to the LOB adapter, leading to a data set being created in the database.


img

Figure 18 – Bridge configuration using an LOB adapter



EDI and Communication via Flat Files with the TPM Portal

With regard to flat files and EDI, Azure Service Bus also benefits from developments realised in the framework of BizTalk Server. This includes, for example, schema descriptions for a wide range of selected X12 messages and the Trading Partner Management Portal (TPM Portal) [REF-3]. BizTalk Server's usual support for EDIFACT has not yet been included in the CTP in question. According to the product team, the final version is to feature this aspect. Business partners can be created in the portal. Thereby a differentiation is made between managed and unmanaged business partners. The first are the parties for whom the portal is operated; the latter are the ones the managed business partners wish to communicate with (Fig. 19).


img

Figure 19 – TPM Portal


Agreements can be entered into by the individual partners. These agreements define which partner sends which EDI messages to which other partner as well as the fields in the messages that identify the individual partners. For every agreement, a pipeline is configured (Fig. 20). It defines via which transport protocol ("transport") the messages are received and/or sent, which (X12) message format ("protocol") they must comply with and with which map their XML representation should be mapped to another message ("transform") and, finally, to where the message which is eventually transformed should be routed ("route").

The present CTP offers a selection between the HTTP, FTP and AS2 transport protocols; during routing, relay endpoints, topics, queues and bridges may be addressed. In addition, an address may be defined where messages that have been failed to process will be sent.


img

Figure 20 – Configuring an agreement in the TPM Portal



Support for Flat Files

Undoubtedly one of the most important innovations of the current CTP is the support for reading flat files. Sending flat files, however, is not yet possible, but the product team aims at supporting this function to, in one of the future (pre-)versions.

As the service bus internally almost exclusively works with XML-based data, the developer must transform the received flat files into XML documents using a one-way bridge. Therefore a special XML schema definition is used, as with BizTalk Server. This definition contains annotations giving information about where the described elements are to be found in the flat file. If a one-way bridge is equipped with such a schema, it transforms the received data into the XML representation described therein even before the first stage has been performed.


Using the Flat File Schema Wizard

Luckily, the developer does not have to create flat file schemas manually and can use a wizard. Anyone well acquainted with BizTalk Server will experience a déjà vu, even more so as this wizard has been taken from BizTalk.

Using the flat files in Listing 6 as an example, the subsequent description shows how the Flat File Schema Wizard is used. When taking a closer look at this flat file, one will notice that it describes an order. The first line contains header information, the second line contains several contact people separated by semicolons (;). Their fields - names and email addresses of the respective contact people - are separated by a pipe (|).

The individual order lines are located below the contacts. These consist of a product number and a number. Here, the delimiter is replaced by fixed positions: The product number starts in position one; the number in position nine.

			
Listing 6
ORDER;2012-05-01
CONTACTS;Max|max@acme.com;Susi|susi@acme.com
P-001   3
P-002   7
P-003   20
End
			

In order to create a flat file schema for data with this structure, the developer adds a new Generate Flat File Schema type element to the project (right click on the folder in Solution Explorer | Add | New Item | Generate Flat File Schema). Visual Studio then starts the wizard. On the first page, the developer creates a sample file with the expected structure. He also defines the name of the XML element which should serve as the root element after the XML conversion and adds an XML namespace for the document. On the second page, he marks the lines to be parsed in a text field. The standard selection comprises all lines.

The third page of the wizard with the heading Select Record Format is pretty exciting. This page defines the way in which the parts of the root element which has been previously defined can be parsed (see Fig. 21). The following options may be chosen: By Delimiter Symbol and By relative Positions. As the individual elements of the order in Listing 6 are mainly separated by vertical spacing, the first option is selected for the example detailed in this article.


img

Figure 21 – Selecting record formats


On the next page the developer defines the desired delimiter (child delimiter) as well as a tag identifier, if necessary (Fig. 22). The latter is a string from which the parser is able to recognise the information of the respective section. For the case in question {CR}{LF} is used as a delimiter, which represents vertical spacing. The prefix ORDER; is used as a tag identifier.


img

Figure 22 – Defining delimiters and optional tags


On the page named Child Elements the developer is asked to describe the individual elements which are created when splitting the text at the defined delimiter in more detail (Fig. 23). In the present example, the date is defined as Field Element in the first line. This means that this text area contains an "atomic" field and does not have to be divided any further. For the second line which contains the contact data, the developer selects the Record element type, even more so as this is a section which must be split into individual contacts with a name and an e-mail address for each. For the third line, which already represents the first order position, the Repeating Record element type is used. Similar to contact information, this is a record which is to be split into its individual parts by the parser. However, as the order positions may be spread across several lines, the Repeating Record type is to be used instead of the Record type. As the other lines also contain order positions and therefore do not offer new information about the structure of the flat file, they might as well be ignored, as defined by using the Ignore element type.


img

Figure 23 – Describing individual fields in detail


The next page is called Schema View (Fig. 24). On this page, the developer chooses the (Repeating) Record which they want to describe next in more detail. Subsequently they repeat the steps described until they have described every (Repeating) Record of which the flat file is composed. Thereby it must be taken into consideration that the positions of the individual fields must be entered instead of a delimiter for the order line definition.


img

Figure 24 – Selection of the (repeating) record to be defined next



Retrieving Data via FTP

With FTP Source the current CTP now offers the possibility to retrieve data from an FTP server. Besides the definition of the required reference data, such as address of the FTP server, username, password and directory, it also allows the definition of the expected code, the use of SSL and the restriction of the files to be read using a data mask (for example, *.txt to read only data with the ending txt). Moreover, it can be defined if the messages to be downloaded are available as flat files or in the form of XML documents.

The new CTP, however, does not provide the possibility to send data to an FTP server, although it is expected that one of the new (pre-)versions will offer this feature.


Tracking

To find out which actions the service bus has carried out when the message undergoes processing, and where potential failures may have occurred, the developer can now request tracking for every bridge. To do so, he selects Track Properties in the property window of the desired bridge. This property is based on the dialog shown in Fig. 25. The Track Data Flow option activates tracking. Moreover, the data extracted from the message which should be recorded for tracking can be defined in the check box which is located below.

Apart from this information, certain events are always tracked - even when the developer does not select any properties to be tracked - for example if the processing of a message fails.


img

Figure 25 – Definition of the tracking behavior of a bridge


Currently, there is no GUI to retrieve the recorded information, but the developer can access it using tools such as Fiddler [REF-5] through a REST-based interface. The request in Listing 7, for example, provides all information which has been recorded by the XmlOneWayBridge1 bridge in the namespace steyer-edi. Thereby, the space represents a security token received by Azure Access Control Service (ACS) [REF-6] which grants access to the service bus. This token may also be received through a REST interface. A simple application automating this process is appended to this article.

			
Listing 7
GET https://steyer-edi.servicebus.appfabriclabs.com/
XmlOneWayBridge1/tracking/default Host: steyer-edi.servicebus.appfabriclabs.com Authorization: WRAP access_token="[...]" End

To specifically access certain information, the developer can make use of an OData compliant URL parameter. For example, including the query string ?$filter=EventLevel+eq+'Error' only produces recorded information containing the description of errors. An overview of the filter possibilities offered by the service bus is to be found in [REF-7].


Importing XML Schemas from WSDL

The current CTP also facilitates the integration of SOAP-based services by allowing the import of XML schemas located in WSDL documents. To make use of this possibility, the Add Schemas from Service option is to be selected from the context menu of any given folder in Solution Explorer. When doing so, the developer may choose between providing a local WSDL file and providing a URL leading to a service endpoint based on WS Metadata Exchange. Unfortunately, WSDL files referring to external XML schemas only cannot be processed unless they are downloaded and made available to the import routine.


Conclusion

The options included in the CTP of December 2011 make Azure Service Bus a complete and highly available integrated solution on a pay-per-use basis. It helps Microsoft create an attractive USP among the cloud providers, even more so as the cost-extensive and time-consuming purchase and installation of fail-proof hardware and software solutions is rendered obsolete.

Many of the possibilities included come from Microsoft BizTalk Server. Therefore proven and tested developments and experience, such as maps or schemas for EDIFACT and X12, are used. But it also can be noted that concepts which, in the past, were not successful in the Service Bus area have been given a rethink and replaced by new ones. The not too successful Specific Language domain, for example, used to define itineraries and which is to be found in the BizTalk ESB Toolkit, has been replaced by the bridge configuration which can be used intuitively. For these reasons, among others, the present CTP already seems to be well-advanced and has whetted users' appetite for an on-premise version which would enhance the poorly equipped ESB market in the .Net area.


References

[REF-1] Azure Service Bus CTP http://www.microsoft.com/en-us/download/details.aspx?id=17691

[REF-2] Labs Portal, http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=17691

[REF-3] TPM-Portal, https://edi.appfabriclabs.com

[REF-4] NuGet, http://nuget.codeplex.com/

[REF-5] LOB-Adapter http://www.microsoft.com/biztalk/en/us/adapter-pack.aspx