img
Mandy Chessell

Mandy Chessell

Biography

Mandy joined IBM in 1987. She is an IBM Distinguished Engineer, Master Inventor and member of the IBM Academy of Technology Leadership Team. Her current role is the Chief Architect for InfoSphere Solutions in the IBM Information Management CTO office. She leads the design of common information management patterns for different industries and solutions. This includes the Next Best Action solution and the strategy for Information Virtualization.

In earlier roles Mandy led the development of new features for the CICS, Encina, TxSeries and WebSphere products. She has over 50 issued patents worldwide in the fields of transaction processing, event management, business process management and model driven development.

In addition to her technical responsibilities, Mandy is involved in initiatives designed to enhance technical vitality with the IBM technical community. This includes mentoring, serving on technical career development and promotion boards along with Women in Technology initiatives.

Outside of IBM, Mandy is a Fellow of the Royal Academy of Engineering and a visiting professor at the University of Sheffield, UK. In 2001 she was the first woman to be awarded a Silver Medal by the Royal Academy of Engineering and in 2000 she was one of the "TR100" young innovators identified by MIT's Technology Review magazine. In 2006 she won a British Female Innovators and Inventors Network (BFIIN) "Building Capability" award for her work developing innovative people and the BlackBerry "2006 Best Woman in Technology - Corporate Sector" award. More recently she was granted an honorary fellowship of the Institution for Engineering Designers (IED), she won the "2012 Cisco everywoman Innovator of the Year" and in 2013, Plymouth University awarded Mandy a Honorary Doctor of Science.

Contributions

rss  subscribe to this author

Harald Smith

Harald Smith

Biography

Harald Smith is currently a Software Architect with IBM, with a diverse 30 year career specializing in information quality, integration, and governance products and solutions and a long-time interest in patterns and design practices. His work has spanned software product management, information technology, consulting services, technical support, system auditing, and business process re-engineering, and includes 4 issued patents worldwide. He writes extensively including his current blog "Journeys in the Information Landscape", particularly on methodology, best practices, and capabilities with IBM products and the topics of Big Data and Information Governance, and is an IBM developerWorks Contributing Author. Harald is also an IBM Certified Solution Developer in Information Management.

Contributions

rss  subscribe to this author

Bookmarks



Design Patterns for Information in a Service-Oriented Architecture Published: October 24, 2013 • Service Technology Magazine Issue LXXVII PDF

Abstract: Organizations are driven by the information they manage. Organizations in this fast-paced world need to be more agile and more integrated, which is where service-oriented architecture adds value. Behind the services, there are systems that manage the information exchanged across the services.

This article is about design patterns for information management techniques that improve the information services used by your organization. It describes a pattern language called the Patterns of Information Management (PoIM). PoIM contains over 200 linked design patterns that use consistent terminology and concepts, resulting in a body of work that can be used at all levels, from the enterprise architect down to the individual project.

An example design pattern for an enterprise service bus (ESB) is used to illustrate the structure of a design pattern and how such a pattern links to other PoIM patterns to create a pattern language.

Managing Information

The success of service-oriented integration is determined by the quality of the information exchanged across the service interfaces. If the information is useful and reliable but difficult to get by any other means, then callers will flock to connect to a shared service.

Information that is core to the operation of an organization, such as information about customers, products, assets, and contracts, is often duplicated across many systems, since each system needs access to this information to perform its tasks. Where this type of information resides within any one system, it reflects the silo (typically the line of business) in which the system operates, leading to inconsistencies in this information across the different systems.

Consider creating a service that returns up-to-date customer details in a situation where a customer has multiple accounts and interacts through multiple channels, but where each account and channel are implemented by different systems. A centralized server that can consolidate, de-duplicate, and resolve inconsistencies is needed to integrate the details for each customer. This server (called a master data management hub) provides authoritative information through services and synchronizes this information with other systems, creating a consistent view of customer data across the organization.

Similar consolidation techniques are deployed with historical data for reporting and analytical services when we create data warehouses and data marts. And more recently, social and machine-generated data can be captured, synthesized, and made available through services using Big Data technology.

Information governance is the process wherein standards and policies relating to the management and use of information are regulated. Its supporting technology and practices assure the quality and protection of information throughout its lifetime.

All of these techniques come from the information management discipline. They are invaluable to enterprise architects and solution architects, but little understood. This article is an introduction to these design patterns.

Introduction

As architects, we wish to use best practices and proven approaches to create robust solutions for our organizations. Those of us who have been in the industry for many years know there are fundamental principles of software engineering that do not change, even though the emergence of new technology seems endless.

When we document best practices, it is important to separate the fundamental principles from the technology's specifics. Then, when we encounter a new technology, its behavior can be mapped to the fundamental principles and special note is made of the technology's specifics that occur.

For example, Apache Hadoop provides a new type of repository and processing mechanism for processing information. When we compare it to existing database technology, such as a data warehouse, it is obvious that these are very different technologies.

However, there are similarities:

  • Both are repositories so information must be fed in and retrieved.
  • The information within the repository must be appropriately protected and removed when it is no longer needed.
  • The level of quality of the data affects how it can be used.

The major differences between the two are:

  • The parallel processing paradigm in the programming model for Apache Hadoop, versus the set-based programming paradigm by which data warehouses are designed.
  • The fact that Apache Hadoop is a schema-on-read type of system, whereas the data warehouse sets the schema of the data before it is written to the repository.

Design patterns document best practices, or architectural principles, in a consistent manner. They assemble a variety of information using a consistent terminology that helps us extract and understand the significant similarities and differences between technologies and approaches.

In this article we illustrate the value of design patterns using the Patterns of Information Management (PoIM) pattern language. PoIM covers the design patterns for managing information in distributed and interconnected systems.

Below is an example design pattern for an enterprise service bus (ESB). It was chosen as the example since it is most likely an architectural approach that you are familiar with and is a logical extension to the PoIM pattern language. This specific pattern is not included in the PoIM pattern language but is written in the same style, using the component patterns from the patterns of information management pattern language. As such, it is written in a technology-agnostic style.

The patterns in PoIM are grouped into pattern groups. The enterprise service bus pattern has been added to the Information Solution pattern group. The lead pattern in a given pattern group is named after the pattern group. It describes a generic design pattern while the rest of the patterns in the group are specializations. Information Solution patterns are composite patterns, built up from other underlying information management design patterns.

Table 1 shows the pattern summaries (also known as patlets) for Information Solution and Enterprise Service Bus:

Icon Pattern Name Problem Solution
img Information Solution An organization recognizes there is a missing capability, or a major issue with the way they manage an aspect of their information. Create a project, or series of projects, to transform the way the information is managed by the organization's people and information systems.
img Enterprise Service Bus The point-to-point integration between systems is complex whenever the needs of the service requesters are inconsistent with the capabilities of the service providers, both in terms of the type of connection that is used and the scope of the functions available. Create an intermediary system to act as a broker between the service requesters and the service providers.

Table 1 – A summary of the Information Solutions Pattern Group is displayed [REF-1].

Notice that each pattern has a name and an icon. The name should be descriptive and easy to remember. The icon is a visual reminder of the pattern's content. We also use the pattern icons in the pattern descriptions when a pattern is being used as a component in the pattern's solution. The pattern summary includes a summary of the problem that the pattern solves and the recommended solution.

The full pattern description for the enterprise service bus is documented next, with the following subsections:

  • Context – the circumstances under which you would consider using this pattern
  • Problem – a description of the problem this pattern solves
  • Example – an example of the problem
  • Forces – the factors that make this problem tricky to solve
  • Solution – and explanation how the pattern recommends that you solve the problem
  • Consequences – the benefits and liabilities associated with using the pattern
  • Example Resolved – illustration of how the pattern is used to solve the problem illustrated in the example shown
  • Known Uses – where has this pattern been used or documented
  • Related Patterns – descriptions of other patterns that are relevant to this pattern

Each section adds another dimension to the understanding of the design pattern's intent and potential value.

The Enterprise Service Bus Pattern

Context

An organization has adopted a service-oriented architecture as their system integration strategy, and is experienced with using service-oriented integration for point-to-point exchange of messages between two systems.

Problem

The point-to-point integration between systems is complex whenever the needs of the service requesters are inconsistent with the capabilities of the service providers, both in terms of the type of connection that is used and the scope of the functions available.

Example

A fictitious retailer called MCHS Trading has three systems accepting orders from customers. These are called E-Shop, Mail-Shop, and Stores. The orders flow to a Shipping system that communicates with the appropriate warehouse to request that the ordered goods be sent to the customer's address. The order is also sent to the Invoicing system to ensure that the order is paid for.

Figure 1 illustrates how these order-processing systems are connected. No single system sees the end-to-end order processing process:

img

Figure 1 – MCHS Trading's order-processing systems [REF-2].

MCHS Trading has recently introduced two Master Data Management (MDM) hubs. Customer Hub is a system for synchronizing customer data between E-Shop, Mail-Shop, and Stores. Product Hub is the system where products are chosen for the catalogues, descriptions and images of each product are perfected, and suppliers are selected. Subsets of this product information are distributed read-only to the other systems.

MCHS Trading wishes to introduce a new Customer-Care system to help employees answer questions on the status of orders, record issues with products, and update customer details. This is the first stage for providing a social media interface for customer service, enabling customers to check on their orders and chat with employees electronically.

How should the Customer-Care system connect with the other systems, especially given the fact that each system was introduced at a different point in time and supports different communication mechanisms?

Forces

  • A call to a remote service has a higher latency than a call to a local function, because of network delay and the time taken to package and unpackage the messages that must flow to describe the service request and the service response.
  • Each system typically uses different data structures and data verification rules. These must be reconciled between systems that are using service-oriented integration.
  • There are many communication mechanisms available to enable systems to exchange messages with one another. These mechanisms have each been in vogue at different times. A system is likely to only support the communication mechanisms that were in vogue when it was commissioned. Many organizations therefore find they have a polyglot of communication mechanisms that must be supported in order to integrate their systems together.
  • Each system may use its own security protocols and techniques.
  • There may not be a one-to-one match between the needs of the service requesters and the capabilities of the service providers.
  • The same information may be stored in multiple systems. However, the values may not be consistent and the identifiers used in each system typically follow a different scheme.

Solution

Create an intermediary system to act as a broker between the service requesters and the service providers.

This intermediary system is called an enterprise service bus. The implementation of each service within the enterprise service bus is responsible for issuing the remote calls to the appropriate systems and combining the results. It must handle the different communication mechanisms, security credentials, and data formats and standards.

This is illustrated in Figure 2, which depicts the following steps:

  1. The service requesters issue service calls as if they were calling the service providers directly.
  2. The service request is routed to the enterprise service bus.
  3. The service implementation inside the enterprise service bus calls the appropriate service providers to fulfill the request.
  4. Each service provider performs the requested function and returns the result to the service in the enterprise service bus. The results are combined and sent to the service requester.
img

Figure 2 – An enterprise service bus is being used as an intermediary.

Consequences

Some of the more important benefits of using this solution are:

  • The enterprise service bus can support the optimal services for the service requesters and hide the complexities of which systems need to be called in order to implement them.
  • The enterprise service bus can support a common information model for the services it supports, which provides consistent message structures for its services. For example, an address would always be formatted using the same structure.
  • The enterprise service bus can hide the variety of communication mechanisms supported by the service providers. It may also support multiple communication mechanisms across all of its supported services to provide choice to the service requesters.
  • The enterprise service bus can act as a control point for the governance of the agreed practices and standards laid out in the organization's service-oriented architecture. This could include security and related service level agreements.

Conversely, some of the liabilities associated with implementing an intermediary system are:

  • The enterprise service bus will add latency to the service calls, since the messages must make two network hops in either direction in order to pass through the enterprise service bus.
  • The enterprise service bus is another system to run and support in the organization.
  • In a decentralized organization, there may not be an obvious owner to develop and operate a single centralized enterprise service bus. When this is the case, each internal group may support its own enterprise service bus to shield its systems from the service calls from other groups and to implement any agreed best practices and standards.

Example Resolved

The Customer-Care system is given access to the services it needs through an enterprise service bus. These services offer clean interfaces to information about customers, their loyalty card status, products offered, the orders made, and how far the orders are through processing.

The enterprise service bus handles the implementation of these services. It is responsible for making the appropriate calls to the different systems involved in the order processing and extracting of the appropriate information. It also makes calls to the Customer Hub system for customer data. This system ensures customer information is kept consistent amongst the order taking systems: E-Shop, Mail-Shop, and Stores. For product information, the enterprise service bus calls the Product Hub since it is the most appropriate place to retrieve details of products and report problems that customers are having with a specific product.

The enterprise service bus hides the system complexity from the Customer-Care system. The services it offers are also adopted by other applications that need access to the core operational data of the organization (Figure 3).

img

Figure 3 – An enterprise service bus is being used to support the Customer-Care system.

Known Uses

The enterprise service bus is used by organizations that are experienced in using service-oriented integration between different parts of the business and wish to create further decoupling between the systems that are service providers and those that are service requesters.

Related Patterns

The Enterprise Service Bus pattern relies heavily on the PoIM Information Service pattern group that includes Local Information Service, Remote Information Service, and Triggering Information Service. Together, these patterns describe the substitutability of the remote information service for a local one, and how an information service triggers new work when called from a remote system.

The processing that is performed by the enterprise service bus runs on a specialized server. In the PoIM patterns, this server is described by the Information Broker pattern in the Information Node pattern group.

The processing that occurs inside the service implementations in the enterprise service bus follows the Information Federation Process pattern from the Information Process pattern group.

Moving beyond the Enterprise Service Bus

The enterprise service bus improves access to the organization's information and is able to hide the complexity of how function and information is distributed between the systems. What it cannot do is resolve inconsistencies in information stored in different systems, or errors or gaps in the information stored within a single system. These issues must be resolved using information management techniques and technology. Otherwise, service-oriented integration becomes an effective technique for distributing bad data further.

A Service-Oriented Architecture needs a companion Information Architecture to cover how information is managed within systems and ensure that what is exchanged across the services is valid.

The information architecture defines standards for information, such as a common information model comprised of:

  • semantic or business-focused descriptions (in text) of the types of information that the organization needs
  • common schema structures and standards for messages that are exchanged between systems
  • common schema and standards for data stored in shared databases, data warehouses, and data marts

It also covers the additional systems and integration mechanisms that are used to consolidate, validate, enrich, and synchronize information between systems so that their information is kept consistent. This integration capability is in addition to the services. We call this the information supply chain that defines how information flows between the systems and ensures that information synchronization remains efficient, with minimal latency. When an error in a data item is detected, it can be traced back through every affected system to the source and corrected.

Each type of information is typically synchronized using its own information supply chain. For example, Figure 4 shows MCHS Trading's information supply chain for product information that emits from Product Hub. Product Hub is the single source of product information and so is called a cascading information supply chain.

img

Figure 4 – The product information supply chain is shown in a flowchart [REF-3].

Figure 5 shows the information supply chain for customer information. In this case, customer information may originate from E-Shop, Mail-Shop, or the Stores system. It is sent to the Customer Hub, which is then responsible for matching it to existing information about the customer. Each system uses a different approach for identifying people and contains duplicate records for individuals, with different subsets of attributes filled in in each record. Once the Customer Hub has the new information about the customer matched up with existing information, it synchronizes this new data with each of the relevant records in the E-Shop, Mail-Shop, or the Stores system. This type of information supply chain is called the Hub Interchange Information Supply Chain.

img

Figure 5 – The customer information supply chain is illustrated in a flowchart [REF-4].

The information integration mechanisms must operate continuously in the background so that the information is in the right place when services are called on the participating systems.

Figure 6 summaries the pattern groups in PoIM:

img

Figure 6 – A summary of the Patterns of Information Management pattern group is shown [REF-5].

Conclusion

In this article we have provided an example of a design pattern for an enterprise service bus (ESB). We chose a familiar concept to illustrate how effective the design pattern approach is for documenting design guidance, and to whet your appetite for more detail on how information should be managed in a service-oriented environment.

References

[REF-1] Chessell, Mandy and Harald Smith, Patterns of Information Management, IBM Press, 2013, p.578

[REF-2] Chessell, Mandy and Harald Smith, Patterns of Information Management, IBM Press, 2013, p.33

[REF-3] Chessell, Mandy and Harald Smith, Patterns of Information Management, IBM Press, 2013, p.35

[REF-4] Chessell, Mandy and Harald Smith, Patterns of Information Management, IBM Press, 2013, p.40

[REF-5] Chessell, Mandy and Harald Smith, Patterns of Information Management, IBM Press, 2013, p.67

Book Attribution

Mandy Chessell and Harald Smith are authors of the book, "Patterns of Information Management", published by Pearson/IBM Press, ISBN 0-13-315550-1, May 2013, © Copyright 2013 by International Business Machines Corporation; for more info please visit the publisher page: http://www.ibmpressbooks.com/store/patterns-of-information-management-9780133155501