As a middleware expert Jürgen works at Oracle EMEA Alliances and Channels, responsible for Oracle’s EMEA fusion middleware partner business. He is the founder of the Oracle SOA & BPM and the WebLogic Partner Communities and the global Oracle Partner Advisory Councils. With more than 5000 members from all over the world the Middleware Partner Community are the most successful and active communities at Oracle. Jürgen manages the community with monthly newsletters, webcasts and conferences. He hosts his annual Fusion Middleware Partner Community Forums and the Fusion Middleware Summer Camps, where more than 200 partners get product updates, roadmap insights and hands-on trainings. Supplemented by many web 2.0 tools like twitter, discussion forums, online communities, blogs and wikis. For the SOA & Cloud Symposium by Thomas Erl, Jürgen is a member of the steering board. He is also a frequent speaker at conferences like the SOA & BPM Integration Days, JAX, UKOUG, OUGN, or OOP.
Berthold Maier works in the T-Systems International department of Telekom Germany as Enterprise Architect. He has more than 19 years experience as developer, coach and architect in the area of building complex mission critical applications and integrations scenarios. Within eleven years as Oracle employee he has held several leading positions including chief architect in the consulting organization. Hi is the founder of many frameworks and take over the responsible for reference architectures around BPM/SOA and Enterprise Architecture Management. Berthold is also well-known as a conference speaker, book author and magazine writer.
Hajo Normann works for Accenture in the role of SOA & BPM Community of Practice Lead in ASG. Hajo is responsible for the architecture and solution design of SOA/BPM projects, mostly acting as the interface between business and the IT sides. He enjoys tackling organizational and technical challenges and motivates solutions in customer workshops, conferences, and publications. Hajo leads together with Torsten Winterberg the DOAG SIG Middleware and is an Oracle ACE Director and an active member of a global network within Accenture, as well as in regular contact with SOA/BPM architects from around the world.
Rolf Scheuch is co-founder of OPITZ CONSULTING and as Chief Digtial Officer responsible for the strategic development concerning the service portfolio and the necessary technology knowhow. Beside this Rolf is still active as a management coach with focus on developing a business goal based IT-strategy and establishing an effective "business and IT alignment". Furthermore Rolf is well-known speaker at numerous conferences and an author of articles and books with topics like Master Data Management, Process controlling and BPM.
Danilo Schmiedel is one of the leading BPM and SOA System Architects at OPITZ CONSULTING. He has been involved in large integration-, business processes automation and BPM / SOA development projects where he implemented solutions for various customers. His main field of interest is focused on the practical use of BPM and SOA on a large scale. Additionally he works as BPM and SOA project coach. Danilo is a frequent speaker in the German Java and Oracle communities and has written numerous articles about the above topics. Before joining OPITZ CONSULTING Danilo worked as Software Engineer in several international projects. The Leipzig University of Applied Science has awarded his outstanding reputation in 2009.
Guido Schmutz works as Technology Manager for the IT services company Trivadis. He has over 25 years as a software developer, consultant, architect, trainer, and coach. In Trivadis he is responsible for SOA, BPM and application integration, and is head of the Trivadis Architecture Board. His interests lie in the architecture, design, and implementation of advanced software solutions. He specializes in Java EE, Spring, Oracle SOA Suite and Oracle Service Bus. He is a regular speaker at international conferences and is the author of articles and several books. Guido is an Oracle ACE Director for Fusion Middleware & SOA.
Bernd Trops is a Senior Principal Consultant at Talend Inc. In this role he is responsible for client project management and training.
Bernd is responsible for all Talend projects within the Deutsche Post and the introductions of new versions and components.
Before Talend, Bernd was a Systems Engineer working on various projects for GemStone, Brocade and WebGain and therefore has extensive experience in J2EE and SOA. From 2003 to 2007 Bernd Trops worked as a SOA Architect at Oracle.
Clemens worked as Chief Architect for the Shared Service Centre, Global Business Services, Boehringer Ingelheim in architecture, master data, service management and innovation.
At the moment he works with holistic enterprise architecture that provides the methodological platform for the new master data management.
He previously worked as a Platform Architect at Oracle Inc. in the United States, where he helped to develop next product strategy as well as the SOA BPM Suite.
Torsten Winterberg works for Oracle Platinum Partner OPITZ CONSULTING. As a director of the competence center for integration and business process solutions he follows his passion to build the best delivery unit for customer solutions in the area of SOA and BPM. He has long-time experience as developer, coach and architect in the area of building complex mission critical Java EE applications. He is a known speaker in the German Java and Oracle communities and has written numerous articles on SOA/BPM related topics. Torsten is part of the Oracle ACE director team (ACE=Acknowledged Community Expert) and leads the DOAG middleware community.
MDM and SOA: Be Warned! Published: December 19, 2013 • Service Technology Magazine Issue LXXIX PDF
"You will waste your investment in SOA unless you have enterprise information that SOA can exploit."
This quote from Gartner, Inc. describes the relationship between service-oriented architecture (SOA) and master data management (MDM) very vividly. An essential principle behind SOA is the reuse of components and the flexibility of using these components to support new functionalities or processes. MDM provides universal components (or services) for consistent data maintenance and distribution. Here the architecture concepts and principles of SOA run into MDM.
This article begins by giving a brief motive for using MDM and a conceptualization. It will then go on to present typical variants for possible MDM architecture concepts, and illustrate the interplay of MDM and SOA with reference to the architecture pattern.
Increasing pressure from competition means that business models and underlying business processes have to be adapted in ever shorter cycles. At the same time, globalization and the digital networking of companies are making interaction with external business partners even more complex. Securely exchanging high-quality data is crucial for increasing efficiency in processes. This is where the central issue, the quality of information, and therefore its security in transactions, evaluations, and reports, all stem from. Once a company is no longer in a position to provide a consistent and consolidated view of its central business objects, implementing a company-wide policy for master data management becomes a good idea.
Unfortunately, in many companies today it is common for IT systems to be unable to keep up with fast changes in organization, business, and even technology. As a result, on the companies' side, a vast, ever-growing web of IT systems with known integration problems comes into being. This heterogeneity accounts for a variety of challenges when using master data that include differences in:
The result of these problems is inconsistent information about underlying business objects, the master data. As we have already mentioned, data is essential input for all processes. If this source of information runs dry or costs for the interaction based on the data become too high, the value of the information becomes questionable and the company becomes faced with serious concerns. MDM on the business side and the SOA approach on the IT side can counteract this problem together.
BARC's analysts assert: "Over half of all IT experts believe data quality … to be the biggest challenge". BARC goes on to say that "75 % of participants think master data management is the most important trend for their company" [REF-1].
In general use of the IT language, there is a common, basic understanding of master data and its importance for the company and the IT systems concerned. Master data form the basis of all business activities in a company and its business processes. Master data describe the basic business objects of a company, and should therefore be considered as a company's "virtual capital" [REF-2]. In his presentation at the 7th Stuttgarter Softwaretechnik Forum in September 2011, Professor Dieter Spath even claimed that information, and therefore also the master data, should be considered assets similar to a company's equipment and be subject to asset management [REF-3].
MDM organizes how the master data is handled, something that every company needs in order to carry out its business processes.
In the literature, there is a broad range of definitions for master data and master data management [REF-4] [REF-5] [REF-6]. This article uses the following definitions:
MDM is implemented, in-line with the definition, company-wide. For this to work, companies reassess the business processes and IT systems with regard to the changed use of the master data. Furthermore, the initiatives and projects of MDM itself are monitored and controlled via a management system. Finally, the strategy for the MDM is embedded into the general company strategy, and thereby contributes indirectly to improving added value. The introduction of MDM and its sustainable operation is a business transformation [REF-7].
For this reason, MDM is not an IT project, but rather a development plan for business. IT provides an infrastructure for this business transformation and plays a part in the MDM development plan, although control and initiation should be conducted through business.
Generally, reference frameworks are used for structuring and communicating complex correlations. The reference framework presented here structures the essential elements of the MDM development plan and subsequently serves as the framework for the procedure. The following explanations are based on business engineering, an approach that was developed by the Institute of Information Management at the University of St. Gallen for designing business transformations based on the strategic use of IT systems [REF-8].
Within its reference framework, business engineering considers three levels: the strategy, the organization, and the system architecture (data and applications). These areas have to be designed to successfully execute a business transformation in general, and therefore also within the framework (Figure 1) of the planning and performance of the MDM development plan [REF-9].
Figure 1 – The reference framework for the MDM development plan.
The instruments for formulating the vision and strategy are used in most cases to communicate and control medium or long-term development plans with organizational changes. The MDM team creates an overall concept or vision for organizing and managing MDM. This concept conveys the purpose of the MDM, explains the reasons for the change, outlines the goals, and describes guidelines for its use. It must be ensured that the MDM concept does not contradict the established company goals.
The operationalization at an abstract level stems from this vision, through the formulation of the strategy with initiatives for the MDM. The strategy specifies the fields of activity and mirrors the wishes and ideals of specific decision-makers. In connection with the vision, the strategy describes the expectations of the future situation. Finally, as part of the MDM strategy, the road map and milestones are developed and initiatives for the accompanying change management are defined.
The MDM development plan is a subject that embraces the whole company. MDM activities, processes, functionality, and structures must be coordinated across the different business areas of a company. To achieve this, the MDM requires a management system and process and structural organization to sustainably guarantee its success. The MDM requirements defined in the functional architecture form the basis for designing the process, structural organization, and the necessary IT support. The MDM management system operationalizes the development plan for the MDM strategy. It determines the point of departure for establishing the MDM, defines the processes and organization, and matches the assignment of key data to the processes.
At the heart is the adaptation of the existing process organization required for use as part of the MDM. The standards and parameters associated with the master data must be integrated into the company's operating and recurring work cycles. On the one hand, this affects the operating core processes and their activities, which users perform as part of their line function or roles. On the other hand, MDM-specific administrative processes and data governance must be implemented to ensure operational capability and continuous improvements in how master data are used.
A suitable structural organization forms the basis for processes. Employees are hierarchically included in the structural organization, according to their roles in the processes. This may be in their original line function or in a business reporting line, such as in the form of a matrix organization. The functional architecture structures the business requirements for the MDM, and acts as a basis for architecture decisions and the planning of necessary MDM processes and IT components.
MDM System Architecture
As a business transformation, MDM pursues the goal of implementing master data management across the company. To enable this at justifiable operating costs, IT must support the process. This applies, on the one hand, to the manually supported processes of the MDM itself, and, on the other, to the automated processes of data processing and distribution.
For this, a clear system architecture that includes the interdependencies of the systems is necessary. The system architecture for the MDM describes both the current situation and the planned target architecture. If the enterprise architecture approaches are followed [REF-10], the following results are meaningful for the MDM development plan:
The design areas include the application architecture with the necessary MDM-specific systems, supporting IT components, the integration architecture for master data logistics, and the underlying system infrastructure. The application systems and candidates for MDM are checked to ensure they provide the functionalities, and assessed with appropriate criteria. The application and integration components are based on an infrastructure platform that is considered separately from the infrastructure architecture. The information architecture performs a special role with MDM. The information architecture describes the master data objects, associations, and their attributes, in addition to cross-company information flows (flow of values and commodities).
The importance of the MDM's data and metadata means that this must be anchored as a design area within the framework [REF-11]. The information architecture models support the other design areas:
In the meantime, the organization and execution of an MDM development plan in the form of a program represents a best practice approach:
"Therefore the greatest challenges to success are not technical—they are organizational. It is largely because of these issues that MDM should be considered as a program and not as a project or an application" [REF-12].
From this point forward, this article limits itself to considering the system architecture and the technical aspects of using SOA and MDM.
SOA & MDM
An essential advantage to SOA is the loose coupling of the IT components. This promotes component reuse and makes it simpler and more flexible to use them to support new functionalities or processes [REF-13].
MDM should be based on service-oriented concepts and provide universal components (or services) for consistent data maintenance and retrieval of master data. Here the architecture concepts SOA are again incorporated into MDM. There are two different views supported by this claim:
MDM Business Service
Accessing master data objects such as products, customers, or business partners is necessary throughout all areas of the company, and thereby across the function and management areas: "Their high reusability and the comparable ease with which standardization takes place mean that access to master data objects creates ideal service candidates." [REF-7 14].
This opinion is also held by the Masons of SOA, who regard accesses to master data objects as "business entity services," and highlight that master data services amount to a considerable proportion of the developed services in SOA [REF-11 15].
A trend can also be detected here as well:
"The bundling of observed master data services into independent application domains anticipates the development of application architectures. In the future, central master data systems which offer services for accessing global master data objects will play an even more important role in application architectures" [REF-16].
Figure 2 demonstrates the transformation from uncontrolled to controlled management of master data:
Figure 2 – The transformation to controlled management of master data.
This results in MDM becoming a fundamental component of SOA. Establishing an SOA in the company increases the frequency of the use of central master data services, and thereby implicitly their reuse. Establishing central, managed services for accessing and processing data is therefore a sensible next step.
Performing a service decomposition leads to the discovery that, as part of the reusable services, services that undertake central tasks for managing the lifecycle of master data across the service domains are also required.
In this context, typical difficulties with managing and maintaining the master data are also encountered at the data access level, even independently from an MDM development plan. They include:
In summary, the service-oriented concepts act as leverage for MDM through the:
MDM Information Services
Service decomposition requires central master data services. These support the secondary, more strategic approach to implementing an Information-as-a-Service (IaaS) [REF-17].
Figure 3 demonstrates their support of IaaS:
Figure 3 – MDM supports IaaS approaches.
The underlying idea is very simple. A facade which delegates access to the IaaS is used so that single applications or services do not have to individually implement access to the data. This can be considered as a virtualization of data access, as the data sources are now transparent at the layer to be accessed. This provides central control over the typical CRUD operations on the required master data. Control over validations is now guaranteed, and any inconsistencies in the maintenance of different applications and services are resolved.
If this approach is taken as part of MDM, the three underlying challenges related to the "information-as-a-service (IaaS)" approach can be managed through technical and organizational measures.
From the consumer's point of view, this means:
In Table 1, the central services of MDM are contrasted with the eight factors that determine master data value:
Table 1 – MDM central services are contrasted against the eight factors.
This eliminates the two problems that are typically encountered when managing heterogeneous master data in different applications that have varying functionalities and decreasing data quality:
Data Storage Architecture Pattern
The type of architecture that is selected for data storage has a far-reaching impact on MDM. There are two main approaches for storing and managing master data [REF-18]:
SPoT stands for Single Point of Truth, which is sometimes referred to as a "golden record" [REF-19]. Ultimately it means the existence of a unique, valid master data object.
However, both approaches seldom exist in a "pure" form in practice. They are more likely to be used in combination with other technologies. In this respect, systematization occurs as is demonstrated in Figure 4, in three different approaches to data storage:
Figure 4 – The underlying architecture pattern for MDM.
In this architecture pattern, master data is maintained through one or several of the leading business applications for master data. Here, an MDM solution that is implemented as something of an independent, central application, should also be understood as a managing business application. The global master data, the guarantee of the master data quality, and the processes for maintenance are determined through the managing business applications. Master data is propagated to the various, mostly local, applications by a system for master data logistics. This guarantees that the master data remains consistent.
This approach is widespread in practice, and allows for centralized control which standardizes the master data company-wide. The data is consistently changed through one or more business applications, and underlies the identical master data quality management or methods for guaranteeing quality.
The approach of consolidating master data on a platform that is so frequently described in literature corresponds to this approach, whereby the managing application in this case corresponds to the data management process. The master data model is harmonized, as it is central. This approach is extremely good for company units that can agree jointly on global master data. The definition of a SPoT is through the managing system or the central database.
This approach assumes a high level of autonomy for the different system worlds, and decentralized implementation. As with database systems that are based on the "inverted lists" principle, there is a central list with references ("registry") which are required for the unique identification of a master data object specification. The actual pieces of information (master data values) are distributed between the different systems. In this concept, reading master data is connected to a high network load per retrieval instance, and is therefore recommended more for MDM approaches that provide a low degree of overall master data views.
A central problem can occur in this approach, in the area of data quality management. Since the information is managed and stored in the decentralized systems, distributed governance is needed. This circumstance almost inevitably leads to greater autonomy for the distributed units and a decline in centralized control. Only the consistency of the attributes for identification has to be guaranteed to be standardized through all closed systems.
One weakness of the registry-based approach is the high network load. One weakness of the centralized approach is the centralization of the functionality through managing business applications. A decentralized approach which follows federal principles and permits coexistence with the existing systems allows these weaknesses to be counterbalanced. It resembles the master/master and master/slave relationships with replications in a database.
In this case, local copies in the distributed MDM systems are also provided and reconciled using master data distribution logistics. The copies and replications can thereby be "subtly" adjusted to the necessary degree. As with the registry approach, DQM takes place locally but with global starting points, as all relevant data are kept in the master in the "middle" (similar to the consolidated approach).
The weak points of this concept are already known from the replication environments. There are problems with control through successful handshaking as well as with the restart points during interruptions. The security of the data in the network cluster is also not sufficiently guaranteed. From our point of view, this approach requires a considerable number of interventions in the existing infrastructure.
Scenarios for Use
The use of SOA approaches will now be considered in light of the different architecture patterns, with different scenarios for use discussed individually.
The SOA leverages are lowest for this architecture approach. Data management is through a business application that is often encapsulated. This application indicates the necessary processes for maintaining the master data and contains the routines for guaranteeing data quality. Data distribution mostly takes place as an ETL process, or as the conventional and established EAI approaches (Table 2).
Table 2 – The specifications of the Centralization architecture pattern.
Using SOA is vitally important with this architecture approach (Table 3). The propagation of master data is in a distributed environment, and generally uses the transaction or dataset-oriented procedure for propagating changes here. To display the information, different services need to be activated through the network. These services read the information that belongs to the identifying data from the registry in the specific operational systems. This data then arrives back at the starting point via the network, where the read data is compiled into a master dataset.
An integration procedure based on loose coupling offers advantages here. Each change to the structure or to a version of the master data from the local systems must be reported to the central registry in a similar manner.
Table 3 – The specifications of the Registry architecture pattern.
The use of SOA is essential for this architecture approach as well (Table 4). Data storage is in distributed systems and also in distributed data storage. Consequently, the maintenance and management of master data takes place in a distributed environment, and each system starts the propagation of master data logistics. Replication itself can take place asynchronously or synchronously to enable different approaches even in the SOA on the side of implementing the replication, including conflict resolution (similar to the "registry approach").
Table 4 – The specifications of the Coexistence architecture pattern.
In conclusion, the question still remains as to whether you could pursue a consistent SOA approach without any MDM. Andrew White and Jon Radcliffe (Gartner, 2010) believe that is not possible. In their opinion, the success of SOA projects stands and falls with the use of MDM:
"Through 2011, 70% of SOA projects in complex, heterogeneous environments will fail to yield expected business results benefits unless MDM is included."
We will close with that judgment.
[REF-1] BARC: "Data Warehousing 211 – Status Quo, Herausforderungen und Nutzen, White Paper," BARC Institut, p. 28, Würzburg, July 2011. p. 28, 35
[REF-2] M. Kaufman: "Master Data Management"
[REF-3] D. Spath: "Unternehmensgut: Stammdaten" in the conference transcript from the Stuttgarter Softwaretechnik Forums 2011, Stuttgart, Frauenhofer Verlag, 2011. p. 29
[REF-4] A. Berson, L. Dubov: "Master Data Management and Customer Data Integration for a Global Enterprise," McGraw Hill, 2007
[REF-5] A. Dreibelbis, E. Hechler, I. Milman, M. Oberhofer, P. Van Run, D. Wolfson: "Enterprise Master Data Management: An SOA Approach to Managing Core Information," 2008, IBM Press. p. 23
[REF-6] B. Otto: "Funktionsarchitektur für Unternehmensweites Stammdatenmanagement, Bericht Nr.: BE HSG/CC CDQ / 14," Institute of Information Management at the University of St. Gallen, 2009
[REF-7] J. W. Schemm: "Zwischenbetriebliches Stammdatenmanagement," Springer, 2009. p. 79 et seq.
[REF-8] H. Österle, R. Winter, F. Höning, S. Kurpjuweit, P. Osl: "Business Engineering: Core-Business-Metamodell," in: "WISU – Das Wirtschaftsstudium," 2007
REF-9] B. Otto: "Funktionsarchitektur für Unternehmensweites Stammdatenmanagement, Bericht Nr.: BE HSG/CC CDQ / 14," Institute of Information Management at the University of St. Gallen, 2009. p. 14
[REF-10] Staehler et al.: "Enterprise Architecture, BPM und SOA für Business-Analysten," Hanser, 2009
[REF-11] J. W. Schemm: "Zwischenbetriebliches Stammdatenmanagement," Springer, 2009. p. 79 et seq.
[REF-12] M. Kaufman: "Master Data Management". p. 16
[REF-13] T. Erl: "SOA Design Patterns," Prentice Hall Service-Oriented Computing Series. p. 756-757
[REF-14] J. W. Schemm: "Zwischenbetriebliches Stammdatenmanagement," Springer, 2009. p. 223
[REF-15] B. Maier, H. Naumann, B. Trops, C. Utschig-Utschig, T. Winterberg: "SOA Spezial Ready for Change," Software und Support Verlag, 2009. p. 35 et seq.
[REF-16] J. W. Schemm: "Zwischenbetriebliches Stammdatenmanagement," Springer, 2009. p. 223
[REF-17] Allen Dreibelbis, Eberhard Hechler, Ivan Milman, Martin Oberhofer, Paul Van Run, Dan Wolfson: "Enterprise Master Data Management, An SOA Approach to Managing Core Information," 2008, IBM Press. p. 86 et seq.
[REF-18] A. Dreibelbis, E. Hechler, I. Milman, M. Oberhofer, P. Van Run, D. Wolfson: "Enterprise Master Data Management, An SOA Approach to Managing Core Information," 2008, IBM Press. p. 23
[REF-19] M. Kaufman: "Master Data Management"