Abstract: In today's IT world, service-oriented architecture is not a new term for enterprises. It has been adopted widely and quickly by many. However, most SOA adoptions focus only on the pilot phase and production implementation to meet immediate business needs. Because of the requirements posted by the business team, very few are adopting SOA for enterprise standardization. Since most adoptions are triggered by business, IT departments do not have enough time to think through post-implementation strategies, such as infrastructure optimization. In addition, many of us have been practicing traditional software development methodologies for decades, which do not reflect the specific needs for SOA adoption phases in pre and post implementation. In reality, the payback and higher ROI will only happen if there is accurate monitoring, measuring and optimization during adoption phases, and proper governance must follow. This is where many enterprises have failed to have appropriate strategy and governance models. The purpose of this article is to shed light on the importance of service and process monitoring, measuring and optimization, which will lead to better results.
In order to understand the importance of optimizing the enterprise services and business processes, we should understand some common issues of SOA implementation. How can SOA help to monitor the business functions and processes? How can it provide opportunities to build mature business processes? In this article, we are going to explore the answers to these questions and more, including topics such as:
- Purpose of the service and business process optimization
- Measure to identify the service and business process improvements
- Service and business process optimization
- SOA governance lifecycle for optimization
- Key roles in service and business process optimization
- Benefits of service and process optimization
Purpose of Service and Business Process Optimization
Many enterprises are adopting a service-oriented approach to accomplish integrations and business process enabling. The availability of SOA tools, technologies and products will certainly help to build the enterprise services and business processes quickly. However, in many real situations, the SOA implementation team has to compromise or skip some (or most) of the service development guideline processes or best practices to meet the project timeline set by the business team. Because of this situation, first time service identifications and process developments might not be very accurate or optimistic. This is because of the complexity in implementing service-oriented architecture and business processes. In some cases, the implementation may not have a clear picture of the organization roadmap or other upcoming requirements. Implementing SOA at the enterprise level requires enormous amounts of vision and strategy to be successful on the first shot.
The following are the most common issues identified in service implementations. These are consolidated points from various SOA adopters.
- A developed service might not meet the whole purpose
- Scope of the service may be limited to a specific integration
- Service granularity level differences might require rework in the service interfaces
- There is less control on the business process or service so invocations may result in redundant process activity
- Services are not adopted in all the enterprise infrastructures or systems [REF-2]
- Process performance is very limited due to large payload size, too many numbers of service invocations
- Duplication of services leads to too many dependencies between services [REF-2]
Similarly, there may be some level of compromise that needs to be made while defining the business processes that run on top of the service infrastructure. Business processes should be defined based on the enterprise's business strategy and goals. However, most of the implementation will only focus on achieving the existing business processes that use the service-oriented approach. This approach results in a gap between the current version of the process and the expected version. Some of the common issues that would force the process development team to compromise during the process definition are as follows:
- If there is any compromise decision made in service enabling, then some level of compromise is required in defining the business processes that consume those services
- The business does not have a clear picture of the process requirements, which force the definition of redundant business processes
- New processes are introduced by the process development group that might require change in the flow of business functions
- Lack of process governance might trigger user objections on the processes
Therefore such compromises may address the immediate process and integration needs, but do not help to identify the opportunities for business process improvement and for leveraging IT assets. The service-orientation and business processes not only help to meet the process and integration needs quickly, but also help generate reports, measurements and findings that can improve opportunities. It is the business and governance team's responsibility to find the necessary budget and scope to fine-tune the service infrastructure after implementation.
Now, how do we identify the services and business processes that need to be optimized?
Measure to Identify the Services and Processes Needing Improvements
Either there is a scope for service optimization or not. Enterprise services and business processes should be monitored at runtime. Business Activity Monitoring (BAM) is a monitoring methodology with dashboard capabilities that helps to monitor all services and business processes [REF-4]. BAM can also help to capture the messages exchanged by interfaces in integration and in business processes. That message data helps to generate the business forecasting reports. In addition, the BAM dashboard provides business insights in the SOA operations, such as key performance indicator report, which help to identify the business and integration priorities and runtime control on the processes. This helps to optimize the business needs. Service monitoring is not new to an SOA approach. Even in an EAI approach, the audit mechanism is used in many implementations to capture the transaction information. If there is any message failure or redundant activity that happens in the SOA infrastructure, this approach will help drill down to find the reason for the failure. However, many SOA adoptions do not have effective monitoring mechanisms or use a BAM kind of approach, which makes the infrastructure less clear. If there are no monitoring or event notification mechanisms in the service infrastructure, then any debugging and data analysis will require enormous amounts of time and effort. Even if a service infrastructure includes BAM tools, it is not utilized effectively. Monitoring and measuring the performance of the activities that happen in the SOA and business process infrastructure will certainly help to optimize the service assets.
Other than monitoring service infrastructure, there are various ways we can identify the service improvement opportunities. However, these are not as efficient as implementing a monitoring and measuring approach.
Service and Business Process Optimization
Enterprise service and service-oriented business process optimizations are mostly related to each other in that they require improvements on both sides to gain higher results. If there are any performance, scope, granularity, or stability issues in the services, this could influence the performance of the business process that consumes those services to meet business objectives. In some cases, the granularity of the message that flows between systems can lead to performance slowdown or processing overhead in services.
Mostly service optimization needs will vary, but it is purely based on the metrics that are collected at runtime. Each of the services in an SOA layer should undergo quality tests in the area of service usability, service performance, service scope, and stability to collect performance metrics [REF-3]. Those test results should be properly analyzed to generate the metrics. The collections of service performance metrics should be consolidated to identify any improvement opportunities. There are two types of improvements that need to be addressed to meet the ultimate goal. They are technical improvements and functional improvements. The technical improvements are purely based on web service performance such as hardware performance, protocol type, connectivity model, parsing technique etc. The functional improvements include the service scope alignment, message model, granularity, process model etc.
Business process optimization should be a systematic approach to help the enterprise identify process improvement opportunities and fine-tune the processes for better business results [REF-1]. In a service-oriented world, process optimization requires monitoring the deployed business process during operation to find the key performance indicator. Business process optimization includes process utilization, performance, stability, availability, and state management.
Business process improvement can be achieved by the following methods:
- Assess the existing business processes and identify the new processes, simulate the process model by building a POC of that process, then validate
- Administrator the lifecycle of the business process by applying policies and standards
- Deliver the message data that flows through the business processes to the business team for validation again, keeping in mind the business goals
- Measure the performance of the services separately from the business process that consumes those services, and then compare the metrics for improvement
SOA Governance Lifecycle for Optimization
Typically, the SOA governance lifecycle should include phases like planning, defining, enabling, and measuring. We are not going to explore what we should do in planning, defining and enabling phases since these are common methodologies followed in the software development lifecycle. However, there is some misunderstanding about the measuring phase of the SOA lifecycle. The measuring phase is neither the maintenance phase of the implementation nor is it meant to just manage the SOA infrastructure by making the services and business processes active. It is purely about monitoring the performance of the business and integration services, the business processes, the SLA, and the data that flows through the services and processes, such as exceptions and reasons for those exceptions etc. Measuring the performance of the service infrastructure is important to improve the service assets.
Once we have the metrics of the service and business processes, it is important to optimize the services and business processes. However, do we need a separate phase for optimization? Certainly not. Optimization should also go through the phases of planning, defining and enabling. Once we have identified the list of services and processes that require re-architecture, we need to start planning the identification of the service for the new models or processes, such as the alternate data flow model, as well as the impact analysis etc. In some cases we may have to retire some of the services that are not being used by any integration or business process.
Figure 1 – SOA Governance Lifecycle that Supports Optimization.
As per the governance methodology in Figure 1, service optimization should go through the phases with the various tasks. Some of the common tasks are to identify the changes in the service layer, to validate against reference architecture, to plan the implementation, to design the service candidates, to build the service solutions, to design the business processes, to integrate the service and processes, to check the quality and performance, to simulate the processes, to deploy the upgraded process and services, as well as to retire the existing unwanted service etc. Therefore, the service governance lifecycle includes the phases to handle process improvement and optimization.
Key Roles in Service and Business Process Optimization
Measuring the service and business process performance in the enterprise infrastructure and optimizing them requires some level of concentration from both the business and IT teams. In order to make the whole optimization process a success, the following business and IT roles should play key contributions [REF-1]:
- Business leaders
- Process/Operations owners
- SOA Architect
- Business process architect
- Development managers
Business leaders are really leading the enterprise because they know the business model, processes and the potential growth, and therefore generate the business plan that has the strategic business target. As part of the business plan, they define the performance expectation for business processes. The goals of the service-oriented business processes are to meet the performance expectations defined by the business leaders. One of the indicator reports of the key performance indicator is a directional indicator [REF-5] that provides the performance of the entire business.
Therefore, the contribution from the business leaders is in setting up the performance goals and plans that help to identify the gap between the existing business processes and their corresponding services. The outcome of this analysis should be the list of services and business processes that require improvement or alteration.
Process or operations owners are responsible for defining the business process and operational activities required to meet the business plan set by the business leaders. In some enterprises, operations and process owners may be two different people, but the objective of both of these roles would be to meet the business plan. In some cases, the set of business operation activities that are defined by the operations owners will help to define the business processes. Process owners will design the business processes and own the processes. They actually simulate the processes to measure whether the designed process can help to meet the business objectives. All of the simulated processes should be implemented in the IT system using a workflow language or BPEL to make it like a system process.
Process owners should also help to identify the process improvement opportunities by validating the metrics generated during the measuring phases along with input from other sources.
The responsibilities of an SOA architect are many during the implementation phases. From an optimization prospective, they should play a key role in improving the service infrastructure by analyzing the metrics measured from the operations environment. Also they should: identify the service candidates for the defined business process, identify dependencies between services that are undergoing optimization, find the services that can play a role in the business processes, define the message model, provide the service-oriented solutions for new developments etc. SOA architects should also analyze the business processes defined by the process owners and work with the BPM architect to define the processes in the workflow language or BPEL. The SOA architect is also responsible for providing solutions to standardize the service models and to design for higher performance.
Business Process Architects
Business process architects predominantly play the role of transforming the business processes defined by the process owners into the form of workflow language or BPEL format. Business process architects should have a good knowledge of service-orientation, so they will be able to identify the services that can be used to achieve the business process requirements. Business process architects should work closely with process and operations owners. In the optimization phases, the business process architect should identify the improvement opportunities in the existing business processes and fine-tune the systems. They should also work with process owners to simulate business process optimization and measure the performance. Sometimes they will help the SOA architects define the services that are required to meet the business processes defined by the process owners.
The role of the development manager in the service and process optimization should be to plan and enable phases. They may not have good contributions in identifying the process optimization opportunities, but they must understand the changes required in the services and process components to meet the mature application. They will help to identify the efforts and timeline required to align the existing services and processes so that they meet the optimization objectives. Sometimes the process improvement requires a multi-phase approach. In that case, the development managers will help to track the changes through a change control process and will assign the appropriate tasks to the respective teams.
Benefits of Service and Process Optimization
The objective of service and business process optimization is to improve the enterprise operational capabilities to meet the business goals. In addition, the following are the benefits expected as a result of SOA infrastructure measurement and optimization:
- Will improve the performance of enterprise level services to meet higher SLA targets
- Will improve the business process performance. This should help meet the business functions more quickly and directly or indirectly boost the organization growth
- Business leaders will have a clear vision of the organizational activities and bottlenecks
- Key performance indicators will help to predict the organization's progress and sales channels
- Will help to mature the SOA infrastructure by fine-tuning the service capabilities
- Will help to build enterprise level standardization, data flow and data availability
- Will lower the new development and enhancement costs
- Will help to achieve higher ROI by increasing service reusability
Service-oriented architecture and business process adoption requires effort in monitoring services and measuring performance. This helps increase the efficiency of how the business and processes are executed in the enterprise. Many enterprises adopt without such vision and strategy, only to realize that there are problems later on. The goal should be to adopt SOA and business processes with higher maturity.
[REF-1] Wikipedia. "Business Process Improvement". Last modified on 12 February 2012. http://en.wikipedia.org/wiki/Business_process_improvement
[REF-2] HP. "Optimizing SOA". Transforming Your Enterprise Magazine. Spring 2008. http://h71028.www7.hp.com/enterprise/cache/599209-0-0-225-121.html
[REF-3] Caganoff, Saul. "SOA Metrics for Continuous Improvement". soabloke. http://www.soabloke.com/2008/01/07/soa-metrics-for-continuous-improvement/
[REF-4] Wikipedia. "Business Activity Monitoring". Last modified on 14 December 2011. http://en.wikipedia.org/wiki/Business_activity_monitoring
[REF-5] Wikipedia. "Performance Indicator". Last modified on 7 February 2012. http://en.wikipedia.org/wiki/Performance_indicator