Philip Wik is a Senior Data Architect for American Express. Philip has worked for JP Morgan/Chase, Wells Fargo, Honeywell, Boeing, Intel, and other companies in a variety of applications development, integration, and architectural roles. He has published three books through Prentice-Hall:
How to Do Business With the People's Republic of China, How to Buy and Manage Income Property, and (along with other co-authors) Next Generation SOA.
The Annotated Next Generation SOA Business Case – Part II Published: September 20, 2015 • Service Technology Magazine Issue XCII PDF
This article is the conclusion of an annotation to the case study from Chapter 7 in Next Generation SOA: A Concise Introduction to Service Technology & Service-Orientation [REF-1]. To read the first part of this article series, click here.
Actions and Analysis
May: Strategic Planning Commences
Page 95: "In the last week of May, Robert holds a two-day strategic planning workshop with key thought leaders from the company's IT and business operating units, including executives, managers, architects, and business analysts."
Page 102: "As RYLC begins its transformation journey by automating business processes and introducing SOA, its architects establish reference architecture."
When we have firings, we must recalibrate business and architectural policy. While layoffs will free up cash and can provide a bounce to the stock, the underlying problems will still fester. Clients, employees, and shareholders will view firing people just to cut costs as desperate. Therefore, layoffs must have solutions to fix the firm's problems. We must look at our business model for boosting productivity and enhancing revenue which includes leveraging the company's key stake holders, thought leaders, and managers.
We should have undistracted discussions away from the office, perhaps at a local hotel. We should do this over several days to let those great incubators of inventions – bus, bed, bath, etc - and ideas work their magic. This workshop should have high bandwidth presentations of people in front of people who are in front of a whiteboard instead of webinars or teleconferences. It's a greenhouse for change, a forum for honest debate, and a place for slaying sacred cows and finding solutions. It's a time to be tough-minded and make difficult calls that effect large numbers of people. And it's a time to imagine. To close our eyes to see. From this sea of creative and divergent thinking will come a swarm of ideas which will merge into the next big idea which will in turn build the epics and themes that will grease the rails of the enterprise.
It's also a time to meerkat threats. The meerkat, a beast from the mongoose family, stands sentry, constantly scanning the African sky and ground. It will chirp alarms that vary depending on the urgency of the threat which could range from eagles to jackals to snakes. Vulnerability threat assessment should be part of the workshop as a way to avoid the warping effects of our wishes.
So let's ask: what's the chance that our enterprise will collapse? We can find the answer by brainstorming threats such as the loss of a key contract, executive, or lawsuit. Secondly, we develop a consensus confidence level that a company will realize these negative events, from ten to one hundred percent. Thirdly, we assign a load coefficient (M) between 0.1 and 1.0 to show the impact of failure for each event. For example, the loss of the CEO might be 0.2 while the loss of a contract might be 0.6. Finally, we multiply across, then down, and divide by the count of events (N) to get the probability that our company will fail.
1. M * % = P(1)
2. M * % = P(2)
3. M * % = P(3)
N. M * % = P (N)
(P(1) * P(2) * P(3) * P(N))/N = % chance that the company fails
Is this product a useful metric? No. The ingredients of this stew, spiced by hope, fear, and groupthink, are guesses. Also a firm could recover from the brink of collapse after its assumed tipping point. But its value lies in the process. We think the unthinkable as a way to foretell problems and find solutions.
This workshop will introduce a swirl of factoids such as the number of widgets a division sold last year. Architecture is epistemology. Our analysis isn't a search for facts but a search for truth. One has little to do with the other, but the best spin is always factual. Evidence has no meaning apart from its rational coherence, litigated in dialectic of debate and viewed through a prism of doubt. What we fix in our mind as true depends on where we place the center of gravity in our realms of information and misinformation. Tradition, fear, impetuosity, authority, and bluster can crush that tender plant of truth. It takes rigorous integrity to resist baking precooked conclusions with cherry picked facts and to grope towards a clearer reality in what the poet T.S. Eliot calls this "wilderness of mirrors."
We must destroy before we can build. The great prizes come from iconoclasts with an enterprise vision. That vision might mean taking the business down to its studs. Leave roles and reputations at the door as we seek creative ideas and ask foundational questions. What is the market telling us? What do we do well? Where do we fail? How can we make more money? Don't settle for "it's too hard" or "that's not the way we do things" or "we don't have the funding." Those half-truths mask the higher truth of our inner world where the battle for business is either lost or won. For the conditions to exist in which everything remains static is axiomatic [REF-2]. We must change things as they are to things as they should be, and then we must change.
SOA preordains nothing. We preordain everything. War, peace, boom, and bust make most of our choices. But still we can choose. Responsible for ourselves and others and acting in the solitude of hard choices, we must recreate ourselves and our corporations. Business success comes to those who can make the choices to effectively mobilize SOA. "Make big plans" Daniel Burnham, the 19th century American architect and city planner says. "Aim high in hope and work, remembering that a noble, logical diagram once recorded will not die, but long after we are gone be a living thing, asserting itself with ever growing insistence." In the soil of SOA, our plans will take on a life of their own. SOA in the context of a revitalized culture and community is our agent for business success. It's that bridge between a profitless, contracting, turning-the-lights-off firm and a profitable, growing, next-generation enterprise.
Direction matters. Despite state-of-the-art technology, world-class leadership and staff, an ample budget, and little competition, an enterprise failed because of its direction. That enterprise was the RMS Titanic whose sinking in 1912 resulted in the drowning of more than 1,500 passengers. Cast a compelling vision, aim high, and think big. But above all, get the direction right.
From RYLC's discussions came an enterprise of reference architecture: a charter of principles and the direction for the next five years. It envisioned building out with clouds, business intelligence (BI), complex event processing (CEP), Big Data, and mobile communication. RYLC used different cloud models to obtain and store data. It centralized common IT services into a community cloud and used SaaS for web conferencing, social media, office applications, and customer relationship management. RYLC used BI to get real time analytics on sales, purchasing, and customer behavior and CEP to identify trends and aggregate in-house transactional data with external market events. It deployed a cloud-based Hadoop tool to provide real time analysis of Big Data logging to identify problems and alert relevant support teams. RYLC also rolled out mobility to let customers lease a car with Web interfaces or on their smart phones.
August: RYLC Buys an Enterprise Service Bus
Page 101: "In August, Raj focuses on completing the vendor assessments before purchasing a vendor-based ESB SOA suite and an open-source tool set for testing, data access, change management, security management, and performance optimization."
Page 102-103: "As part of the lead service architect's solution, a new cloud-based Customer Service Profile service is developed to query, enter, and maintain customer profiles in a cloud environment. A new composite Customer Profile Migration service is created to assist in the gradual migration of customer information to a newly created cloud-based profile store, which is invoked whenever a customer profile returns zero hits for a customer name."
It can be better to use open source middleware rather than vendor products. There are exceptions and bakeoffs will show those exceptions. Select software that supports multiple geographies, time zones, languages, and tenancies. We should view these technologies as design patterns, and as such seek reusability, service autonomy, and agnostic capabilities. Systems development should have more to do with configuration than coding. Avoid custom definitions of orchestrations or compositions. Instead use the features provided by a vendor-based Business Process Management (BPM) or an Enterprise Service Bus. For some service technologies, we may require programming to add business logic to the generated code. In other cases, a model can generate the source code for the service. A BPM tool, for example, can generate a Business Process Execution Language, an XML orchestration specification of abstract business processes.
If SaaS solutions aren't available, consider Commercial Off The Shelf packages. If neither of the previous approaches are viable, build an in-house solution using a standard set of tools and technologies. We should consider all three models with an understanding of their costs, benefits, and limitations. Consider the total cost of ownership, including long term support costs. Don't default to either the cheapest solution (open source) or the most familiar solution (brand names). Expose functionality as reusable composite services, present user interfaces through web browsers and mobile devices, employ Single Sign On, and deploy applications into a virtualized environment. Use proven technologies from established vendors. Standardize on one or two key vendors for core infrastructure. Define one integrated way of protecting data access, one reporting solution, and one master data management. For vendor solutions, consider risks relating to licensing costs in relationship to projected growth and contractual locking.
RYLC's new cloud-based customer and vehicle profile services became operational after four months. RYLC surveys showed a steady increase in customer satisfaction and returning customers. The solution used was the multi-channel endpoint design pattern that coordinates multiple physical sources to provide a virtual service provider and the legacy wrapper pattern that hides legacy systems behind a structured service façade.
Figure 1 – A flowchart outlining the steps for the customer profile business process, as carried out by the Customer Profile and Customer Profile Migration services
October: RYLC Organizes for Success
Page 101: "Meanwhile, Raj hires an external consultant to lead a six-month project dedicated to transforming RYLC from a waterfall business model to an Agile business model, as well providing the company's developers and architects with on-site training in Web services, XML schema, and canonical data modeling."
Page 107: "In October, RYLC establishes a central SOA Governance Program Office (SGPO) to resolve enterprise-wide conflicts and disputes, which have been especially pronounced between the IT and business development divisions."
We mean Agile to include methodologies such as Extreme Programming, Scrum, Kanbam ("billboard" in Japanese), Lean Six Sigma, Dynamic Systems Development, Feature-Driven Development, and the Scaled Agile Framework (SAFe). As is sometimes true among branches of co-religionists, Scrum purists may not be on speaking terms with SAFe heretics, preferring to accentuate differences rather than commonalities. But Agile emerged in response to market-driven changes: "Agile processes harness change for the customer's competitive advantage." If responding to change is the essence of Agile, then it must follow that those same market forces will change Agile. The dogmatism of the new is worse than the dogmatism of the old.
"None of us is as smart as all of us" Kenneth H. Blanchard notes, and we must embrace not just the form of Agile but its spirit. If the blueprints of a new direction continue to come just from the firm's oligarchy, we will make no progress in moving beyond obsolete assumptions and dated knowledge. Agile lets the collective intelligence of the corporation to not only deliver the best solutions but also to provide ownership of those solutions to all team members.
However, as Agile's bottom-up tendencies contradict SOA federation, we should establish a way to manage services that Agile teams build. Bottom-up development is no substitute for top-down strategy and governance [REF-3]. Jane from IT and Nish from the Business Development division squabbled as to whom should own the SGPO. As identifying services is a business rather than an IT function, Raj correctly placed SGPO within the business division but also mandated close collaboration with IT. Systems and services should be loosely coupled while roles and departments should be tightly coupled to weaken silos and strengthen seams. To manage SOA/BPM solutions, RYLC also established an Enterprise Architecture Board that connects to a steering committee
Good governance empowers for good reasons. Bad governance constrains without good reasons. A good reason answers the question "does this rule help or hurt the delivery of services to the enterprise?" On the contrary, a corollary test could ask "does this rule make us money or cost us money?"
A rule of jurisprudence is that we should favour a few rules that are strictly enforced over many rules that are weakly enforced. Rather than trying to anticipate all that could go wrong, it's better to state broad principles and develop a process that promotes interaction to resolve disputes. This aligns with Agile in valuing "individuals and interactions over processes and tools. [REF-4]"
Backlog grooming drives the ordering of services. We rank user stories from fine to coarse grained. Our scrums come from fine grained stories. We first pluck the low-hanging fruit by developing low-risk, simple services leveraged from existing applications. We then leverage off those triumphs to deliver higher risk, composite services that have no analogue to existing systems. With this process, we build a more efficient factory for building services.
From envisioning and iteration zero to warranty and closure. Agile wouldn't exist if it weren't for robust planning, and these plans include project vision, project road map, release planning, iterative planning, and daily planning. Planning scope will vary according to the size and nature of the enterprise and whether project development is time-boxed or continuous flow. Continuous planning now animates RYLC's operating units, tempered with Eisenhower's dictum that "plans are worthless but planning is everything" [REF-5]. The company's executives would never let yesterday's plans dictate tomorrow's actions but neither would they just wing it. RYLC was becoming progressive and forward-looking rather than complacent and adrift.
Conclusion: A New Enterprise
Page 110: "By the third week of October, RYLC had an operating governance organization that was transmitting service-oriented metrics throughout the enterprise. It also had a growing inventory of services and a stable head count. Fimalac SA's Fitch Ratings raised RYLC's credit rating from C to A+ citing "adequate containment on risk governance issues" without any reference to its service-orientation initiatives. The hedge fund that had representation on RYLC's board increased its holdings by five percent. Wall Street announces that RYLC stock was a buy, with strong ratios, earnings, and discounted cash flows. Technical analysts observed the bullish Golden Swan pattern with a five-year break out into virgin territory on heavy volume in the middle of November, with the company's stock valuation having increased by 113 million dollars since the middle of April".
Here is RYLC's heat map in March and November. It shows RYLC's core business priorities. The columns represent divisions within the business and the rows represent operating units or functions within that division. We color cells within the map to show investment priority. Red cells show units that have the highest priority for investment.
Figure 2 – RYLC's corporate structure in March and December
Rent Your Legacy Car launched a significant transfiguration. Service-technologies married with changes in culture sparked its rebirth from a failing, regional firm to a nimble, multinational powerhouse. A feeling of hope and fun started to energize RYLC with the ferment of ideas flowing from interlocking centers of excellence. The years of inertia had come to an end. SOA had broken the spell. It was a new day for a new enterprise.
We Did It Ourselves
While this story is fiction, the journey that RYLC took is no fairy tale, and it has been done countless times by other companies. RYLC got its success from SOA, a paradigm that prizes business value over technical strategy, strategic goals over project-specific benefits, intrinsic interoperability over custom integration, shared services over specific-purpose implementations, flexibility over optimization, and evolutionary refinement over the pursuit of initial perfection. SOA struck the match that started the conflagration of creative destruction and reconstruction.
Although SOA was the catalyst, the most critical elements in its success had nothing to do with SOA. Other lifelines included championship from the board, hiring new executives, embracing Agile and Big Data, leaning the organization, building structures that fostered accountability, and leading teams that had a passion for hard work, communication, and collaboration. It wasn't just the leaders but many people in many roles that made service-orientation happen. Something changed within them. They became more courageous, and curious, more tolerant of risk and ambiguity, better listeners and speakers, and better followers and leaders. They became better people. And that made all the difference at RYLC. Ancient Chinese poet Lao Tzu put it best: "the best of leaders, when the job is done, when the task is accomplished, the people will say we have done it ourselves."
[REF-1] Thomas Erl, Clive Gee, Jürgen Kress, Robert Laird, Berthold Maier, Vijay Narayanan, Hajo Normann, Pethuru Raj, Leo Shuster, Bernd Trops, Clemens Utschig-Utschig, Philip Wik, and Torsten Winterberg, Next Generation SOA: A Concise Introduction to Service Technology & Service-Orientation, Prentice-Hall, 2014, pp 89 – 113. www.servicetechbooks.com/nextgen.
[REF-2] Email to the author from Doug Blumhardt to whom I reported when he led First Infinity in the late 1990s. As a Senior Vice President at the First National Bank of Chicago, a predecessor company of JP Morgan/Chase, Doug successfully managed this $100 million project, one of the largest projects at the time. "The exact quote is, 'all of the conditions are present for everything to be exactly as it is right now.' I heard the quote in a speech at a conference. The point the speaker was making is that things are the way they are because of the conditions exist for them to be the way they are. If we want to change the circumstance, performance, and future outcomes, we most often have to change the conditions. True leaders focus on conditions-- how things get done, how teams are motivated, how people are rewarded, and the goodwill and cooperation by other organizations and agencies. In other words, they focus on the conditions in which their organizations can succeed, and they hire good people to achieve the actual results. Great leaders such as Jack Welch are so absorbed in creating the right conditions that they don't have a great deal of time to meddle in the vigorous work of the great teams they have created. The job of an executive isn't to get things done. If he's done his job, other around him get good at getting things done. Reach ahead in your mind to some foreseeable point in the future (let's say six months) and imagine how things should be running at that point. Then write down all the conditions that would have to be in place to achieve that performance. Now set about the business of creating those conditions. Any intangible that creates the atmosphere for your effective results is a 'condition'. Thanks for remembering my quote. It was the centerpiece of what I was trying to do at First Infinity."
[REF-3] This e-mail to the author by Marc-Thomas Schmidt, IBM Distinguished Engineer & Chief Architect Watson Platform Next, speaks to the problem that Big SOA tries to solve: "I don't see SOA as inherently top-down. I do think there are other ways to approach SOA such as bottom up ones that take stock of the applications I have and then recycle them into services, where the service design is driven by thoughts about potential re-use, not necessarily by business process use cases. The spectrum between the two extremes leads to federated approaches where there is an attempt to weave together relatively independent service domains without necessarily doing away with the domain specifics. If one wants to make SOA scale, and I think that's what the main problem is that we're facing today when it comes to making SOA work as top-down SOA doesn't scale, we need ways to take into consideration different enterprise topologies, governance domains, and the many other aspects nicely described and triangulated in your paper." See "Effective Top-Down SOA Management In An Efficient Bottom-Up Agile World", SOA Magazine, April 8, 2010 www.servicetechmag.com/I38/0410-1.
[REF-4] Dwight D. Eisenhower: "Remarks at the National Defense Executive Reserve Conference," November 14, 1957. Online by Gerhard Peters and John T. Woolley, The American Presidency Project. www.presidency.ucsb.edu/ws/?pid=10951.
[REF-5] "Manifesto for Agile Software Development", www.agilemanifesto.org