Longji Tang is a Senior Technical Advisor in FedEx IT and Professor of the School of Information Science and Engineering in Hunan University. His research focuses on software architecture and design, service-oriented architecture, service computing, cloud computing, mobile computing, big data computing, and system modeling as well as formalism. He began graduate studies at Penn State University in 1992 and graduated in 1995 with a Master of Engineering degree in Computer Science & Engineering and a Master of Art degree in Applied Mathematics. Longji started his part-time PhD studies in 2005 and obtained his PhD degree in Software Engineering in 2011. He published more than 35 research papers from data science, numeric analysis, and inverse problems to SOA, cloud, and mobile computing. He is one of members of Program Committee in 2013/2014/2015 IEEE Mobile Cloud
Dr. Mark Little is VP Engineering at Red Hat where he leads JBoss technical direction, research, and development. Prior to this he was the SOA Technical Development Manager and the Director of Standards. He was also the Chief Architect and Co-Founder at Arjuna Technologies, as well as a Distinguished Engineer at Hewlett Packard. He has worked in the area of reliable distributed systems since the mid-eighties. His Ph.D.f was on fault-tolerant distributed systems, replication, and transactions. He is currently also a professor at Newcastle University.
API Governance and Management
Published: October 8, 2014 • Service Technology Magazine Issue LXXXVI
Abstract: We live in an era of service computing with cloud computing platforms, social computing, and mobile computing. One of the most significant characteristics of the era is that any device connects to any service and any service connects to any data with a cost-effective way. The connection between device and service as well as between service and data is built by modern Web APIs. The shift is not only for using software in particular business, but also for engaging other business and people - internal developers, partners, customers, and the world at large, through exposing software interfaces by APIs. The trend is creating a new business reality - API Economy. It is leading an evolution of the traditional SOA paradigm to cloud-enabled, social-enabled, and mobile-enabled modern lightweight SOA. There is increasing automation of processes, transactions, and distribution across many industry sectors and organizations. This paper describes the API Economy and the emergence of API management, its building blocks, its role in service infrastructure. Moreover, API-central architecture patterns, its reference architecture, and its deployment topologies can be found in a newly coming book Service Infrastructure.
Emergence of API Management
The Application Programming Interface (API) is an old technology, which has been around for decades, the rise of Web APIs, which includes new majority REST APIs, traditional SOAP-based APIs, and other, lead APIs technology for building mash-up applications, getting data and services to mobile applications, and connecting enterprises to their partners and cloud services. APIs have started their new life in modern elastic, social, mobile world. With the modern Web APIs dramatically growing, and high availability through the internet, increasingly business values, and becoming more and more important as the application landscape of enterprises, APIs quality (security, performance, availability, …) and risk from exposing data and services by using open APIs become main concerns to enterprises. Thus, API management is becoming a very important core component in modern service infrastructure. In this section, the rise, development, and importance of API management are described and discussed. Although API management is a newly defined term, we will see API management is just an extension of SOA Management and provides new technologies and architectural principles, such as developer portal, Key Management, and metering as well as billing facilities that SOA management does not cover. API management is shaping the multi-channel and multi-tenant strategy cross-organizational boundaries.
APIs have been around in hardware and software computing infrastructure for several decades. It has been used as an important component in software systems for specifying how software components or systems should interact with each other, such as, Microsoft Windows API or the Java Enterprise Edition API. However, modern Web APIs are creating business miracle and changing IT landscape. Figure 1 shows you a history of various popular APIs. The modern Web API is not generated from standards, like SOAP APIs, but innovated by modern technology – cloud, mobile and social computing innovators and by the HTTP standard. Modern APIs started around 2000 when saleforce.com officially launched its web-based, enterprise-class, and API-enabled automation called SaaS today, rising dramatically from 2008, and continuing to grow.
Figure 1 – Modern API Milestone
The API is continuing to grow with industry broadly adopting REST APIs. The API Economy has been formed in terms of both API technology advantages and business innovation opportunities. The API technology advantages include:
- REST API simplicity for building ecosystems.
- Easy integration for integrating apps, specifically, mobile apps with services – cloud services and enterprise business services.
- Wider reach allowing anyone to create a new app, such as a website or a widget which can distribute services and information to new audiences and in specific contexts that can be customized to provide tailored user experiences through APIs.
- Exposing information and services for leveraging your investment in SOA assets.
- Providing API access allows content to be created once and automatically published or made available through many channels. Your agency's content is ready for easy sharing and redistribution to deliver your mission directly to more citizens.
We see a lot of successful stories in cloud computing (such as Saleforce-SaaS, Google-PaaS, and Amazon-IaaS), Social Computing (such as Facebook and Twitter), Mobile computing (such as Amazon, Foursquare), and traditional eCommerce. Expedia generates more than $4 billion of revenue a year through its API-powered affiliate network. PayPal processed over $14 billion in payment transactions in 2012 and reached $27 billion in 2013 via its API-enabled business network. Figure 2 depicts both API growth and API Economy booming scene. PragrammableWeb listed 8826 public APIs on March 24, 2013 (see Figure 2), the number of public APIs is projected to reach 30,000 by 2016 by a report .
These numbers not only indicate APIs are growing quickly and the API Economy is booming, but also reflect the important of APIs and their management. In fact, the API is becoming the heart of your mobile app strategy: exposing APIs has gained traction as organization realize that leveraging their data and services across boundaries creates more innovation that drives value to all stakeholders, API Gateway is becoming a core component in mobile computing architecture, API management is becoming a new front tier for enterprise SOA.
Figure 2 – API Growth and API Economy Booming
Definition of API Economy: The API Economy is the economy where companies expose their (internal) business assets or services in the form of (Web) APIs to parties with the goal of unlocking additional business value through the creation of new asset classes. (Cutter Consortium, 2013)
The above definition is based on "economy" prospective. This paper defines the API Economy from a value-added architectural style prospective:
Definition of API Economy From technical prospective: The API Economy can be defined as a software architectural style that combines modern web API capacity with API business model. It has two main principles on information resources and services:
- Build value-add ecosystem for exposing information resources and infrastructure as well as platform resources through web-based APIs
- Create new value-add resources via hybrid style APIs combining different type APIs – public APIs (open APIs), partners’ APIs (open to partners), and private APIs (internal APIs).
The API Economy is changing not only the way companies do business, but also the way they build their service infrastructure and connect their services to customers. The API Economy is emerging in both the IT world and business world. The traditional way to expose companies' information resources or services (1993 – 2000) mainly by web applications is moving to new API-enabled ways through multiple channels which include web, mobile devices, internet TV, connected applications as well as services, connected machines (such as cars), and partners' applications as well as services.
Compared with traditional enterprises, API-enabled enterprises are agile and open and have the following characteristics:
- Adopting flexible as well as simple APIs as major channels in their business
- Enabling business transactions to be driven anywhere and anytime through API layer in service infrastructure
- Providing web, mobile, and other client interfaces as a layer on top of APIs
- Allowing customers to integrate with core service infrastructure directly through well-defined APIs, such as Amazon Elastic Compute Cloud and HP/IBM OpenStack APIs.
In the next section, we will show how API Economy impacts companies' service infrastructure and becomes the driver of API management.
Driving Forces of API Management
In the last section, we described the API Economy, its history, concept, and the characteristics of API-enabled enterprises. The driving forces behind the API Economy include:
- Business Consumers – they expect to access data and content anywhere and anytime across multiple devices and channels.
- Business Companies – they are service providers which want to re-invent interactions with customers, supplies, and partners in cost-effective or ecosystem ways. They expect to speed business and IT innovation and increase scale cross organization boundaries.
- Service Computing – it is based on SOA principles. All APIs are services, which connect to resources of information, infrastructure, and platform, and existing services built on SOA architectural style.
- Cloud Computing – which allows enterprises share their resources and services cross their boundaries through public clouds or cross organizations inside enterprise through private cloud. APIs are the simply and flexible way to allow enterprise to share their resources and services internally and externally.
- Mobile computing – mobile devices are overtaking PCs as the most broadly used devices to access information resources. Moreover, mobile computing wants a lightweight approach for connecting to enterprises' data and services due to mobile devices limited resources. Therefore, mobile computing becomes one of the major driving forces for adopting and developing APIs.
- Social computing – which is open to everyone and every device. Facebook and Twitter are using simple RESTful APIs to connect their social network and social services and allow developers and enterprises to integrate and access their core social platform for their business.
- Big Data and Analytics – Big Data refers to relatively large amounts of structured and unstructured data that require machine-based systems and technologies in order to be fully analyzed. Cloud-based APIs can help companies at both analyzing and distributing big digital data cheaply. The Apache open Hadoop API plus NoSQL database technology, such as MongoDB can make Big Data Analytics cost-effective, scalable, and fault-tolerant.
- Internet of Things (IoT) and Machine to Machine (M2M) – IoT and M2M is a future technology and business, which is one of the new driving forces for the API Economy. API Economy players, such as Layer7 and Apigee predicted how M2M and IoT impacting API Economy future . The APIs will be broadly applied to IoT and M2M as smart devices' Web interfaces connecting to IoT services. The API gateway will be one of the core components in IoT and M2M architectures.
Exposing resources and services to people and allowing developers and partners to access and integrate with companies' core business through APIs increase opportunities and innovation. However, it also increases risks and challenges that include:
- APIs are developer-defined interfaces to services. They are used to encapsulate complexity in application services and selectively expose functionality. Developers can build new solutions based on APIs. However, not all APIs are well defined and perform well. Using a bad API or misusing a good API will cause software system failure or performance issue. A Bad API may put your system at risk. The following two REST APIs represent a security risk. The first one puts the API key in its URL, you may get charge from the service provider if your API key is stolen by other people. The second one's risk is more serious, since its transaction is not protected by both SSL and API key.
- API quality assurance such as availability, scalability, reliability, security, is a main concern for enterprises using open APIs. In today's global economy and complicated IT environment, to make a business transaction, you may need to use internal APIs to connect to core business services in your own data center, use partners' APIs to do a B2B transaction, and you may need to use an open APIs to get additional information. Any API failure in the transaction will cause some failure of the transaction and impact your customer experience. To guarantee API infrastructure quality is a big challenge. The challenge include:
- To guarantee API software quality, must have good API design time governance.
- To guarantee API runtime quality, must have good API runtime governance. Modern composite applications are aggregating and consuming multiple APIs – private, partner, and public APIs at a staggering pace in order to achieve business goals. To ensure API integrity is a big challenge.
- The API governance as extension of exiting SOA governance is new to enterprises. For instance, API testing is a must-have process in enterprise software development lifecycle, to ensure APIs are delivering the necessary level of security, reliability, and performance.
- API service level agreements are concerns for both API providers and API consumers. To reach the agreements and delivery that the API consumers' want is also a challenge. From a report from Parasoft, 90% of respondents report that APIs failed to meet their expectations, in which 68% encountered reliability/functionality issues; 42% met security issues; and 74% encountered performance issues.
- API security is one of the biggest concerns for enterprises. It includes service and infrastructure access security, data security, and trust. API security compliance and protection of services as well as data are challenges.
- API consumers have risks for moving to the new API business model, since they depend on T&C of API providers.
- API Governance is a big challenge, since APIs include internal, external, and open APIs which support different protocols, SOAP, REST, JMS, ... They are developed by different vendors, software startups, and individuals. The API governance challenges include:
- Design Time Governance, such as API versioning, design standards, specifically new REST-style API development standards.
- Run time governance, such as API monitoring, API deployment, and dynamic provisioning.
Facing the above risks and challenges of API Economy, API management is working to reduce the risks, providing solutions to the challenges and protecting API businesses. API management is defined in the next subsection and the relationship between it and SOA governance is discussed.
Definition of API Management
We have seen that the API Economy requires a new service infrastructure – API management that provides API governance and powers the API Economy. This section first defines API management, and then discusses the relationship of SOA and Cloud governance (Chapter 18) and API management.
Definition of API Management: The API management is a set of processes and technologies for governing APIs in a secure and scalable service infrastructure. It includes a minimum set of required functionalities:
- API Developer Portal for managing API development and providing API lifecycle management, and the process and interface for publishing, discovering, maintaining, and overseeing APIs.
- Automate and control connections between an API and the API consuming applications.
- Monitor API traffic and other quality metrics, such as performance as well as reliability (for instance error rate), from applications which use it.
- Provide proper API versioning technology to ensure consistency between multiple API implementations and versions.
- Ensure API scale and improve application performance by dynamic provisioning technology and caching mechanisms.
- Protect API from misuse and any other vulnerability in API access point or endpoint by providing API security solutions which include basic security, such as SSL as well as TLS, and advanced API security, such as API access authentication as well as authorization, key management, and perimeter defense for enterprise-class APIs.
- Provide capability for metering and billing API utilization of commercial APIs.
From the definition of API management we can see that some functionalities, such as monitoring, security are the same as basic SOA governance and management. However, a lot of new functionalities provided by API management, such as API developer portal, key management, and metering as well as billing capacities, are never provided by SOA management. Therefore, API management extends SOA governance and management for new API economy and improving enterprise architecture agility. By Gartner's research, the hybrid approaches with both existing SOA governance and API management can be defined as the Application Services Governance that provides solutions and technologies for guaranteeing success of existing SOA approaches and new API economy.
Role of API Management in Service Infrastructure
API Tier in App Services Infrastructure
The API has become a tier in modern application services compute infrastructure and the API tier is playing a more and more important role. Figure 3 describes the typical API tiers in Application Services Infrastructure. There are two different API tiers:
- API Tier between applications and middleware and/or ESB, which is in the scope of API governance and managed by API management technology, such as the API gateway. The tier is for applications consuming resources and services from backend systems. The majority of the API tier is REST-style API or Web API, and JSON is used as the data exchange format. Another popular API is SOAP-based API which is often used for consuming SOAP web services. Strictly speaking, a traditional (or classical) API is defined as an access method to a service (or a service interface, according to SOA terminology). The SOAP-based API is a kind of traditional API that can be viewed as an in process service. The Web API is a new kind of API that is a remote API service based on HTTP. We mainly discuss the API governance and management for the API tier in this paper.
API tier between middleware or ESB and application services that include existing SOAP web services, Java Enterprise Edition services, .NET MCF services, messaging services, data storage, and other services that are governed and managed by enterprise SOA governance.
Figure 3 – API Tiers in App Services Compute Infrastructure
API Gateway and its Role in App Service Infrastructure
The API economy introduced a new API tier in modern application service compute infrastructure as shown in Figure 3. The API tier is becoming a critical bridge from customers to enterprise services, from enterprise to cloud services as well as your partners' services, and from one cloud to another cloud. Further, the APIs include internal, external, and public APIs. Therefore API security, performance, routing, and multi-tenancy become very challenge for the new API-centric architecture. API management is emerging for governing and managing APIs. In general, API management consists of the following main components:
- API Portal – which is a design-time API governance tool for managing API registry (or publishing), API profile (or documentation), API control, and API development lifecycle.
- API Gateway – which is the core API runtime governance component for managing API runtime behaviors, such as routing, multi-tenancy, security (identity, authentication as well as authorization).
- API Service Manager – which is a component for managing API lifecycle, such as migration, dynamic versioning, deployment, configuration, API changes (such as policy change, configuration change)
- API Monitor – which is part of API runtime governance components for metering the API runtime behaviors, such as performance, usage.
- API Billing or Chargeback – Billing is for utility-oriented public API, such as Amazon EC2 API, and Chargeback in case of on-premise or private cloud. Both are based on metered usage.
In this section, the API gateway and its role in service infrastructure are described and discussed. API gateway consists of the following main common components:
- API routing manager
- API security manager (such as API key management, OAuth and OpenID)
- API mediation
For example, Layer7 has a family of API gateways that are shown in the following Table 1:
||provide the core functionalities needed for enterprise-scale API security and management
||Provide connectivity for accessing SaaS application and other cloud services securely and seamlessly
||Provide centralized governance services integrated across the extended enterprise
|Mobile Access Gateway
||Provide capacity to connect mobile devices and apps to open enterprise information assets and services securely and efficiently
Table 1 – Layer7 API Gateways
The API Gateway – lightweight service mediator simplifying application delivery stack, which acts as a control point between enterprise service infrastructure and the outside world accessed through APIs, which can provide the following main features to modern service compute infrastructure:
- Integration – API gateways can integrate with existing Identity Management (IM) infrastructure, such as CA SiteMinder, to perform both authentication and authorization of API message traffic. API gateway can integrate with existing dynamic service provisioning and offer a highly flexible and scalable solution architecture.
- Anypoint Connectivity – API gateways allow applications to invoke services that run anywhere as well as anytime (such as cloud services, mobile services), and allow apps to seamlessly move any services around at will without affecting existing service infrastructure.
- Mediation – API messaging routing is one of the API gateway's main features. It extends SOA mediation and deliver API message between service consumers and service providers. API gateway routes data, message based on user's identity, content types, therefore it enables data and messages to be sent to appropriate applications securely. Governance – API gateways provide centralized management for API changes, API traffic, API deployment, policy enforcement, and API issue reporting.
- Security – API gateways enable enterprises to secure their Web APIs against hackers' attacks and API abuse. It can be a central security checkpoint through its support to broad security standards, such as SSO, OAuth 2.0, SAML, OpenID. For instance, an API gateway can authenticate internal clients by userid and password, and then it can issue SAML tokens that used to for identity propagation to application servers.
- Transaction – enterprise-class API gateways also supports business transaction through meeting audit requirement as well as PCI compliance and securing sensitive data.
- Performance – some API gateways also provide caching technology for increasing performance, such as Apigee API gateway. Some API gateway integrates XML Accelerate Engine (VXA) to make XML processing faster, such as the Oracle API gateway.
We have introduced API Governance and Management in this paper. The key takeaways are
- Cloud computing, mobile computing and social computing drive the API Economy. It is a new IT development trend that leads IT innovation and IT alignment with its business.
- APIs become a primary customer interface for technology-driven products and services and a key channel for driving revenue and brand engagement.
- APIs increase exposure of enterprise services and data; therefore, increase value of in services and data.
- API management is the key for API Economy success. It is an extension of SOA governance and management and one of core components in modern service infrastructure. It is playing a central point for API-Centric service system integration.
- API-Centric architecture is another enterprise architecture shift. Adopting API-Centric enterprise architecture can improve security, agility, scalability, and cost-effectiveness of the IT service infrastructure.