In the dynamic landscape of modern enterprises, Application Programming Interfaces (APIs) have emerged as indispensable catalysts for innovation, connectivity, and seamless digital experiences. APIs serve as the glue that binds diverse software systems, allowing them to communicate, share data, and collaborate efficiently. This blog post delves into the pivotal role of APIs within enterprises, exploring their significance in fostering agility, driving digital transformation, and unlocking new realms of possibilities. From enhancing operational efficiency to empowering businesses with scalable solutions, the power of APIs extends far beyond mere technical interfaces, shaping the very fabric of today’s interconnected business ecosystem.
In the realm of software development, various types of APIs cater to diverse needs, and their classification is often based on their functionality and communication patterns. RESTful APIs, known for their simplicity and statelessness, are widely used. GraphQL APIs empower clients to request only the specific data they need, promoting efficiency. Amidst this diversity, the importance of standards such as OpenAPI and AsyncAPI cannot be overstated. OpenAPI provides a specification for documenting and designing RESTful APIs, fostering consistency and clarity in communication between developers and systems. AsyncAPI, on the other hand, focuses on the emerging trend of asynchronous APIs, enabling standardized documentation for event-driven architectures. These standards not only enhance collaboration and interoperability but also serve as valuable tools for ensuring that APIs are well-defined, comprehensively documented, and seamlessly integrated into the ever-evolving landscape of modern software development.
The API hierarchy, often referred to as an “API stack,” represents a structured arrangement of APIs in multiple layers within a software architecture. This hierarchical approach is designed to address various functionalities and abstraction levels, creating a cohesive framework for building complex systems. At the base level, component APIs handle core functions, providing a stable groundwork for higher layers to build upon. System APIs sit in an intermediate layer, facilitating communication between different systems – more on this. Finally, at the top layer, there are domain-driven APIs which expose functionalities tailored for use cases. The domain-driven APIs can also be categorized in two different ways based on the type of consumer – internal or external. This hierarchy not only organizes APIs based on their roles but also streamlines development, making it easier to manage, scale, and maintain intricate software systems. The API hierarchy is crucial for establishing clarity, modularity, and efficiency in the development process, allowing developers to focus on specific tasks within a well-defined structure.
When viewed through the lens of the API hierarchy, the picture begins to become clear. However, there are a ton of decision points and implementation details to bring something like this to reality in a complex organization.
Enterprise Standards
The perennial struggle between granting development teams autonomy and enforcing enterprise-wide standards embodies a delicate balancing act within the realm of software development. This is especially true as it relates to implementing APIs. On one hand, empowering development teams with autonomy fosters creativity, innovation, and a sense of ownership over their projects. It allows teams to choose technologies, methodologies, and approaches that best suit their specific needs, potentially leading to more efficient and tailored solutions. However, this autonomy also brings forth challenges, as the absence of standardized practices may lead to fragmented systems, interoperability issues, and increased complexity in maintenance. There are a wide range of technologies and standards available for implementing APIs. Even within a single API standard, such as the OpenAPI specification, the possible divergence of how different APIs are defined can be quite wide. Multiplying these points of divergence across 100’s or even 1000’s of APIs can create an unmanageable mess.
The imposition of enterprise-wide API standards aims to ensure consistency across the organization’s software landscape. Standardization promotes collaboration, ease of integration, and a unified approach to problem-solving. Striking the right balance between autonomy and standards is crucial, as it enables organizations to harness the creativity of individual teams while maintaining a cohesive and manageable software ecosystem that aligns with overarching business objectives. The enterprise not only needs to choose the appropriate standards, but they need to go the extra mile in building a set of robust tools to enforce the standards and provide feedback to API development teams. API standards enforced via manually architectural reviews will never be followed consistently due to human fallibility.
Using System APIs for Abstraction
The system level APIs are there to create internal abstractions not only for the customer, but also for other internal systems. The basic use case is abstracting complex interactions between components in a microservices architecture. If I have a commonly used unit of work that interacts with multiple components, I may want to expose that as a single, domain-driven API call with its own distinct request and response structure. These types of abstractions allow the system owner to make component or architectural changes behind the scenes without changing the API and thus limiting the impact to the consumers.
System APIs can also be created to provide an abstraction layer for third party systems with poorly defined or nonexistent interfaces. Most large companies are complex combinations of COTS systems interacting with the companies own set of custom systems, in varying degrees. These third party systems can be implemented in ways that can be very ill suited for integration by not having any direct APIs for interacting with the system’s business logic and leaving the only option for integration to the underlying database for the application. Needless to say, a third-party database schema is a very terrible choice as any integrations with the system will not only be subject to breaking changes upon system upgrade, but also because the validation logic for the interactions may be bypassed by going straight to the database. If you end up building an custom system-level API to abstract away direct database interactions, you will end up having to create and maintain some duplications of data validation and system level logic which can be very painful.
Building a System API boundary is difficult but it does have it’s benefits. One of the benefits is the ability to regression test changes to the system. These vendor systems seem to be constantly worked on, either with customer-specific enhancements or being patched. One of the main issues with direct database integrations is the testing surface it creates. Let’s say it’s a core system to the enterprise and there ends up with 100 different and distinct integrations with the system’s database If a system’s underlying database schema is updated, the only way to determine if the integrations have been broken would be to perform a full regression of all the integrations. If the regression determines it broke something important, now the project team performing the changes will have to not only wait for the downstream integration to be fixed to be compatible with the new system, the change control will also require the upgrades to the production system to be in lock step. And this is for the integrations that everyone is aware of! I’ve seen breaking changes to a vendor’s database schema have huge impacts simply because of a slew of unknown and unaccounted for integrations were implemented by separate teams across lines of business with no communication channels.
Building a System API for these black-box COTS systems allows you to identify breaking changes quickly and completely based on running the regression against the API. After a system change in a pre-production environment, the regression can identify any breaks and allow for the fixes to be isolated into a single component. This single scope of change can easily be incorporated into the system change’s scope of delivery. It also creates an inherent ownership of the API and aligns it to the team that understands the system best. It allows the practice of “use at your own risk” type of integrations where the team can wash it’s hands of the responsibility of maintenance.
What is the COTS system does come with an API? That’s fantastic. It’s going to reduce the amount of duplicated code in the form of data validation or business logic. However, the vendor APIs are still let another integration point with some form of fragility. Will the vendor follow rules around capability? Will they announce deprecations in a reasonable manner providing time for all integrating systems to make updates? Will the vendor API even follow well known specifications such as OpenAPI or AsyncAPI? Just as in the previous case of integrating to the database, when integrating with an externally managed API, there is a risk of the scope of a breaking change creating a bomb of complexity across the organization. Even if there’s an API, I would still push to include a vendor agnostic, domain-driven system API in front of the vendor’s API to not only insulate against breaking changes, but to provide that powerful regression test capability combined with a single change point.
Defining External APIs for Stability and Security
The company’s external APIs can provide fantastic value to the customer. For many complex services, having a GUI as the only point of interactivity for the customer limits their capabilities to scale. The GUI-only approach would force customers to spend their employees time performing tasks such as updating or synchronizing data between systems. By providing the API, the customer now has the ability to batch updates and generate events to manage the data shared between the company’s systems. The API also creates more stickiness due to the sunk cost of the system integration work required for the company to hook the API into it’s internal systems and workflows.
However, external APIs can be more complex to implement and manage. When building external APIs, stability becomes very important. its one thing to introduce breaking changes internally as there is at least some leverage when it comes to respecting and enforcing deprecation schedules. However with customers, introducing breaking changes can be very painful, if not impossible. Once a customer has completed an api integration, sometimes at great expense, they will not be willing to bear additional costs of updating integration code to support changing apis without a measurable increase in value. This requires getting the API definitions correct the first time and be willing to support the interfaces regardless of the changing systems which ultimately implement the exposed functionality.
When exposing an API externally, robust security measures become paramount to safeguard sensitive data and ensure the integrity of the system. Implementing secure authentication mechanisms, such as API keys, OAuth tokens, or client certificates, helps control access and verify the legitimacy of incoming requests. Encryption through HTTPS adds an extra layer of protection by securing data in transit and preventing eavesdropping. Thorough input validation and sanitization are essential to mitigate the risks of injection attacks, ensuring that malicious data cannot compromise the API. Rate limiting and throttling mechanisms help prevent abuse or overwhelming traffic, protecting against denial-of-service (DoS) attacks. Regular security audits, monitoring, and logging contribute to the ongoing assessment of potential vulnerabilities and enable swift responses to any security incidents.
Summary
APIs are hailed as catalysts for innovation, connectivity, and seamless digital experiences, serving as the connective tissue that binds diverse software systems. The discussion delves into the significance of APIs in fostering agility, driving digital transformation, and unlocking new realms of possibilities within enterprises. It further categorizes APIs into various types such as RESTful and GraphQL, emphasizing the importance of standards like OpenAPI and AsyncAPI in ensuring consistency and clarity. The concept of an API hierarchy, organized into layers, is introduced, highlighting its role in streamlining development and maintaining intricate software systems. The struggle between granting development autonomy and enforcing enterprise-wide standards is examined, underscoring the delicate balance required for effective software development. The blog also emphasizes the use of system APIs for internal abstractions, particularly in microservices architectures. Lastly, the post explores the challenges and benefits of defining external APIs, emphasizing the importance of stability and robust security measures when exposing APIs externally. Overall, it provides a nuanced perspective on the multifaceted landscape of APIs within enterprises, addressing various considerations from development practices to security protocols.