Development

The OpenAPI Specification Explained The OpenAPI Specification is the ultimate source of knowledge regarding this API description format. However, its length is daunting to newcomers and makes it hard for experienced users to find specific bits of information. This chapter provides a soft landing for readers not yet familiar with OpenAPI and is organized by topic, simplifying browsing. The following pages introduce the syntax and structure of an OpenAPI Description (OAD), its main building blocks and a minimal API description. Afterwards, the different blocks are detailed, starting from the most common and progressing towards advanced ones. Structure of an OpenAPI Description: JSON, YAML, openapi and info API Endpoints: paths and responses. Content of Message Bodies: content and schema. Parameters and Payload of an Operation: parameters and requestBody. Reusing Descriptions: components and $ref. Providing Documentation and Examples: description and example/examples. API Servers: servers.

Introduction to OpenAPI Overlay Specification The Overlay Specification defines a document format for information that transforms an existing OpenAPI description yet remains separate from the OpenAPI description’s source document(s). The Overlay Specification defines a mechanism for providing consistent, deterministic updates to a given OpenAPI description, as an aid to automation throughout the API lifecycle. An Overlay can be applied to an OpenAPI description, resulting in an updated OpenAPI description. OpenAPI + Overlays = (better) OpenAPI One Overlay might be specific to one OpenAPI description, or general enough to be used with multiple OpenAPI descriptions. Equally, one OpenAPI description pipeline might apply different Overlays during the workflow.

Use cases for Overlays Overlays support a range of scenarios, including: Translating documentation into another language Providing configuration information for different deployment environments Allowing separation of concerns for metadata such as gateway configuration or SLA information Supporting a traits-like capability for applying a set of configuration data, such as multiple parameters or multiple headers, for targeted objects Providing default responses or parameters where they were not explicitly provided Applying configuration data globally or based on filter conditions

Resources for working with Overlays The GitHub repository for Overlays is the main hub of activity on the Overlays project. Check the issues and pull requests for what is currently in progress, and the discussions for details of future ideas and our regular meetings. The project maintains a list of tools for working with Overlays.

Example: Add multiple parameters to selected operations One of the most requested features for OpenAPI is the ability to group parameters and easily apply all of them together, to either some or all operations in an OpenAPI description. Especially for common parameters that always come as a set (pagination or filter parameters are a great example), it can be more maintainable to use them as a “trait” and apply the set as part of the API lifecycle rather than trying to maintain a source of truth with a lot of repetition. This approach leads to good API governance, since if the collection of fields changes then the update is consistently applied through automation. In the following example, any operations with the extension x-supports-filters set to true will have two inline parameters added to their parameter collection, and an x-filters-added tag for decoration/debugging.

Example: add a license Every API needs a license so people know they can use it, but what if your OpenAPI descriptions don’t have a license? This example shows an Overlay that adds a license to an OpenAPI description. Here’s the Overlay file, with just one action to add or change the info.license fields: overlay: 1.0.0 info: title: Add MIT license version: 1.0.0 actions: - target: '$.info' update: license: name: MIT url: https://opensource.org/licenses/MIT You can use this Overlay with different OpenAPI files to make the same change to a batch of files.

Example: tag DELETE operations To add the same tag to all operations in an OpenAPI description that use DELETE methods, use an Overlay like the example below. This example adds an x-restricted tag to all delete operations: overlay: 1.0.0 info: title: Tag delete operations as restricted version: 1.0.0 actions: - target: $.paths.*.delete update: tags: - x-restricted This overlay adds x-restricted to the tags array for each delete operation. If the tags array doesn’t exist, it’ll be created; if it does, the new tag is added to the existing array. You can use an approach like this to make other changes to all matching operations.

Resolve References A reference is said to be resolved within a tool if: Its target has been identified Any modifications to the target required by the OAS (e.g. because of fields adjacent to "$ref") have been performed on a copy of the target The resulting target value has been associated with the reference source in some way that the tool can easily use when needed A reference is said to be removed if it has been replaced by its (possibly modified) target. Reference resolution usually preserves the referencing information such as the URI used to identify the target, while reference removal usually discards it. In many cases this is not significant, except that not knowing how the parsed OAD relates to the references in your JSON or YAML document may make debugging more difficult. While plain JSON documents form a tree structure, an OpenAPI Description with resolved references is not necessarily a tree, but a graph. Tools that resolve references in-memory and work with the graph structure can process all OADs. Tools that rely on reference removal, either as part of the tool or by a separate pre-processing tool, can only support OADs that form trees.

What are references? A reference is a keyword and value that identifies a reference target with a URI. In some cases, this URI can be treated as a URL and de-referenced directly. In other cases, as we will see in the (forthcoming) guide to resolving references, it is helpful to separate the target’s identity from its location. External references are how multiple documents are linked into a single OpenAPI Description (OAD). This means that referencing impacts how other linkages, such as those that use Components Object names, or values such as operationId in the Path Item Object, work. These other linkages can only work if the document (or with many tools, the specific JSON object) containing the name or other identifier has been referenced.

A taxonomy of references References exist in several variations in the OpenAPI Specification (OAS) versions 3.0 and 3.1, as shown in the following table. Note that an adjacent keyword is a keyword in the same JSON Object (whether it is written in JSON or YAML) as the reference keyword.

Using References in OpenAPI Descriptions OpenAPI Referencing is a powerful tool. It allows managing document size and complexity, and allows re-using shared components and avoiding copy-paste or change management errors. However, the history of referencing and the "$ref" keyword is complex, leading to slightly different behavior depending on the version of the OpenAPI Specification (OAS) that you are using, and on where in your OpenAPI Description (OAD) the reference occurs. There are also other "$ref"-like keywords ("operationRef", "mapping") and behaviors (referencing by component name or operation ID) that are related but somewhat different. Referencing can also be challenging to use due to incomplete and inconsistent support across different tools, some of which require references to be pre-processed before they can read the OAD. The resources in this section explain how to use referencing, and what to look for when assessing the referencing support in your OpenAPI tools.

The Future of References There are plans to reduce the complexity around references in future OpenAPI Specifications. The Moonwalk project is considering a different approach that imports complete documents, associates them with namespaces, and only supports referencing by component name (not "$ref"). A small example can be seen in the Moonwalk deployments proposal, and there are discussions around an initial draft proposal for imports and a few ideas on how to manage interactions with JSON Schema referencing. The proposed Workflows Specification is already using a "sourceDescription" field that is not unlike the Moonwalk proposal.

Keep a Single Source of Truth Regardless of your design approach (design-first or code-first) always keep a single source of truth, i.e., information should not be duplicated in different places. It is really the same concept used in programming, where repeated code should be moved to a common function.

Otherwise, eventually one of the places will be updated while the other won’t, leading to headaches… in the best of cases. For instance, it is also commonplace to use code annotations to generate an OpenAPI description and then commit the latter to source control while the former still lingers in the code. As a result, newcomers to the project will not know which one is actually in use and mistakes will be made. Alternatively, you can use a Continuous Integration test to ensure that the two sources stay consistent.

Add OpenAPI Descriptions to Source Control OpenAPI Descriptions are not just a documentation artifact: they are first-class source files which can drive a great number of automated processes, including boilerplate generation, unit testing and documentation rendering. As such, OADs should be committed to source control, and, in fact, they should be among the first files to be committed. From there, they should also participate in Continuous Integration processes.

Make the OpenAPI Descriptions Available to the Users Beautifully-rendered documentation can be very useful for the users of an API, but sometimes they might want to access the source OAD. For instance, to use tools to generate client code for them, or to build automated bindings for some language. Therefore, making the OAD available to the users is an added bonus for them. The documents that make up the OAD can even be made available through the same API to allow runtime discovery.

There is Seldom Need to Write OpenAPI Descriptions by Hand Since OADs are plain text documents, in an easy-to-read format (be it JSON or YAML), API designers are usually tempted to write them by hand. While there is nothing stopping you from doing this, and, in fact, hand-written API descriptions are usually the most terse and efficient, approaching any big project by such method is highly impractical. Instead, you should try the other existing creation methods and choose the one that better suits you and your team (No YAML or JSON knowledge needed!):

OpenAPI Editors: Be it text editors or GUI editors they usually take care of repetitive tasks, allow you to keep a library of reusable components and provide real-time preview of the generated documentation.

Domain-Specific Languages: As its name indicates, DSL’s are API description languages tailored to specific development fields. A tool is then used to produce the OpenAPI Description. A new language has to be learned, but, in return, extremely concise descriptions can be achieved.

Code Annotations: Most programming languages allow you to annotate the code, be it with specific syntax or with general code comments. These annotations, for example, can be used to extend a method signature with information regarding the API endpoint and HTTP method that lead to it. A tool can then parse the code annotations and generate OADs automatically. This method fits very nicely with the code-first approach, so keep in mind the first advice given at the top of this page when using it (Use a Design-First Approach)…

A Mix of All the Above: It’s perfectly possible to create the bulk of an OpenAPI Description using an editor or DSL and then hand-tune the resulting file. Just be aware of the second advice above (Keep a Single Source of Truth): Once you modify a file it becomes the source of truth and the previous one should be discarded (maybe keep it as backup, but out of the sight and reach of children and newcomers to the project).

Describing Large APIs

Do not repeat yourself (The DRY principle). If the same piece of YAML or JSON appears more than once in the document, it’s time to move it to the components section and reference it from other places using $ref (See Reusing Descriptions. Not only will the resulting document be smaller but it will also be much easier to maintain). Components can be referenced from other documents, so you can even reuse them across different API descriptions!

Split the description into several documents: Smaller files are easier to navigate, but too many of them are equally taxing. The key lies somewhere in the middle. A good rule of thumb is to use the natural hierarchy present in URLs to build your directory structure. For example, put all routes starting with /users (like /users and /users/{id}) in the same file (think of it as a “sub-API”). Bear in mind that some tools might have issues with large files, whereas some other tools might not handle too many files gracefully. The solution will have to take your toolkit into account.

Use tags to keep things organized: Tags have not been described in the Specification chapter, but they can help you arrange your operations and find them faster. A tag is simply a piece of metadata (a unique name and an optional description) that you can attach to operations. Tools, specially GUI editors, can then sort all your API’s operation by their tags to help you keep them organized.

Links to External Best Practices There’s quite a bit of literature about how to organize your API more efficiently. Make sure you check out how other people solved the same issues you are facing now! For example: The API Stylebook contains internal API Design Guidelines shared with the community by some well known companies and government agencies.

Best Practices This page contains general pieces of advice which do not strictly belong to the Specification Explained chapter because they are not directly tied to the OpenAPI Specification (OAS). However, they greatly simplify creating and maintaining OpenAPI Descriptions (OADs), so they are worth keeping in mind.

Use a Design-First Approach Traditionally, two main approaches exist when creating OADs: Code-first and Design-first. In the Code-first approach, the API is first implemented in code, and then its description is created from it, using code comments, code annotations or simply written from scratch. This approach does not require developers to learn another language so it is usually regarded as the easiest one. Conversely, in Design-first, the API description is written first and then the code follows. The first obvious advantages are that the code already has a skeleton upon which to build, and that some tools can provide boilerplate code automatically. There have been a number of heated debates over the relative merits of these two approaches but, in the opinion of the OpenAPI Initiative (OAI), the importance of using Design-first cannot be stressed strongly enough.

The reason is simple: The number of APIs that can be created in code is far superior to what can be described in OpenAPI. To emphasize: OpenAPI is not capable of describing every possible HTTP API, it has limitations. Therefore, unless these descriptive limitations are perfectly known and taken into account when coding the API, they will rear their ugly head later on when trying to create an OpenAPI description for it. At that point, the right fix will be to change the code so that it uses an API which can be actually described with OpenAPI (or switch to Design-first altogether). Sometimes, however, since it is late in the process, it will be preferred to twist the API description so that it matches more or less the actual API. It goes without saying that this leads to unintuitive and incomplete descriptions, that will rarely scale in the future. Finally, there exist a number of validation tools that can verify that the implemented code adheres to the OpenAPI description. Running these tools as part of a Continuous Integration process allows changing the OpenAPI Description with peace of mind, since deviations in the code behavior will be promptly detected. Bottom line: OpenAPI opens the door to a wealth of automated tools. Make sure you use them!

A declarative statement made with the intention of guiding architectural design decisions in order to achieve one or more qualities of a system.

If we take a closer look at this definition, we find several interesting parts in this definition. "[...] intention of guiding architectural design decisions [...]" As a software architect or a team of software engineers, you have to deal with and decide on many architecture issues. But how do you decide these questions? Gut feeling? :-) That's is probably not the right approach. As we learn from the Software Architecture Canvas, there are quality goals that are drivers of architecture.

What are the basic characteristics of good architecture principles?

Comprehensible & clear

Architectural principles should be like marketing slogans.

Testable The principle should be verifiable, whether work is done according to the principle and where exceptions are made.

Atomic The principle requires no further context or knowledge to be understood. In summary, architectural principles should be written to enable teams to make decisions: they're clear, provide decision support, and are atomic.

What are the pitfalls of creating architecture principles? What do you think about the following principle 👇? "All software should be written in a scalable manner."

That's why we've adopted in a product team the following architecture principle. "Use cloud services if being lock-in to a particular cloud provider is acceptable."

Whether this vendor lock-in is acceptable depends on several criteria: The effort required to replace this managed service An acceptable lead time for providing alternatives. Let's take a look at an example technological decision we had to make in the past: We needed to evaluate a centralised identity and access management solution for our SaaS products. In addition to meeting the functional requirements, we had two powerful IAM solutions on the shortlist: Keycloak (self-hosted) Auth0 (Managed, cloud service)

Following the defined principle of "Use cloud services if being lock-in to a particular cloud provider is acceptable." we have concluded that a centralised IAM system should be self-managed and not managed by a third-party provider because it's a huge effort to replace a managed IAM product and therefore there is no reasonable lead time to deploy an alternative. In summary, vendor locking wasn't acceptable to us in this case. So this principle efficiently guides us to the right decision.

Example 2: "Prefer standard data formats over third-party and custom formats"

The next principle was about the selection of protocols for service communication. "Prefer standard data formats over third-party and custom formats"

If you have multiple services that need to communicate with each other, the question of protocol and format arises. In the protocol ecosystem there is a fairly new kid on the block: gRPC gRPC (gRPC Remote Procedure Calls) is a cross-platform, open-source, high-performance protocol for remote procedure calls. gRPC was originally developed by Google to connect a large number of microservices. So in our team, the question is: RESTful HTTP vs. gRPC?

The selection of a protocol thus depends heavily on the quality and change scenarios of the services involved. But if you can meet the quality goals and underlying requirements with both options, like RESTful HTTP vs. gRPC, then consider yourself lucky to have such a principle. This principle helped us choose RESTful HTTP over gRPC because RESTful HTTP is a widely accepted standard data format, while gRPC is more of a third-party format. So here this principle speeds up our decision making, which doesn't mean that we don't rely on gRPC in certain cases.

Software architecture may be changing in the way it's practiced, but it's more important than ever.