Contract Testing with OpenAPI

Contract testing has emerged as a trending topic for developers and teams working with APIs. There is already a pyramid of layers of testing, so do we need another one? What is special about it and what is the value?

But most importantly, from my OpenAPI specification can I implement contract testing?

In this article: let’s talk contract testing with OpenAPI. Is OpenAPI “ready” for contract testing? What are the options, challenges and opportunities?

The question at hand is whether it is possible to implement contract testing using an OpenAPI file. Let me answer this: the answer is no. However, if you continue reading, I will elaborate on the limitations and explore options to address them. 😎

Check out also Testcontainers with OpenAPI to see Contract Testing in action with TestContainers.

---

What is Contract Testing

There is plenty of literature on contract testing, thanks to various tools that have made a breakthrough, with Pact as the leading example. For a clear and simple “getting started” you can start from here.

A good definition of contract testing is a testing technique that makes sure the API consumer and provider meet each other’s expectations.

In (my) even simpler words:

“as I consumer I want to send a request that will receive a given response”.

Like every contract, this commitment must apply until there is a new contract in place. It must be clear to the consumer and honoured by the provider. It should be verifiable by machines (automation).

OpenAPI to Contract Testing

For developers embracing the OpenAPI standard, the obvious question is: how can I leverage my specifications (where I have invested time and effort)? At the end of the day, I have already designed my endpoints and defined parameters, requests and responses. I have also (right?) provided several examples to make the specifications clearer.

The missing link

Unfortunately, the OpenAPI standard yet does not establish a unique relation (a.k.a interaction) between a request and its response. This prevents the spec designers from defining interaction flows that are the essence of contract testing. Let’s have a look:

  /users:
    post:
      summary: Create user
      requestBody:
        content:
          application/json:
            examples:
              example-user-eur:
                $ref: '#/components/examples/create-user-eur'
              example-user-us:
                $ref: '#/components/examples/create-user-us'
      responses:
        '200':
          content:
            application/json:
              examples:
                create-user-1:
                  $ref: '#/components/examples/response-ex'
        '422':
          content:
            application/json:
              examples:
                validation-error:
                  $ref: '#/components/examples/user-error-422'

The YAML extract above is nice and tidy, but it doesn't indicate which request example belongs to which response. OpenAPI 4.0 (still in the brainstorming phase) has initiated a discussion about it, so that’s promising but looking a long way off.

Naming convention approach

One possible approach to this problem is to establish your conventions. In this case, you would name your examples using a predefined prefix.

This works when using $ref to point to re-usable components: the request example create-user-us is associated (1:1) with the response example create-user-us-200 . The examples share the same prefix while the response appends the status code.

/users:
    post:
      summary: Create user
      requestBody:
        content:
          application/json:
            examples:
              example-user-eur:
                $ref: '#/components/examples/create-user-eur'
              example-user-us:
                $ref: '#/components/examples/create-user-us'
              example-user-invalid:
                $ref: '#/components/examples/user-error'      
    responses:
        '200':
          content:
            application/json:
              examples:
                create-user-1:
                  $ref: '#/components/examples/create-user-eur-200'
                create-user-2:
                  $ref: '#/components/examples/create-user-us-200'        
        '422':
          content:
            application/json:
              examples:
                validation-error:
                  $ref: '#/components/examples/user-error-422'

Summarising in a Given-When-Then style:

• given create-user-eur when POST /users then create-user-eur-200

• given create-user-us when POST /users then create-user-us-200

• given user-error when POST /users thenuser-error-422

Pros and cons

There is no significant complexity to implementing this. It only requires a diligent approach to make sure the naming is defined and correctly applied.

On the other side, there are a few limitations:

• it only works with $ref (references to components) and therefore not helpful for requests without a body (schema)

• linting (validation) becomes complicated (involving custom implementation with partial text matching)

• it cannot link multiple requests to the same response, for example, a shared 404 error response

Custom extension approach

Another way is to leverage the OpenAPI extensions. This works in this way: define a custom property x-contract-id that tags the interaction between a request and a response.

  ...
  examples:
    create-user-eur:
      summary: Create user example
      x-contract-id: post-user-eur
      value:
        firstname: Beppe
        lastname: Catanese
    new-user:
      summary: Create user example response
      x-contract-id: post-user-eur
      value:
        id: 00001
        createdOn: 2023-05-31T02:55:00.1Z

Summarising in a Given-When-Then style:

• given create-user-eur when POST /users then new-user

As you notice there is no constraint on naming the examples.

Pros and cons

This works, in my opinion, a lot better. It doesn’t rely on strict naming patterns and it can use the same x-contract-id multiple times (if necessary). I find it convenient that the matching interactions can be spotted more easily by looking or searching in the examples section of the specification.

Another advantage is that, unlike the naming approach, it can be used with parameters or inline examples. This is how we could define a response for a GET request (without a body):

parameters:
  - description: User identifier
    name: id
    in: path
    required: true
    schema:
      type: string
    examples:
      get-transaction:
        x-contract-id: get-user-example-1
        value: 1

The downside is the introduction of an extension. As with any customisation, it introduces a new non-standard element that needs to be dealt with accordingly, typically requiring some extra work by the consuming tool or application.

Final thoughts

Despite the current limitations in supporting contract testing directly from an OpenAPI file, exploiting naming conventions or leveraging extensions can help overcome this.

Being able to define the interaction flows within the OpenAPI specification will give the API designers an additional element to describe the API behaviour. It will also create opportunities for OpenAPI-driven mock servers (where requests and responses are defined within the examples), Postman collection testing and improving support for contract testing tools that today need to define (or record) those interactions.

My preferred solution is the custom extension (for the reasons explained above) and I will be building a POC very soon. Stay tuned.

Photo by sydney Rae on Unsplash

Are you facing similar challenges? Did you find other solutions? I am looking forward to hearing your take on this and starting a debate.