Validating API json structures with Tavern

This article is a continuation of this introduction to Tavern I wrote back in 2021 when Tavern was still in version 1.x. Tavern is now available in version 2.x.

TL;DR: Tavern is an open source YAML cli-based testing tool which we use a lot at Kalvad for our REST API automated tests. This article will assume you are familiar with Tavern and approach a particular topic.

Validating data

With Tavern, you can easily validate the data of your REST API's response, for instance:

  - name: Fetch an animal
    request:
      url: "{base_url}/animal"
      method: GET 
    response:
      status_code: 200 
      json:
        id: !anystr
        name: "cat"

In this simple example we are fetching a single animal and expecting the response to be exactly:

• id : any type of string
• name: the string "cat" (case sensitive)

So for instance this response would pass:

{"id": "abc123", "name": "cat"}

But we are not limited to strings only, you can use any of those data types. You can easily guess what they are supposed to match from their name:

• !anyint
• !anyfloat
• !anystr
• !anybool
• !anylist
• !anydict

Now for a list of animals you could write something like this:

  - name: Fetch some animals
    request:
      url: "{base_url}/animals"
      method: GET 
    response:
      status_code: 200 
      json:
      	animals:
          -
            id: !anystr
            name: "cat"
          -
            id: !anystr
            name: "dog"

Here you would expect a list of those exact 2 animals, so for instance this would pass:

{"animals": [{"id": "abc123", "name": "cat"}, {"id": "abc456", "name": "dog"}]}

This concept works well when you want to test APIs with specific data. Like returning a list of regions in a country, which is the sort of things which never or rarely changes; or in a more complex scenario where you do price calculations for an e-commerce platform. But if you want to test something more dynamic such as the last 25 comments from a blog post, you probably want to test the structure of the response rather than the data itself.

Validating structure

To validate the response data we used the simple YAML structure response:json but to test the structure of the response we are going to use a helper: tavern.helpers:validate_pykwalify. The code for this helper can be found on the Github repository, as you can see its pretty straightforward: we try to load the json and then we pass it to pykwalify to run a check.

If we wanted to rewrite the list of animals example, so we could have 2 platypuses and a blue turtle from time to time, we would do something like this:

  - name: Fetch some animals
    request:
      url: "{base_url}/animals"
      method: GET
    response:
      status_code: 200
      verify_response_with:
        function: tavern.helpers:validate_pykwalify
        extra_kwargs:
          schema:
            type: map
            mapping:
              animals:
                type: seq
                required: True
                sequence:
                - type: map
                  mapping:
                    id:
                      type: str
                      required: True
                    name:
                      type: str
                      required: True

So the structure of our YML example got a bit more complex but still manageable nonetheless, let's break it down:

      verify_response_with:
        function: tavern.helpers:validate_pykwalify

We tell Tavern that we want to load a python function to validate the response.

Then we use extra_kwargs to feed the second parameter of the function (remember the signature of the helper: def validate_pykwalify(response, schema) -> None:), response is always passed on by default, here we are going to give our schema.

Now for the actual schema, because we do not return directly a list of animals, we have a top level key called animals so we need to define a map (a dict basically):

          schema:
            type: map
            mapping:
              animals:

Then we need to define that animals is a list so we need to define a seq:

              animals:
                type: seq
                required: True

Finally we need to define the structure of our list by saying that our "sequence" (list) contains some "map" (dict) with 2 required fields: an "id" of type string and a "name" of type string.

                sequence:
                - type: map
                  mapping:
                    id:
                      type: str
                      required: True
                    name:
                      type: str
                      required: True

Now we are ready to manage the automated testing of a whole zoo with this kind of setup !

Of course you can do some more complex stuff such as nested dict/list, test various types, patterns and formats, and so on. You can find out more on Pykwalify's home page.

Conclusion

Tavern is a very versatile tool which allows you to test your REST APIs in a variety of ways and is continuously getting updated. If you are not using it yet, give it a go and check out their documentation.

If you have a problem and no one else can help. Maybe you can hire the Kalvad-Team.