Learn from experts and connect with global tech leaders at POST/CON 24. Register by March 26 to save 30%.

Learn more →
X

Writing a behaviour driven API testing environment within Postman

Avatar

Behaviour driven test frameworks like mocha and jasmine have certain unalienable advantages. They provide a clean way to organize tests as well as safe and (in this particular case,) isolated assertions to prevent runtime error of the testing spec itself.

For obvious reasons, having the same test specification style in Postman sounds just about right. After all, one may argue that they are testing the behavior of the response! And not to mention the endless benefits of well organized tests, seamless creation and teardown of fixtures, yada yada yada!

the_general_problem_xkcd

I would personally love to write tests within Postman that look like the following. (Never mind the aforementioned xkcd :-/ )

describe("response", function () {
  var responseJSON;
  
  it("must be a valid JSON", function () {
    responseJSON = JSON.parse(responseBody);
    expect(typeof responseJSON).toBe("object");
  });
  
  describe("json", function () {
    it("must not have error object", function () {
      expect(responseJSON.error).toNotBeOk();
    });
    
    it("must have a message", function () {
      expect(responseJSON.message).toBeOk();
    });
  });
});

Thus began my transit-time project to create a tiny (really tiny) test framework that I could include in my Postman Collections and write tests using it. I personally started using it and thought sharing it might help us BDD aficionados. The framework ended up being a gist that is about 0.7kb when minified. I named the script describe-it (such creativity!)

The “describe-it” test framework script

The following piece of minified code is the complete test framework that provides behavior driven testing functionalities (albeit basic) similar to mocha and jasmine. The script does look cryptic — but that is to ensure that it is as lightweight as possible.

(typeof tests!=='object')&&(tests={});var
it=((it=function(k,v){it.d.push(k);it.t[it.d]=1;it.b.forEach(it.c);try{v()}catch(e){it.t[it.d]=0;setTimeout&&setTimeout(function(){throw e;})}it.a.forEach(it.c);it.d.pop()}),
it.a=[],it.b=[],it.c=function(x){x()},it.d=[],it.d.toString=function(){return this.join(' ')},
it.t=tests,it.x=function(v){this.v=v},it.xp=it.x.prototype,it.xp.toBe=function(x){(this.v!==x)&&it._()},
it.xp.toNotBe=function(x){(this.v===x)&&it._()},it.xp.toEql=function(x){(this.v!=x)&&it._()},
it.xp.toNotEql=function(x){(this.v==x)&&it._()},it.xp.toBeOk=function(){!this.v&&it._()},
it.xp.toNotBeOk=function(){this.v&&it._()},it),describe=function(k,v){it.d.push(k);v();it.d.pop()},
expect=function(v){return new it.x(v)},beforeEach=it.b.push.bind(it.b),afterEach=it.a.push.bind(it.a);

The not-so-cryptic version of the script is in the gist: https://gist.github.com/shamasis/7b12f451374314500da3 (Although, my colleagues rightfully contradict that it’s still cryptic.)

The script exposes the following functions:

  • describe (testgroup:string, tests:function);
  • it (testname:string, test:function);
  • beforeEach (setup:function);
  • afterEach (teardown:function);
  • expect (value); // assertion function

These functions work similar to mocha/jasmine and as such referring to their documentation is an obvious shortcut I would take — https://jasmine.github.io/2.0/introduction.html

The “expect” assertion function supports the following checks:

  • expect(something).toBeOk(); // allows truthy variables
  • expect(something).toNotBeOk(); // allows falsey variables
  • expect(something).toBe(somethingElse); // checks equality
  • expect(something).toNotBe(somethingElse); // checks inequality
  • expect(something).toEql(somethingElse); // checks similarity
  • expect(something).toNotEql(somethingElse); // checks dissimilarity

We might need more of these and especially ones that are related to API testing. (Would love to hear more in comments.)

Using the script within Postman

There are two parts to using this within Postman:

  1. Save the script as a global variable
  2. In the test scripts, eval the global variable before using the framework

Saving the script as a global variable

Saving the script as a global variable allows me to re-use the test framework script in multiple requests without having to manually add them. One can simply go and add the contents of the script to a global variable (or an environment variable) with name, say, describe-it-test-script. The documentation article Our Manage Environments documentation outlines how to set and use these variables.

Personally, I save the test script as part of the first request to ensure that the test script is bundled with Postman Collections while running them in Collection Runner or Newman

The following collection fetches the test framework from the gist (so that I can keep it updated with latest content) and saves it in global variables. The subsequent requests use this global variable to get the test framework within each request’s test script.

Sample Collection that uses this test framework:
www.getpostman.com/collections/34163da95c4372e36c39

Import it in your postman and refer to the tests written in the collection called “describe-it” them.

Using the script in tests

Once you have the script stored in a variable, it is easy to use it in any test by running eval on the script. The first line of your test would look somewhat like the following:

eval(postman.getGlobalVariable("describe-it-test-script"));

I know — eval is evil! But that is a separate discussion altogether. If you want to avoid “eval” altogether, simply paste the minified script at the beginning of your tests. But that would make the tests look a bit ugly.

Post this, you can start using the describe, it, expect and other functions of the test framework normally.

Working with the test framework

Say, you have already saved the framework in a global variable, and you would like to run a test that ensures that a request responds with a 200 status code and serves HTML content type. The tests for the same would look like:

eval(postman.getGlobalVariable("describe-it-test-script"));
 
describe ("response", function () {
  it ("must have html content type header", function () {
    expect(responseHeaders['Content-Type'])
      .toBe('text/html; charset=utf-8');
  });
 
  it ("must return a 200 status code", function () {
    expect(responseCode.code).toEql(200);
  });
});

We can write as many tests as we want and nest as many describe blocks as we need. If there are runtime errors in one it block, the other tests would still continue to execute sequentially.

Postman Collection Run The collection shared in this post (named describe-it,) has examples of tests written using this micro framework. It has tests written with failing and passing scenarios. And upon running the collection in Postman Collection Runner, it outputs the test results grouped within their respective describe blocks.

How it works

The it function executes the test parameter function within a try-catch block. If the function executes without an error, it adds to the test description parameter (along with the description of it’s parent describe blocks) to the Postman tests variable and sets it to a truthy value. In the event that an expect assertion fails or some other error occurs, the tests object key is set to a falsey value.

In the above example, if all tests pass, the tests object looks like

tests: {
  "response must have html content type header": 1,
  "response must return a 200 status code": 1
}

Comment

Your email address will not be published. Required fields are marked *


This site uses Akismet to reduce spam. Learn how your comment data is processed.

16 thoughts on “Writing a behaviour driven API testing environment within Postman

  • Avatar

    Hello Shamasis, thanks for that!
    This is a great addition to testing in Postman. Would be great to see it comes with standard Postman. Another idea is to have it as an extension or plugin.

    • Avatar

      I would like to know more about why this way of testing is so useful to you. It would help us craft this in right direction.

      • Avatar

        Hello @shamasis:disqus , it’s mainly because the test and the results are easy to read, this turns out also to be easy to design.

  • Avatar

    Using Tests how can I compare JSON response to previously saved JSON response from same request?

    • Avatar

      You could store the JSON in environment variable and when the request is re run, compare the results. @tmack8001’s script is cool.

  • Avatar

    Itest I have done this before. I have even uses “Tests” in postman to compare between a legacy system and the new system that we are replacing said legacy system with (only works if systems are written to in parity).

    The simplest way to answer your question is via the following:

    // store the responseBody for future comparisons
    var oldResponse = postman.getEnvironmentVariable(“unique-to-this-test”);
    if (oldResponse === null) {
    tests[‘no previous value to compare’] = true;
    postman.setEnvironmentVariable(“unique-to-this-test”, responseBody)
    } else {
    // compare old response with current response
    // depending on what you are doing something like the following will be good enough, but this requests the responseBodies to be exact.
    tests[‘response matches previous’] = (oldResponse === responseBody)
    }

    If you want to do say a deepEquals() where the attributes in the responseBody are the same, but are in a different order then you will need to customize the above a bit to your specific needs. And obviously I prefer to use the behaviour testing that is mentioned in this blog post, so feel free to update the script I gave to be except(responseBody).toEql(oldResponse) instead.

    This way allows you to test any type of response and not just JSON, if you need to parse the json and compare it you should then do something different.

  • Avatar

    I saved the script as global variable but when I run tests, I get ‘describe is not defined’.

    • Avatar

      Did you run the ‘eval’ code in the first line of your tests? This line ensures that the functions are available in the script.

      eval(postman.getGlobalVariable(“describe-it-test-script”));

  • Avatar

    why don’t let us include external scripts… instead this hacky global var + eval nonsense?

  • Avatar

    This really need to be integrated into Postman. Atomic test is a great protection but it does not help to test complete scenarii. Kudo on this awesome premise.

    • Avatar

      I’d like to see it as a Post-success trigger. If all my atomic tests succeed then launch the scenarii.

  • Avatar

    After one year, did this ever made it into the core product?
    If not, are there any plans on incorporating it?

    Thanks

  • Avatar

    Is there a way to get this method to work within newman?
    When I try to load an exported environment file with the “describe-it-test-script”, newman won’t load it and make the test suite functions available to my test collection.

  • Avatar

    Hi there,
    Any news about some ability to import and use multiple scripts in test collections.
    Tried and used the setGlobalVariable / eval trick. It works like a charm (I extensively use postman-bdd that way), but is quite limitative when having a bunch of scripts.
    One of the tighter limit with postman_collection is about source control.
    A new easy mechanism to have external libraries made available would be great for that. So would be great to not update collection and tests guids on each collection import (same as above, makes version control quite tedious).
    Anyway, thanks for your great job.

  • Avatar

    It doesn’t seems to work any more. When i run the test collection, I have an error saying :”There was an error in evaluating the test script: Error: it._ is not a function”
    Have you any clue to solve the problem ?