How to write powerful automated API tests with Postman, Newman and Jenkins

Unless your development team is running on a six-month or an year-long cycle, you would be practicing Continuous Integration. From the ThoughtWorks definition of Continuous Integration:

Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

While source control systems have made it trivial to set up shared repositories of code and view what people have been sharing, few teams are able to practice well the second part – verifying check-ins through an automated build process. Writing tests and especially tests for APIs, can be a tedious process. Nobody would disagree that having tests is good and we should be running them as often as possible. In practice, writing tests, verifying whether they are working, setting up the testing environment and then eventually plugging them with the build system is hard. Teams tend to skip this part accruing a ton of technical debt in the process.

We believe that in the context of API development, this process can be made a lot easier. This is where Postman departs from just being a REST client in your arsenal. Postman contains a full-featured testing sandbox that lets you write and execute Javascript based tests for your API. We won't go into the specifics here but do check out this tutorial for more detail.

Screen Shot 2016-06-27 at 15.12.11

The next logical step is hooking up Postman with your build system. This is where Newman, Postman’s command line companion comes in. Think of Newman as Postman’s Collection Runner engine that sends API requests, receives the response and then runs your tests against the response.

In this tutorial, we are going to use Jenkins as an example. The same steps apply everywhere though. Jenkins is one of the most popular continuous integration servers available right now. Jenkins primary goal is to help you build and test software projects continuously. You can also monitor executions of externally run jobs too.

Newman and Jenkins are a perfect match. Lets start setting this up. We are using Ubuntu as a target OS as in most cases your CI server would be running on a remote Linux machine.

Install Jenkins: The process is straightforward. Just follow the instructions here: https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu

Install NodeJS and npm. Newman is written in NodeJS and we distribute the official copy through npm. Follow the instructions for Linux to install nodejs and npm here: https://docs.npmjs.com/getting-started/installing-node

Install Newman through npm install -g newman. This would set up newman as a command line tool in Ubuntu.

Run a sample Postman Collection. We are assuming that you already have a Postman Collection with some tests. I am using a collection that sends two requests to echo.getpostman.com. You can download this sample collection if you want to follow the example. This is what the output looks in Postman’s Collection Runner:

Screen Shot 2016-06-27 at 15.14.02

Some of my tests are failing intentionally in the screenshot. We will fix this later in the tutorial.

To run this collection inside newman, use the command [command here]. If everything is set up nicely, you should see the output below.
Screen Shot 2016-06-27 at 16.31.48

Jenkins exposes an interface at http://localhost:8080. This is what it looks like:

step-3

Create a new job by clicking on the "New Item" link on the left sidebar. Then select a "Freestyle Project" from the options shown. I am calling my project Jenkins_Newman_Test.

step-4

Add a build step in the project. The build step executes a Shell command.

step-5

The command is:

newman -c jenkins_demo.postman_collection --exitCode 1.

Note here that we are using the newman command parameter “exitCode” with the value 1. This denotes that newman is going to exit with this code that will tell Jenkins that everything did not go well. Click the save button to finish creating the project.

step-6

Lets run this build test manually by clicking on the "Build Now" link in the sidebar.

step-7

Jenkins indicates that the build has failed with a red dot in the title. We can check why with the console output from newman.

step-8

Click on the "Console Output" link in the sidebar to see what newman returned.

step-9

Lets fix these tests inside Postman and then try again. You can download the updated collection here.

step-10

I have updated my collection now. Running these tests inside Jenkins tells me that everything worked!

step-11

Jenkins indicates that the build succeeded with a blue ball. If you are wondering why it is not green, check out this article.

Lets change the Jenkins build trigger as every 30 mins. You can do this by clicking on "Configure project" in the main project window and then scrolling down.=. The syntax for setting the frequency is H/30 * * * *

step-12

And we are all done! Jenkins will now run newman every 30 mins and will tell you whether the build failed or succeeded. In a bigger set up, newman will be part of your build process and probably not the entire process. You can set up notifications and customise Jenkins as per your needs.

You can use a wide variety of other configurations to make your collection more dynamic. Check out the other tutorials on our blog to see how to do this. Postman, Newman and Jenkins will give you a lot of power along with flexibility. We are continuously evolving the testing sandbox and let us know your suggestions on our Github issue tracker. Of course, do share this article on Twitter if you found it useful!

  • Andreas Poppernitsch

    Hi! Great job on this, works like a charm. Is ist possible to export / dump a postman collection automatically (via some shell script for example) ?

    • Postman

      Glad you liked the tutorial. It is not possible to export the Collection automatically but we will have hooks for this through our Sync service soon.

      • tmack8001

        Would be awesome to be able to reference postman's environment setups for my Jenkins integration. For the time being I'm just periodically exporting the environment variables along with a snapshot of the collections to a common place for Jenkins to reference.

      • Amit Kumar

        Hi! Great job on this, works like a charm. Is ist possible postman can integrate with Travis CI for testing.? If is is possible please help me.

      • Amit Kumar

        Hi! Great job on this, works like a charm. Is ist possible to integrate postman with Travis CI for API testing? if its possible please help me on this.

  • Pingback: How to write powerful automated API tests with Postman, Newman and Jenkins http://blog.getpostman.com/2015/09/03/how-to-write-powerful-automated-api-tests-with-postman-newman-and-jenkins/ via @postmanclient()

  • Pingback: This week in API land #20 | Restlet - We Know About APIs()

  • Srinivas Kalyan

    I am trying to Integrate Postman scripts using Newman with Bamboo. Can any one suggest help me. I am unable to install newman facing some issues with proxy since it is behind a corporate proxy.

    Any tips on configuring the proxy to run behind corporate?

  • Claire Thomson

    Hi I'm running on Windows, but due to this have to use a batch file rather than shell – the demo above has the fails as expected but is marked as Finished: SUCCESS and makes as passed in Jenkins. I'm guessing it's something to do with the –exitcode 1? any ideas how I get this to work as expected?

    • Claire Thomson

      It's OK I've sorted this out – was missing the -x code in the batchfile calling newman

      • Pablo Sanz

        I'm having the same problem. Can you write the content of your batch file (or a similar example)? Thanks

  • Claire Thomson

    OK another question, in the jenkins console output I'm not getting ticks or crosses – instead I'm getting √ or × which font do I need to correct this issue? thanks

  • Ahmet Anıl Gürbüz

    Hi, i run collections on command line at jenkins as url. Like this "newman -u https://www.getpostman.com/collections/XXXXXXX -e TEST.postman_environment -g globals.postman_globals –stopOnError" but i can not see to fail cases on jenkins's console output. its look like all of them passed but one case must be fail.

  • Lana

    I have an issue running Newman command from Jenkins(below is Console output). I can successfully run newman command from cmd prompt(newman -c C:UserslholmesDesktopPostmanCollectionsReciva.json.postman_collection)

    Console:
    Building in workspace C:Program Files (x86)JenkinsworkspaceJenkins_Newman_Test
    [Jenkins_Newman_Test] $ sh -xe C:WindowsTEMPhudson6298287477471424761.sh
    The system cannot find the file specified
    FATAL: command execution failed
    java.io.IOException: Cannot run program "sh" (in directory "C:Program Files (x86)JenkinsworkspaceJenkins_Newman_Test"): CreateProcess error=2, The system cannot find the file specified
    at java.lang.ProcessBuilder.start(Unknown Source)

    environment: Windows, running Jenkins on http://localhost:8080/.

    Any advise?

    • Uri Gorodetsky

      Try using 'Execute Windows batch command' action instead of 'Execute shell'

  • Assaf Lowenstein

    Hi, This looks very nice!
    Is there a way to display a results historical trend line like the one available for Junit? maybe using an additional plugin?
    Thanks!

  • Pingback: 10 Continuous Integration Tools to Spur API Development | Nordic APIs |()

  • Pingback: Tutoriel Postman | Nouvelles Chroniques d'Amethyste()

  • Ravinder Singh Tanwer

    Hi Team, while configuring postman collection with Jenkins, i am getting below error in Jenkins console output. i installed NodeJS and npm and nowman.

    Building in workspace C:Users#####.jenkinsjobsAPI_Testworkspace

    [workspace] $ cmd /c call C:Users#####AppDataLocalTemphudson#######.bat

    C:Users#####.jenkinsjobsAPI_Testworkspace>newman -c D:/apitest.json.postman_collection

    'node' is not recognized as an internal or external command,

    operable program or batch file.

    Build step 'Execute Windows batch command' marked build as failure

    Finished: FAILURE

    • Shanmuga Raja Ramaswamy

      You may have to specify the full path for newman – instead of newman -c ……the command should be C:node_modules.binnewman -c

  • Madhu CS

    Is there a way to write data to external CSV file ?

  • amar

    Hey Guys,

    How will we handling login and logout flow:

    While login we get authorization token and same token is need to logout, how will we using postman.setNextRequest("");

    and then performing tests on logout api.

    Any help is highly appreciable.

    Thanks in advance.

    p.s new to api testing.

    • spouki

      Hi!
      For me I use a collection where I have my login request and logout request (this way I don't have to call postman.setNextRequest("") BUT if you absolutely need the request caller you need to write the query name in the double quotes like this :
      login request name : GET login
      logout request name : POST logout
      in the test section of GET login, when you have done your verification you write postman.setNextRequest("POST logout"); and it will work fine)

      And then I recommend you to take a look at environment and global variables ! It's a great plus to handle your session token in all your requests !

      I hope this helps you!

      spouki 🙂

  • Fadly Mahendra

    What kind of API you test for every 30mins?
    I plan to run my all of API (around 100 APIs) by daily, is it the right approach?

  • Tariq Ghalib

    Is there a way to run server side database queries (eg: SQL queries) to build test data from the database and then read it as an input for postman API calls?

  • arunan annamalai

    I am trying to run the json collection through jenkins windows command prompt. I given the "newman run C:Sanity.postman_collection.json". I am getting response as

    Building in workspace C:Program Files (x86)JenkinsworkspaceOne touch
    [One touch] $ cmd /c call C:windowsTEMPhudson6168983955557336367.bat

    C:Program Files (x86)JenkinsworkspaceOne touch>newman run C:Sanity.postman_collection.json
    'newman' is not recognized as an internal or external command,
    operable program or batch file.

    But when i am trying came from Windows command prompt its working. Looks weird why through jenkins its not working and through windows cmd its working

  • Georgi

    Has anyone tried to generate code coverage for Sonar, while running the Postman tests?

  • How to integrate postman/Newman with failsafe maven plugin ?

  • Pingback: Introduction to Postman with examples | Automation Rhapsody()

  • Sree Bandla

    This seems nice and i've tried it works like charm, It would be great if any one tried to use any plugins to display results, i.e.serenity plugin for serenity tests but in this case i'm not sure of what to use.

  • Pingback: Postman: gestiona y construye tus APIs rápidamente - Paradigma()