Learn from experts and connect with global tech leaders at POST/CON 24. Register by March 26 to save 30%.

Learn more →
X

How to write powerful automated API tests with Postman, Newman and Jenkins

Avatar

Unless your development team is running on a six-month or an year-long cycle, you would be practicing Continuous Integration. From the ThoughtWorks definition of Continuous Integration:

Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

While source control systems have made it trivial to set up shared repositories of code and view what people have been sharing, few teams are able to practice well the second part – verifying check-ins through an automated build process. Writing tests and especially tests for APIs, can be a tedious process. Nobody would disagree that having tests is good and we should be running them as often as possible. In practice, writing tests, verifying whether they are working, setting up the testing environment and then eventually plugging them with the build system is hard. Teams tend to skip this part accruing a ton of technical debt in the process.

We believe that in the context of API development, this process can be made a lot easier. This is where Postman departs from just being a REST client in your arsenal. Postman contains a full-featured testing sandbox that lets you write and execute Javascript based tests for your API. We won’t go into the specifics here but do check out this tutorial for more detail.

Screen Shot 2016-06-27 at 15.12.11

The next logical step is hooking up Postman with your build system. This is where Newman, Postman’s command line companion comes in. Think of Newman as Postman’s Collection Runner engine that sends API requests, receives the response and then runs your tests against the response.

In this tutorial, we are going to use Jenkins as an example. The same steps apply everywhere though. Jenkins is one of the most popular continuous integration servers available right now. Jenkins primary goal is to help you build and test software projects continuously. You can also monitor executions of externally run jobs too.

Newman and Jenkins are a perfect match. Lets start setting this up. We are using Ubuntu as a target OS as in most cases your CI server would be running on a remote Linux machine.

Install Jenkins: The process is straightforward. Just follow the instructions here.

Install NodeJS and npm. Newman is written in NodeJS and we distribute the official copy through npm. Follow the instructions for Linux to install nodejs and npm here.

Install Newman through npm install -g newman. This would set up Newman as a command line tool in Ubuntu.

Run a sample Postman Collection. We are assuming that you already have a Postman Collection with some tests. I am using a collection that sends two requests to echo.getpostman.com. You can download this sample collection if you want to follow the example. This is what the output looks in Postman’s Collection Runner:

Screen Shot 2016-06-27 at 15.14.02

Some of my tests are failing intentionally in the screenshot. We will fix this later in the tutorial.

To run this collection inside newman, use the command [command here]. If everything is set up nicely, you should see the output below.
Screen Shot 2016-06-27 at 16.31.48

Jenkins exposes an interface at http://localhost:8080. This is what it looks like:

step-3

Create a new job by clicking on the “New Item” link on the left sidebar. Then select a “Freestyle Project” from the options shown. I am calling my project Jenkins_Newman_Test.

step-4

Add a build step in the project. The build step executes a Shell command.

step-5

The command is:

newman -c jenkins_demo.postman_collection --exitCode 1.

Note here that we are using the newman command parameter “exitCode” with the value 1. This denotes that newman is going to exit with this code that will tell Jenkins that everything did not go well. Click the save button to finish creating the project.

step-6

Lets run this build test manually by clicking on the “Build Now” link in the sidebar.

step-7

Jenkins indicates that the build has failed with a red dot in the title. We can check why with the console output from newman.

step-8

Click on the “Console Output” link in the sidebar to see what newman returned.

step-9

Let’s fix these tests inside Postman and then try again. You can download the updated collection here.

step-10

I have updated my collection now. Running these tests inside Jenkins tells me that everything worked!

step-11

Jenkins indicates that the build succeeded with a blue ball. If you are wondering why it is not green, check out this article.

Let’s change the Jenkins build trigger as every 30 mins. You can do this by clicking on “Configure project” in the main project window and then scrolling down.=. The syntax for setting the frequency is H/30 * * * *

step-12

And we are all done! Jenkins will now run Newman every 30 mins and will tell you whether the build failed or succeeded. In a bigger set up, Newman will be part of your build process and probably not the entire process. You can set up notifications and customize Jenkins as per your needs.

You can use a wide variety of other configurations to make your collection more dynamic. Check out the other tutorials on our blog to see how to do this. Postman, Newman and Jenkins will give you a lot of power along with flexibility. We are continuously evolving the testing sandbox and let us know your suggestions on our Github issue tracker. Of course, do share this article on Twitter if you found it useful!

Comment

Your email address will not be published. Required fields are marked *


This site uses Akismet to reduce spam. Learn how your comment data is processed.

30 thoughts on “How to write powerful automated API tests with Postman, Newman and Jenkins

  • Avatar

    Hi! Great job on this, works like a charm. Is ist possible to export / dump a postman collection automatically (via some shell script for example) ?

    • Avatar

      Glad you liked the tutorial. It is not possible to export the Collection automatically but we will have hooks for this through our Sync service soon.

      • Avatar

        Would be awesome to be able to reference postman’s environment setups for my Jenkins integration. For the time being I’m just periodically exporting the environment variables along with a snapshot of the collections to a common place for Jenkins to reference.

      • Avatar

        Hi! Great job on this, works like a charm. Is ist possible postman can integrate with Travis CI for testing.? If is is possible please help me.

      • Avatar

        Hi! Great job on this, works like a charm. Is ist possible to integrate postman with Travis CI for API testing? if its possible please help me on this.

  • Avatar

    I am trying to Integrate Postman scripts using Newman with Bamboo. Can any one suggest help me. I am unable to install newman facing some issues with proxy since it is behind a corporate proxy.

    Any tips on configuring the proxy to run behind corporate?

  • Avatar

    Hi I’m running on Windows, but due to this have to use a batch file rather than shell – the demo above has the fails as expected but is marked as Finished: SUCCESS and makes as passed in Jenkins. I’m guessing it’s something to do with the –exitcode 1? any ideas how I get this to work as expected?

    • Avatar

      It’s OK I’ve sorted this out – was missing the -x code in the batchfile calling newman

      • Avatar

        I’m having the same problem. Can you write the content of your batch file (or a similar example)? Thanks

  • Avatar

    OK another question, in the jenkins console output I’m not getting ticks or crosses – instead I’m getting √ or × which font do I need to correct this issue? thanks

  • Avatar

    Hi, i run collections on command line at jenkins as url. Like this newman -u https://www.postman.com/collections/XXXXXXX -e TEST.postman_environment -g globals.postman_globals --stopOnError but i can not see to fail cases on jenkins’s console output. its look like all of them passed but one case must be fail.

  • Avatar

    I have an issue running Newman command from Jenkins(below is Console output). I can successfully run newman command from cmd prompt(newman -c C:UserslholmesDesktopPostmanCollectionsReciva.json.postman_collection)

    Console:
    Building in workspace C:Program Files (x86)JenkinsworkspaceJenkins_Newman_Test
    [Jenkins_Newman_Test] $ sh -xe C:WindowsTEMPhudson6298287477471424761.sh
    The system cannot find the file specified
    FATAL: command execution failed
    java.io.IOException: Cannot run program “sh” (in directory “C:Program Files (x86)JenkinsworkspaceJenkins_Newman_Test”): CreateProcess error=2, The system cannot find the file specified
    at java.lang.ProcessBuilder.start(Unknown Source)

    environment: Windows, running Jenkins on http://localhost:8080/.

    Any advise?

    • Avatar

      Try using ‘Execute Windows batch command’ action instead of ‘Execute shell’

  • Avatar

    Hi, This looks very nice!
    Is there a way to display a results historical trend line like the one available for Junit? maybe using an additional plugin?
    Thanks!

  • Avatar

    Hi Team, while configuring postman collection with Jenkins, i am getting below error in Jenkins console output. i installed NodeJS and npm and nowman.

    Building in workspace C:Users#####.jenkinsjobsAPI_Testworkspace

    [workspace] $ cmd /c call C:Users#####AppDataLocalTemphudson#######.bat

    C:Users#####.jenkinsjobsAPI_Testworkspace>newman -c D:/apitest.json.postman_collection

    ‘node’ is not recognized as an internal or external command,

    operable program or batch file.

    Build step ‘Execute Windows batch command’ marked build as failure

    Finished: FAILURE

    • Avatar

      You may have to specify the full path for newman – instead of newman -c ……the command should be C:node_modules.binnewman -c

  • Avatar

    Is there a way to write data to external CSV file ?

  • Avatar

    Hey Guys,

    How will we handling login and logout flow:

    While login we get authorization token and same token is need to logout, how will we using postman.setNextRequest(“”);

    and then performing tests on logout api.

    Any help is highly appreciable.

    Thanks in advance.

    p.s new to api testing.

    • Avatar

      Hi!
      For me I use a collection where I have my login request and logout request (this way I don’t have to call postman.setNextRequest(“”) BUT if you absolutely need the request caller you need to write the query name in the double quotes like this :
      login request name : GET login
      logout request name : POST logout
      in the test section of GET login, when you have done your verification you write postman.setNextRequest(“POST logout”); and it will work fine)

      And then I recommend you to take a look at environment and global variables ! It’s a great plus to handle your session token in all your requests !

      I hope this helps you!

      spouki 🙂

  • Avatar

    What kind of API you test for every 30mins?
    I plan to run my all of API (around 100 APIs) by daily, is it the right approach?

  • Avatar

    Is there a way to run server side database queries (eg: SQL queries) to build test data from the database and then read it as an input for postman API calls?

  • Avatar

    I am trying to run the json collection through jenkins windows command prompt. I given the “newman run C:Sanity.postman_collection.json”. I am getting response as

    Building in workspace C:Program Files (x86)JenkinsworkspaceOne touch
    [One touch] $ cmd /c call C:windowsTEMPhudson6168983955557336367.bat

    C:Program Files (x86)JenkinsworkspaceOne touch>newman run C:Sanity.postman_collection.json
    ‘newman’ is not recognized as an internal or external command,
    operable program or batch file.

    But when i am trying came from Windows command prompt its working. Looks weird why through jenkins its not working and through windows cmd its working

  • Avatar

    Has anyone tried to generate code coverage for Sonar, while running the Postman tests?

  • Avatar

    How to integrate postman/Newman with failsafe maven plugin ?

  • Avatar

    This seems nice and i’ve tried it works like charm, It would be great if any one tried to use any plugins to display results, i.e.serenity plugin for serenity tests but in this case i’m not sure of what to use.

  • Avatar

    Works great.
    But please update the deprecated command:
    newman -c jenkins_demo.postman_collection –exitCode 1.

    Now it’s:
    newman run jenkins_demo.postman_collection

  • Avatar

    Nice artikel! I will use this with TeamCity.
    One remark I think you forgot the set the correct command line:

    “To run this collection inside newman, use the command [command here]”

  • Avatar

    Pls note that in the latest versions of newman,
    newman -c jenkins_demo.postman_collection will do.

    No more exitcode…

  • Avatar

    Hi Team, Can Postman and Jenkins create scheduled POSTS to run an API? I want the POSTMAN to schedule weekly POSTs that will start an Cisco Meeting Server LDAP sync using their built-in API

    • Avatar

      Hi Jim, You will need to set up a cron job in Jenkins and use Newman. Please see our Learning Center page here to learn how.