Automating API Testing with Postman, Newman and DevOps Pipelines

So you've written an API test suite in Postman with excellent test naming conventions.

It's time to get DevOps to do the rest of the hard work for you and schedule nightly or weekly runs of your tests.

Firstly, you'll need to export your test suite as JSON as well as the environment and global JSONs. Open up your favourite text editor and change all the environment and global variable values to tokens using __{variable_name}__ (double-underscores):

{
	"id": "c2064aba-2c4d-4b8b-b025-c81af835798a",
	"name": "Contact API Production",
	"values": [
		{
			"key": "url",
			"value": "__ApiUrl__",
			"type": "default",
			"enabled": true
		},
		{
			"key": "companySearchTerm",
			"value": "__CompanySearchTerm__",
			"type": "default",
			"enabled": true
		},
		{
			"key": "companyId",
			"value": "__CompanyId__",
			"type": "default",
			"enabled": true
		}
	],
	"_postman_variable_scope": "environment",
	"_postman_exported_at": "2022-05-24T13:42:02.825Z",
	"_postman_exported_using": "Postman/9.18.2"
}

{
	"id": "6fcc71a9-8e5a-4431-86bb-0bd426788177",
	"values": [
		{
			"key": "username",
			"value": "__Username__",
			"type": "default",
			"enabled": true
		},
		{
			"key": "password",
			"value": "__Password__",
			"type": "secret",
			"enabled": true
		}
	],
	"name": "Globals",
	"_postman_variable_scope": "globals",
	"_postman_exported_at": "2022-05-23T11:01:16.517Z",
	"_postman_exported_using": "Postman/9.18.2"
}

And upload all of these files to a DevOps repo. Now we need a new build pipeline, so create one. I like the classic pipeline editor, but a YAML example is below.

The first build step is to use the "Replace Tokens" task to fill our environment and global JSON files with the correct settings for this environment. The "Target Files" should be:

**/*_environment.json
**/*_globals.json

And the "Token Pattern"

__...___

The 2nd step is to install Newman using the "npm" task. Change the "Command" to "Custom" and the "Command and arguments" to:

install -g newman

The 3rd task is "Command line".

newman run "$(Build.SourcesDirectory)\Contact API\CE Connector.postman_collection.json" -e "$(Build.SourcesDirectory)\Contact API\Contact API.postman_environment.json" -g "$(Build.SourcesDirectory)\workspace.postman_globals.json" -x -r junit --reporter-junit-export "$(build.artifactstagingdirectory)\Results\Contact API.xml" --insecure

This is Newman running your tests from the CLI. The first parameter is the test export JSON, the -e parameter is the environment variable export JSON, and the -g  parameter is the global variable export JSON.

-x finishes the test suite even in the event of an exception, and we use junit to define the results file.

The 4th step is "Publish Test Results". Change the "Test result format" to JUnit, the "Test results files" pattern to **/*.xml and the "Search folder" to $(build.artifactstagingdirectory).

That's it! Change the pipline triggers to whatever schedule you want, and each run will publish the results in the run. Of course, you probably want some reporting on these so you don't have to go hunting down the release each time it runs, so we'll add a 5th step - "Email Report". Set whatever recipients and options you want in here, with the "Email Subject" of:

[{environmentStatus}] {passPercentage} tests passed for Contact API on build $(Build.BuildNumber)

environmentStatus and passPercentage are built-in properties of the task.

Here's the YAML for the whole pipleline described above.

pool:
  name: On Premise
  demands: npm

steps:
- task: qetza.replacetokens.replacetokens-task.replacetokens@5
  displayName: 'Inject variables'
  inputs:
    targetFiles: |
     **/*_environment.json
     **/*_globals.json
    tokenPattern: rm

- task: Npm@1
  displayName: 'Install Newman'
  inputs:
    command: custom
    verbose: false
    customCommand: 'install -g newman'

- script: 'newman run "$(Build.SourcesDirectory)\Contact API Production.postman_collection.json" -e "$(Build.SourcesDirectory)\Contact API Production.postman_environment.json" -g "$(Build.SourcesDirectory)\workspace.postman_globals.json" -x -r junit --reporter-junit-export "$(build.artifactstagingdirectory)\Results\Results.xml" --insecure'
  displayName: 'Run Contact API Production tests'

- task: PublishTestResults@2
  displayName: 'Publish test results'
  inputs:
    testResultsFiles: '**/*.xml'
    searchFolder: '$(build.artifactstagingdirectory)'

- task: epsteam.EmailReportExtension.EmailReport.EmailReport@1
  displayName: 'Send Email Report Always'
  inputs:
    subject: '[{environmentStatus}] {passPercentage} tests passed for "Contact API Production" on build $(Build.BuildNumber)'
    toAddress: '$(ReportRecipients)'
    ccAddress: '$(ReportCcRecipients)'
    defaultDomain: mydomain.co.uk
    maxTestFailuresToShow: 1000
    smtpConnectionEndpoint: 'SMTP Connection'