Search code examples
apipostmanazure-service-fabricnewman

What are best practises to run integration tests?

We have more than 150 postman tests. They are ** integration tests** and they run against actual databases and service fabric instances. They fail as they are not lined up with development who is merging to integration from time to time.

They are great to find some errors. It is a set of tests run on each new build of a product to verify that the build is testable before the build is released into the hands of the test team. We are using Newman to run them from the console. At the same time we want to improve our continuous deployment pipeline.

Questions

1.Where should we hold/run them ? Is there a cloud tool to run postman API tests ?

  1. How should we use/approach to them ? ( After every commit? Daily ? )

  2. Can we call postman API tests as integration or smoke tests ?

enter image description here

enter image description here enter image description here

enter image description here

Solution

  • My understanding of smoke tests is that they should be are relatively small in size (150 tests seems at the first sight too much) and practically "never" (or not too often) fail. You want to include only mission critical endpoints for your application and the tests should execute very fast.

    The scope of the smoke tests would be to mark a release / build or installation as unacceptable / failed by testing for simple failures, such as (but not necessarily limited to) is the status code 200 (or something else) and is the response in JSON format.

    I would not rely on smoke tests for find actual bugs in a specific REST endpoint, but rather to get a general overview that things are running.

    1.Where should we hold/run them ? Is there a cloud tool to run postman API tests ?

    Use version control to save them and use Jenkins or another CI tool in order to run them.

    Additionally you might want to run smoke tests after a deployment on a staging or production server.

    Postman offers some paid tools as well.

    How should we use/approach to them ? ( After every commit? Daily ? )

    They should be part of your pipeline. Fail fast! Run them if possible and reliable after every commit or build. If this is not possible because of external dependencies that are unreliable during the day - for example - run them at night.

    Can we call postman API tests as smoke tests ?

    You can call them whatever you like!

    The question is more like: "what are you exactly trying to achieve?". If your some tests do a bit too much or fail to often, it may be because they are more like integration tests.