How to run idempotence test for ansible role via Test-Kitchen?

Question

It is pretty easy with Travis-CI:

  # Run the role/playbook again, checking to make sure it's idempotent.
  - >
    ansible-playbook -i tests/inventory tests/test.yml --connection=local --sudo --extra-vars "take_ownership_of_tmp=$OWN_TMP"
    | grep -q 'changed=0.*failed=0'
    && (echo 'Idempotence test: pass' && exit 0)
    || (echo 'Idempotence test: fail' && exit 1)

but I cannot use Travis, because I need to test my role on Debian systems. I use Test-Kitchen with ansible_playbook provisioner.

For example:

my .kitchen.yml

---
driver:
  name: vagrant

provisioner:
  name: ansible_playbook
  hosts: test-kitchen
  ansible_verbosity: 2
  ansible_verbose: true
  require_ansible_repo: false
  require_chef_omnibus: false
  require_ansible_omnibus: true
  require_chef_for_busser: false
  ansible_omnibus_url: https://raw.githubusercontent.com/neillturner/omnibus-ansible/master/ansible_install.sh

platforms:
  - name: debian-6.0.10-64-nocm
    driver_config: 
      customize:
           memory: 1024
           cpus: 2
      box: puppetlabs/debian-6.0.10-64-nocm
      box_url: https://atlas.hashicorp.com/puppetlabs/boxes/debian-6.0.10-64-nocm/versions/1.0.2/providers/virtualbox.box


suites:
  - name: default

verifier:
  ruby_bindir: '/usr/bin'

test playbook (test/integration/default/default.yml) is very simple

---
- hosts: all
  roles:
    - preconf
  vars:
    #some vars here

I may add second call of preconf role to the default.yml, but it is not helps.

With one call kitchen returns me a number of changed items:

   PLAY RECAP ******************************************************************** 
   localhost                  : ok=12   changed=8    unreachable=0    failed=0  

But with two calls it returns sum of items instead of two separate results

   PLAY RECAP ******************************************************************** 
   localhost                  : ok=24   changed=10   unreachable=0    failed=0 

So, how I can run the playbook second time and check the results for idempotence test?

Answer 1

Solved for myself with BATS test:

#!/usr/bin/env bats
#

#
# Idempotence test
#

@test "Second run should change nothing" {
    run bash -c "ansible-playbook -i /tmp/kitchen/hosts /tmp/kitchen/default.yml -c local | grep -q 'changed=0.*failed=0' && exit 0 || exit 1"
    [ "$status" -eq 0 ]
}

UPD Serverspec variant:

describe command('ansible-playbook -i /tmp/kitchen/hosts /tmp/kitchen/default.yml -c local') do
  its(:stderr) { should match /changed=0.*failed=0/ }

  its(:exit_status) { should eq 0 }
end