Why won't these chai tests fail?

Question

We have some simple "is this really working" chai tests of an electron app using spectron and WebdriverIO. The example code we started with is from

https://github.com/jwood803/ElectronSpectronDemo as reported in https://github.com/jwood803/ElectronSpectronDemo/issues/2, the chai-as-promised tests are not catching mismatches, so I thought I would add some additional tests to find out why Chai is not failing tests where the electron app has text that doesn't match the expected unit test text.

Let's start with something really simple, the rest of the code is at https://github.com/drjasonharrison/ElectronSpectronDemo

describe('Test Example', function () {
    beforeEach(function (done) {
        app.start().then(function() { done(); } );
    });

    afterEach(function (done) {
        app.stop().then(function() { done(); });
    });

    it('yes == no should fail', function () {
        chai.expect("yes").to.equal("no");
    });

    it('yes == yes should succeed', function () {
        chai.expect("yes").to.equal("yes");
    });

The first unit test fails, the second succeeds.

And when we put the assertion into a function this still detects the failure:

it('should fail, but succeeds!?', function () {
    function fn() {
        var yes = 'yes';
        yes.should.equal('no');
    };
    fn();
});

So now into the world of electron, webdriverio, and spectron, the application title is supposed to be "Hello World!", so this should fail, but it passes:

it('tests the page title', function () {
    page.getApplicationTitle().should.eventually.equal("NO WAY");
});

Hmm, let's try a more familiar test:

it('should fail, waitUntilWindowLoaded, yes != no', function () {
    app.client.waitUntilWindowLoaded().getTitle().then(
        function (txt) {
            console.log('txt = ' + txt);
            var yes = 'yes';
            yes.should.equal('no');
        }
    );
});

Output:

    ✓ should fail, waitUntilWindowLoaded, yes != no
txt = Hello World!

It succeeds? What? Why? How?

Answer 1

Found it! If you look at https://github.com/webdriverio/webdriverio/blob/master/examples/standalone/webdriverio.with.mocha.and.chai.js

you will see that you need to return the promise from each of the tests. This is typical for async chai/mocha tests:

it('tests the page title', function () {
    return page.getApplicationTitle().should.eventually.equal("NO WAY");
});

If you do that, then the chai test is actually correctly evaluated.