Search code examples
unit-testingspockdbunit

Spock with DbUnit test cases slowness

We have implemented Spock + Db unit framework as a part of automated unit testing.

We have now 2000 test cases (Features) for 150 Specifications with DbUnit.

Here, We add required entries in the DB and then test the behavior of each method.

What We have observed that it takes around 2 Hrs and 30 Mins around time to execute these test cases.

I have timestamped setup fixture and added time stamp in a feature method. Below are my observations:

allergy.dao.AllergyFormDAOSpec > Get Allergy Form STANDARD_OUT
    setup method execution started at : Fri Jan 12 19:00:42 IST 2018

allergy.dao.AllergyFormDAOSpec > API to get Allergy Form STANDARD_OUT
Feature method execution started at : Fri Jan 12 19:00:44 IST 2018
Feature method execution ended at: Fri Jan 12 19:00:45 IST 2018
Total time taken to run the one test case: 242

cleanup method execution started at : Fri Jan 12 19:00:45 IST 2018
Total time taken to run a feature method : 2531

Here, I have observed that it takes average 2-4 seconds to load a feature method after setup. But, Original test case execution time is less than a second.

I want to know if I can get pointers on what could be delay here ? As, 3 seconds for 2000 test cases means almost 1 Hr and 30 Min of time taken by Spock other than real feature execution.

To Summarize, We want to reduce total time taken by Spock test cases when we daily run it.

Spec

package allergy.dao

import java.util.Date

import org.dbunit.IDatabaseTester;
import org.dbunit.ext.mssql.InsertIdentityOperation;

import allergy.AllergyForm;
import be.janbols.spock.extension.dbunit.DbUnit;
import spock.lang.Shared
import util.MasterSpec

class AllergyFormDAOSpec extends MasterSpec {
    def dao = new AllergyFormDAO();
    @Shared Date timeStart1
    @Shared Date timeEnd1

@DbUnit(configure={ IDatabaseTester it ->
    it.setUpOperation = InsertIdentityOperation.REFRESH
    it.tearDownOperation = InsertIdentityOperation.DELETE
})
def content =  {
    allergy_form(formId:99999,formName:'DummySpockForm',displayIndex:1,deleteFlag:0,is_biological:1)
    allergy_form_facilities(id:99999,formId:99999,facilityid:2)
    form_concentration(id:99999,formId:99999,name:'1:100',deleteflag:0,displayindex:1)
}

def setup(){
    timeStart1 = new Date()
    println "setup method execution started at : " +  timeStart1
}

def "API to test delete Form facility"(){
    def startTime = new Date()
    println "Feature method execution started at : " +  startTime
    given:"form Id is given"
        def formId = 99999
    when:"delete form facilities"
        def result =dao.deleteFormFacilities(null, formId)
    then:"validate result"
        (result>0)==true
        def endTime = new Date()
        println "Feature method execution ended at: " +  endTime
        println 'Total time taken to run the one test case: '+ (endTime.getTime() - startTime.getTime())
}

def cleanup() {
    timeEnd1 = new Date()
    println "cleanup method execution started at : " +  timeEnd1

    def difference = timeEnd1.time - timeStart1.time
    println "Total time taken to run a fixture method : " + difference
}
}

MasterSpec

package util

import com.ecw.dao.SqlTranslator
import catalog.Root
import spock.lang.Shared
import spock.lang.Specification

import javax.sql.DataSource

/**

 */
class MasterSpec extends Specification {

@Shared
Properties properties = new Properties()
@Shared
public DataSource dataSource
@Shared
protected xmlDataSource = [:]

static int timeCntr = 0;

//setup is to read xml file's content in xmlDataSource Hashmap
def setup(){

    //Get Running Class name without its package
    def className = this.class.name.substring(this.class.name.lastIndexOf('.') + 1)
    def resourceAnno = specificationContext.currentFeature.featureMethod.getAnnotation(FileResource)

    if(resourceAnno != null){
        def files = resourceAnno.xmlFiles()
        def packageName = (this.class.package.name).replaceAll('\\.','/')

        for(int i=0;i< files.length;i++){
            def f = new File("src/test/resources/"+packageName+"/"+className+"/"+files[i])
            def engine = new groovy.text.GStringTemplateEngine()
            def template = engine.createTemplate(f).make(null)
            def xmlString = template.toString()

            //load the hashmap with file name as Key and its content in form of string as Value
            xmlDataSource.put(files[i].split("\\.")[0],xmlString)
        }
    }
}

def setupSpec() {
    Date timeStart = new Date()

    File propertiesFile = new File('src/test/webapps/myApp/conf/connection.properties').withInputStream {
        properties.load it
    }

    String strDBName = getPropertyValue("myApp.DBName")
    if(strDBName.indexOf('?') > -1){
        strDBName = strDBName.substring(0, strDBName.indexOf('?'))
    }
    String strServerName = getPropertyValue("myApp.DBHost");
    if(strServerName.indexOf(':') > -1){
        strServerName = strServerName.substring(0, strServerName.indexOf(':'))
    }
    String strUrl = getPropertyValue("myApp.DBUrl")
    String strPort = strUrl.substring(strUrl.lastIndexOf(':') + 1)

    //FOR MSSQL
    System.setProperty("myApp.SkipJndi", "yes")
    //dataSource = new JtdsDataSource()
    Object newObject = null;
    if(SqlTranslator.isDbSqlServer()){
        newObject = Class.forName("net.sourceforge.jtds.jdbcx.JtdsDataSource").newInstance()
    } else if(SqlTranslator.isDbMySql()){
        newObject = Class.forName("com.mysql.jdbc.jdbc2.optional.MysqlDataSource").newInstance()
    }

    dataSource = (DataSource)newObject
    dataSource.setDatabaseName(strDBName)
    dataSource.setUser(getPropertyValue("myApp.DBUser"))
    dataSource.setPassword(getPropertyValue("myApp.DBPassword"))
    dataSource.setServerName(strServerName)
    dataSource.setPortNumber(Integer.parseInt(strPort))

}
}
Solution

  • To analyze the root cause of this issue, we have did run an errand with below scenarios :

    1) 1000 Spock test cases without any Db or mocking dependencies (PowerMock)

    Sample Code to explain the scenario:

    package mathOperations;

import groovy.lang.Closure
import mathOperations.Math
import spock.lang.Specification

class MathSpec extends Specification {
    def objMath =new Math()

    def "API to test addition of two numbers"() {
        given :"a and b"
            def a=10
            def b=5
        when: "Math.AddNumber is called with given values"
            def result =objMath.addNumber(a,b)
        then: "Result should be 15"
            result==15
    }

    def "API to test subrtaction of two numbers"() {
        given :"a and b"
            def a=10
            def b=5
        when: "Math.subtractNumber is called with given values"
            def result =objMath.subtractNumber(a, b)
        then: "Result should be 5"
            result==5
    }

    def "API to test multiplication of two numbers"() {
        given :"a and b"
            def a=10
            def b=5
        when: "Math.multiplyNumber is called with given values"
            def result =objMath.multiplyNumber(a,b)
        then: "Result should be 50"
            result==50
    }
    def "API to test division two numbers"() {
        given :"a and b"
            def a=10
            def b=5
        when: "Math.divisionNumber is called with given values"
            def result =objMath.divisionNumber(a,b)
        then: "Result should be 2"
            result==2
    }

    def "API to test whether given both numbers are equal - Affirmative"() {
        given :"a and b"
            def a=10
            def b=10
        when: "Math.equalNumber is called with given values"
            def result =objMath.equalNumber(a,b)
        then: "It should return true"
            result==true
    }

    def "API to test whether given both numbers are equal - Negative"() {
        given :"a and b"
            def a=10
            def b=11
        when: "Math.equalNumber is called with given values"
            def result =objMath.equalNumber(a,b)
        then: "It should return false"
            result==false
    }
}

    --> It took 25.153 secs including build time and below is the report below

    Spock test cases without any Db or mocking dependencies

    2) 1000 Spock test cases with mocking (PowerMock)

    Sample Code to explain the scenario:

    package mathOperations;

import groovy.lang.Closure
import mathOperations.Math
import spock.lang.Specification
import org.powermock.core.classloader.annotations.PrepareForTest
import org.powermock.modules.junit4.rule.PowerMockRule
import org.junit.Rule
import utils.QRCodeUtils
import org.powermock.api.mockito.PowerMockito
import static org.powermock.api.mockito.PowerMockito.mockStatic
import static org.mockito.BDDMockito.*

@PrepareForTest([QRCodeUtils.class])
class MathSpec extends Specification {
    def objMath =new Math()

    @Rule PowerMockRule powerMockRule = new PowerMockRule()

    def "API to test add two numbers"() {
        given :"a and b"
        def a=10
        def b=5
        when: "Math.AddNumber is call with given values"
        mockGetOTPAttemptStatus(5)
        def result =objMath.addNumber(a,b)
        then: "result should be 15"
        result==15
    }

    def "API to test subract two numbers"() {
        given :"a and b"
        def a=10
        def b=5
        when: "Math.subtractNumber is call with given values"
        mockGetOTPAttemptStatus(5)
        def result =objMath.subtractNumber(a, b)
        then: "result should be 5"
        result==5
    }
    def "API to test multiple two numbers"() {
        given :"a and b"
        def a=10
        def b=5
        when: "Math.multiplyNumber is call with given values"
        mockGetOTPAttemptStatus(5)
        def result =objMath.multiplyNumber(a,b)
        then: "result should be 50"
        result==50
    }
    def "API to test divide two numbers"() {
        given :"a and b"
        def a=10
        def b=5
        when: "Math.divisionNumber is call with given values"
        mockGetOTPAttemptStatus(5)
        def result =objMath.divisionNumber(a,b)
        then: "result should be 2"
        result==2
    }
    def "API to test modulo of a number"() {
        given :"a and b"
        def a=10
        def b=5
        when: "Math.moduloNumber is call with given values"
        mockGetOTPAttemptStatus(5)
        def result =objMath.moduloNumber(a,b)
        then: "result should be 0"
        result==0
    }

    def "API to test power of a number"() {
        given :"a and b"
        def a=10
        def b=2
        when: "Math.powerofNumber is call with given values"
        mockGetOTPAttemptStatus(5)
        def result =objMath.powerofNumber(a,b)
        then: "result should be 0"
        result==8
    }

    def "API to test numbers are equal -affirmative"() {
        given :"a and b"
        def a=10
        def b=10
        when: "Math.equalNumber is call with given values"
        mockGetOTPAttemptStatus(5)
        def result =objMath.equalNumber(a,b)
        then: "It should return true"
        result==true
    }

    def "API to test numbers are equal -negative"() {
        given :"a and b"
        def a=10
        def b=11
        when: "Math.equalNumber is call with given values"
        mockGetOTPAttemptStatus(5)
        def result =objMath.equalNumber(a,b)
        then: "It should return false"
        result==false
    }

    void mockGetOTPAttemptStatus(int status) {
        mockStatic(QRCodeUtils.class)
        when(QRCodeUtils.getOTPAttemptStatus(anyInt())).thenReturn(status)
    }
}

    --> It took 9 mins 14.222 secs including build time and below is the report

    Spock test cases with mocking

    3) 1000 Spock test cases with only Dbunit. (Usually, We insert average 15-20 table entries in a test case. Here, We added the same)

    Sample Code to explain the scenario:

    package mathOperations;

import groovy.lang.Closure
import java.sql.Statement
import mathOperations.Math
import spock.lang.Shared
import util.BaseSpec
import catalog.Root
import spock.lang.Ignore
import org.dbunit.ext.mssql.InsertIdentityOperation
import be.janbols.spock.extension.dbunit.DbUnit
import org.dbunit.IDatabaseTester

class MathSpec extends BaseSpec {

    @Shared root
    def objMath =new Math()

    @DbUnit(configure={
        IDatabaseTester it ->
        it.setUpOperation = InsertIdentityOperation.REFRESH
        it.tearDownOperation = InsertIdentityOperation.DELETE
    })
    def content =  {
        table1(id:99,MasterFile:'UnitTestFile',DataElementName:'test',DataElementDBColName:'TestDbCol',DataElementTableName:'TestTable')
        table2(id:99,MasterFile:'UnitTestFile',DataElementName:'test',DataElementDBColName:'TestDbCol',DataElementTableName:'TestTable')
        table3(id:99,Code:'T00.0',Status:'A',LongDesc:'Unit Testing')
        table4(id:99,Code:'T00.0',Status:'A',LongDesc:'Unit Testing')
        table5(id:99,Code:'T00.0',Status:'A',LongDesc:'Unit Testing')
        table6(id:99,Code:'T00.0',Status:'A',LongDesc:'Unit Testing')
        table7(id:99,Code:'T00.0',Status:'A',LongDesc:'Unit Testing')
        table8(id:99,Code:'T00.0',Status:'A',LongDesc:'Unit Testing')
        table9(id:99,Code:'T00.0',Status:'A',LongDesc:'Unit Testing')
        table10(id:99,Code:'T00.0',Status:'A',LongDesc:'Unit Testing')
    }

    def "API to test addition of two numbers"() {
        given :"a and b"
            def a=10
            def b=5
        when: "Math.AddNumber is called with given values"
            def result =objMath.addNumber(a,b)
        then: "Result should be 15"
            result==15
    }

    def "API to test subrtaction of two numbers"() {
        given :"a and b"
            def a=10
            def b=5
        when: "Math.subtractNumber is called with given values"
            def result =objMath.subtractNumber(a, b)
        then: "Result should be 5"
            result==5
    }

    def "API to test multiplication of two numbers"() {
        given :"a and b"
            def a=10
            def b=5
        when: "Math.multiplyNumber is called with given values"
            def result =objMath.multiplyNumber(a,b)
        then: "Result should be 50"
            result==50
    }
    def "API to test division two numbers"() {
        given :"a and b"
            def a=10
            def b=5
        when: "Math.divisionNumber is called with given values"
            def result =objMath.divisionNumber(a,b)
        then: "Result should be 2"
            result==2
    }

    def "API to test whether given both numbers are equal - Affirmative"() {
        given :"a and b"
            def a=10
            def b=10
        when: "Math.equalNumber is called with given values"
            def result =objMath.equalNumber(a,b)
        then: "It should return true"
            result==true
    }

    def "API to test whether given both numbers are equal - Negative"() {
        given :"a and b"
            def a=10
            def b=11
        when: "Math.equalNumber is called with given values"
            def result =objMath.equalNumber(a,b)
        then: "It should return false"
            result==false
    }
}

    --> It took 57 mins 18.136 secs including build time and below is the report Spock test cases with only Dbunit

    We have drawn a conclusion that Spock does not take time to run test cases but Power Mock which does instrumentation SO Link and DbUnit (Loads everything using reflection) were bottlenecks.

    Solution: We are going with in house framework to insert/delete database data instead of DbUnit. Also, We have replaced Power Mock with JMockit

    End Result: As I did post in the question total time from 2 Hrs and 30 Mins, has been now reduced to 6 Minutes for these 1000 test cases. :)