Search code examples
jmeterfunctional-testinge2e-testingbeanshell

Using jMeter for functional e2e testing

We're using JMeter for e2e testing of a GraphQL API and have some problems that only the GUI shows us the test results, when running from the console there is no useful "test summary" visible, and jMeter always exits with a "0" exit code indicating "success" to our CI.

I've reproduced a minimal example included below to indicate what I'm trying to do:

  • use an HTTP sampler to make a request to a JSON API
  • make a "JSON Assertion" on the result
  • Print results of failed samplers or assertions to a file or console with a bean shell listener

My goal is to print something like the "View Results Tree" as a passed/failed summary to the console where jMeter was run, or write it to a file.

Our tests make use of simple HTTP samplers with JSON Assertions, my goal was to try and use a Bean Shell Listener and check if any samplers or JSON assertions had errors, and write to a file in either case.

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="4.0" jmeter="4.0 r1823414">
<hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true">
    <stringProp name="TestPlan.comments"></stringProp>
    <boolProp name="TestPlan.functional_mode">false</boolProp>
    <boolProp name="TestPlan.tearDown_on_shutdown">true</boolProp>
    <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
    <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
        <collectionProp name="Arguments.arguments"/>
    </elementProp>
    <stringProp name="TestPlan.user_define_classpath"></stringProp>
    </TestPlan>
    <hashTree>
    <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
        <boolProp name="LoopController.continue_forever">false</boolProp>
        <stringProp name="LoopController.loops">1</stringProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">1</stringProp>
        <stringProp name="ThreadGroup.ramp_time">1</stringProp>
        <boolProp name="ThreadGroup.scheduler">false</boolProp>
        <stringProp name="ThreadGroup.duration"></stringProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
    </ThreadGroup>
    <hashTree>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="HTTP Request" enabled="true">
        <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
            <collectionProp name="Arguments.arguments"/>
        </elementProp>
        <stringProp name="HTTPSampler.domain">jsonplaceholder.typicode.com</stringProp>
        <stringProp name="HTTPSampler.port"></stringProp>
        <stringProp name="HTTPSampler.protocol">https</stringProp>
        <stringProp name="HTTPSampler.contentEncoding"></stringProp>
        <stringProp name="HTTPSampler.path">/posts/1</stringProp>
        <stringProp name="HTTPSampler.method">GET</stringProp>
        <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
        <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
        <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
        <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
        <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
        <stringProp name="HTTPSampler.connect_timeout"></stringProp>
        <stringProp name="HTTPSampler.response_timeout"></stringProp>
        </HTTPSamplerProxy>
        <hashTree/>
        <JSONPathAssertion guiclass="JSONPathAssertionGui" testclass="JSONPathAssertion" testname="JSON Assertion" enabled="true">
        <stringProp name="JSON_PATH">$.userID</stringProp>
        <stringProp name="EXPECTED_VALUE"></stringProp>
        <boolProp name="JSONVALIDATION">false</boolProp>
        <boolProp name="EXPECT_NULL">false</boolProp>
        <boolProp name="INVERT">false</boolProp>
        <boolProp name="ISREGEX">true</boolProp>
        </JSONPathAssertion>
        <hashTree/>
    </hashTree>
    <BeanShellListener guiclass="TestBeanGUI" testclass="BeanShellListener" testname="BeanShell Listener" enabled="true">
        <boolProp name="resetInterpreter">false</boolProp>
        <stringProp name="parameters"></stringProp>
        <stringProp name="filename"></stringProp>
        <stringProp name="script"></stringProp>
    </BeanShellListener>
    <hashTree/>
    </hashTree>
</hashTree>
</jmeterTestPlan>

I tried a Bean Shell Listener with the following contents, but I don't find the JSON Assertion results in the variables/context available to the Bean Shell Listener.

import org.apache.jmeter.services.FileServer;

// Open File(s)
f = new FileOutputStream(FileServer.getFileServer().getBaseDir()+"/results.txt", true); 
p = new PrintStream(f); 

// Write data to file 
p.println( "sampleResult " + sampleResult.toString() + ": " + sampleResult.isSuccessful() );
p.println( "sampleEvent  " + sampleEvent.toString());
p.println( "prev  " + prev.toString());
p.println( "props  " + props.toString());
p.println( "ctx  " + ctx.toString());
p.println( "vars  " + vars.toString());

// Close File(s)
p.close();f.close();

We are not a Java shop, we use GitLab/etc for our CI, and the ideal case would be to be able to run our jMeter tests in the console, and see a summary of output as one would see when doing unit testing of a scripting language or similar, with a red/green summary of passed and failed tests.

Solution
    1. Be aware that as per JMeter 3.1 it is recommended to use JSR223 Test Elements for any form of scripting. So I would suggest switching to JSR223 Listener instead.
    2. You can access AssertionResults as prev.getAssertionResults()

    3. Groovy SDK allows working with files much easier, so you could use something like:

      def results = new File('results.txt')
def newLine = System.getProperty('line.separator')
results << 'Sampler Name: ' + prev.getSampleLabel() + ' Successful: ' + prev.isSuccessful() 
if (!prev.isSuccessful()) {
    prev.getAssertionResults().each {assertionResult ->
        results << ' Error message: ' << assertionResult.failureMessage << newLine
    }
}

      in order to get output in form of:

      Sampler Name: HTTP Request Successful: false Error message: No results for path: $['userID']

      See Apache Groovy - Why and How You Should Use It article for more information on Groovy scripting in JMeter.

    You can also consider running your test using Taurus tool which provides real-time console reporting and powerful Pass/Fail Criteria subsystem where you can specify thresholds and actions to be taken when they are hit/exceeded. So you can configure Taurus to exit with non-zero code in case of failures which can be used for scripting or continuous integration.