Search code examples
karate

Karate combine retry until and compareImage

I want to test if a fetched thumbnail is correct. It might not correct the first time, so I want to retry every couple of seconds. That's why I want to combine retry until and compareImage. Is there any way to achieve this?

This is what I currently have:

@getThumnailAndCompare
Scenario: Get thumbnail and compare to reference file
  * configure imageComparison = { engine: 'ssim', failureThreshold: 0.5 }
  * configure retry = { count: 3, interval: 2000 }

  Given url 'https://127.0.0.1:8443'
  And header Accept = 'image/jpeg'
  And path 'api/thumbnail'

  * def expectedThumbnail = 'classpath:thumbnails/expected.jpg'
  And retry until responseStatus == 200 && karate.match("karate.compareImage(expectedThumbnail, response) == '#present'").pass

  When method GET

It works fine when the thumbnail matches on first try. I added the karate.match for the check to succeed. Is there a better way?

But when it first fails and then even passes on the 2nd or 3rd try, the scenario still fails due to ImageCompare throwing (see below). This is where I'm stuck.

08:26:05.519 [main] INFO  c.intuit.karate.core.FeatureRuntime - classpath:com/barco/nexxis/nmsplus/common/thumbnail.feature - call by tag at line 22: getThumbnailAndCompare
08:26:10.941 [main] ERROR com.intuit.karate - image comparison failed: latest image differed from baseline more than allowable threshold: 55.89009307671784% >= 0.5%
08:26:11.010 [main] WARN  com.intuit.karate - retry condition evaluation failed: js failed:
>>>>
01: responseStatus == 200 && karate.match("karate.compareImage(expectedThumbnail, response) == '#present'").pass
<<<<
org.graalvm.polyglot.PolyglotException: js failed:
>>>>
01: karate.compareImage(expectedThumbnail, response)
<<<<
org.graalvm.polyglot.PolyglotException: latest image differed from baseline more than allowable threshold: 55.89009307671784% >= 0.5%
- com.intuit.karate.ImageComparison.checkMismatch(ImageComparison.java:211)
- com.intuit.karate.ImageComparison.compare(ImageComparison.java:196)
- com.intuit.karate.core.ScenarioEngine.compareImageInternal(ScenarioEngine.java:1068)
- com.intuit.karate.core.ScenarioBridge.compareImage(ScenarioBridge.java:293)
- <js>.:program(Unnamed:1)

- com.intuit.karate.graal.JsEngine.fromJsEvalException(JsEngine.java:224)
- com.intuit.karate.core.ScenarioEngine.evalJs(ScenarioEngine.java:1192)
- com.intuit.karate.core.ScenarioEngine.evalKarateExpression(ScenarioEngine.java:2143)
- com.intuit.karate.core.ScenarioEngine.evalKarateExpression(ScenarioEngine.java:2062)
- com.intuit.karate.core.ScenarioEngine.match(ScenarioEngine.java:1729)
- com.intuit.karate.core.ScenarioBridge.match(ScenarioBridge.java:643)
- java.base/java.lang.invoke.SpreadHandle.invokeExact_thunkArchetype_X(SpreadHandle.java:100)
Solution

  • Wow that's a pretty impressive use of Karate. What I suggest is to use karate.get('response') which will ensure the "current" value is retrieved. I don't think you are using karate.match() or karate.compareImage() correctly, so try this:

    * def imageOk =
"""
function(expectedThumbnail) {
  var status = karate.get('responseStatus');
  if (status != 200) {
    karate.log('response is not 200, will retry');
    return false;
  }
  var bytes = karate.get('responseBytes');      
  var result = karate.compareImage(expectedThumbnail, bytes);
  karate.log('image compare result:', result);
  if (result.mismatchPercentage > 5) {
    karate.log('mismatch percentage is too high, will retry');
    return false;    
  }
  return true;
}
"""
# ...
* retry until imageOk('classpath:thumbnails/expected.jpg')

    The above code is guesswork and I have not tested it. Would be great if you confirm what karate.compareImage() returns as a JSON and whether using mismatchPercentage works as expected. Also I assumed responseBytes is a better option, but use response if that is what works for you.

    I don't think that karate.match() was going to do what you were expecting. Guess you have already seen this answer.