Search code examples
parsinggroovyclasspathkatalon-studio

Parsing Groovy script from a restrictive project (tied to a framework/IDE)

Problem

I am in the process of porting a project from a Katalon Studio project (closed system), to a more standard Groovy project. Currently, I'm trying to build an automated tool that will parse my scripts and classes to provide changes to my structure and syntax. I'm having a lot of difficulty parsing the groovy scripts due to the fact that the files are not buildable from outside of Katalon Studio (if I import the project into something like IntelliJ IDEA, I can't build).

Objective

The goal is to be able to parse the scripts somehow without the need for a custom parser (Groovy would make this pretty difficult, as it's syntactically lax).

Summary of the Katalon Studio Project

It's set up like an Eclipse project. I have pasted the classpath below. The .class files are located in bin/keyword.

<?xml version="1.0" encoding="UTF-8"?>
<classpath>
    <classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
    <classpathentry kind="src" output="bin/keyword" path="Keywords"/>
    <classpathentry kind="src" output="bin/listener" path="Test Listeners"/>
    <classpathentry kind="src" output="bin/lib" path="Libs"/>
    <classpathentry kind="src" output="bin/groovy" path="Include/scripts/groovy"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.kms.katalon.core_1.0.0.202004070746.jar">
        <attributes>
            <attribute name="javadoc_location" value="file:/C:/Program%20Files/Katalon%20Studio/configuration/resources/apidocs/com.kms.katalon.core/"/>
        </attributes>
    </classpathentry>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/aalto-xml-1.0.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/aspectjweaver-1.9.1.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/bcpkix-jdk15on-1.58.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/bcprov-jdk15on-1.58.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/browsermob-core-2.1.5.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/cglib-3.2.10.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/commons-codec-1.10.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/commons-collections4-4.1.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/commons-validator-1.6.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/curvesapi-1.04.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/dnsjava-2.1.8.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/java-client-7.0.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/jna-4.1.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/jna-platform-4.1.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/jzlib-1.1.3.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/kat-server.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/littleproxy-1.1.0-beta-bmp-17.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/mitm-2.1.5.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/netty-all-4.0.51.Final.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/poi-3.17.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/poi-ooxml-3.17.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/poi-ooxml-schemas-3.17.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/proxy-vole-1.0.4-SNAPSHOT-jar-with-dependencies.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/repackaged-sikuli-api-1.0.2-standalone.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/selenium-leg-rc-3.141.59.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/selenium-server-standalone-3.141.59.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/slf4j-api-1.7.25.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/soap-builder-1.0.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/soap-common-1.0.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/soap-legacy-1.0.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/spring-context-5.1.0.RELEASE.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/stax-api-1.0.1.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/stax2-api-4.0.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/testng-6.11.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/configuration/resources/lib/xmlbeans-2.6.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/net.sf.supercsv.super-csv_2.1.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.dom4j_1.6.1.v201312101416.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.apache.commons.io_2.4.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.apache.commons.lang_2.6.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.apache.xmlbeans_2.6.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.google.gson_2.8.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.jsoup_1.7.2.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.microsoft.sqlserver.sqljdbc4_4.1.5605.100.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.apache.commons.dbutils_1.6.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.oracle.jdbc.ojdbc7_12.1.0.2.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.postgresql.jdbc41_9.4.1208.jre7.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.eclipse.osgi_3.11.3.v20170209-1843.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.kms.katalon.constant_1.0.0.202004070746.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.eclipse.persistence.eclipselink_2.5.0.RC1.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.kms.katalon.selenium-standalone_1.0.0.202004070746.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/atu.testrecorder.screen-recorder_2.1.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.google.guava_23.0.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.kms.katalon.util_1.0.0.202004070746.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.apache.httpcomponents.httpclient_4.5.1.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.apache.httpcomponents.httpcore_4.4.4.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.apache.httpcomponents.httpmime_4.5.1.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.apache.commons.lang3_3.6.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.greenrobot.eventbus_3.1.1.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.assertj.core_3.10.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.junit_4.12.0.v201504281640/junit.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.fasterxml.jackson.core.jackson-databind_2.9.1.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.fasterxml.jackson.core.jackson-core_2.9.6.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.fasterxml.jackson.core.jackson-annotations_2.9.1.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/ch.qos.logback.classic_1.2.3.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/ch.qos.logback.core_1.2.3.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/slf4j.api_1.7.25.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.fusesource.jansi_1.17.1.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.kms.katalon.netlightbody_1.0.0.202004070746.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.kms.katalon.poi_1.0.0.202004070746.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/xml-apis_1.4.1.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.apache.commons.commons-text_1.6.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.mysql.cj_8.0.17.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.kms.katalon.proxyvole_1.0.0.202004070746.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.apache.commons.codec_1.7.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.apache.commons.logging_1.2.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.hamcrest.core_1.3.0.v201303031735.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.beanshell.bsh_2.0.0.0b4.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.apache.ant.launcher_1.9.6.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.kms.katalon.core.appium_1.0.0.202004070746.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.eclipse.equinox.common_3.8.0.v20160509-1230.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.kms.katalon.core.webui_1.0.0.202004070746.jar">
        <attributes>
            <attribute name="javadoc_location" value="file:/C:/Program%20Files/Katalon%20Studio/configuration/resources/apidocs/com.kms.katalon.core.webui/"/>
        </attributes>
    </classpathentry>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.json_20140107.0.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.kms.katalon.core.mobile_1.0.0.202004070746.jar">
        <attributes>
            <attribute name="javadoc_location" value="file:/C:/Program%20Files/Katalon%20Studio/configuration/resources/apidocs/com.kms.katalon.core.mobile/"/>
        </attributes>
    </classpathentry>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/org.apache.commons.validator_1.5.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.kms.katalon.core.cucumber_1.0.0.202004070746.jar">
        <attributes>
            <attribute name="javadoc_location" value="file:/C:/Program%20Files/Katalon%20Studio/configuration/resources/apidocs/com.kms.katalon.core.cucumber/"/>
        </attributes>
    </classpathentry>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/io.cucumber.core_3.0.2.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/io.cucumber.datatable_1.0.3.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/io.cucumber.datatable-dependencies_1.0.3.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/io.cucumber.expressions_5.0.19.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/io.cucumber.gherkin_5.0.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/io.cucumber.html_0.2.7.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/io.cucumber.java_3.0.2.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/io.cucumber.tag-expressions_1.1.1.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/io.cucumber.junit_3.0.2.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.kms.katalon.core.webservice_1.0.0.202004070746.jar">
        <attributes>
            <attribute name="javadoc_location" value="file:/C:/Program%20Files/Katalon%20Studio/configuration/resources/apidocs/com.kms.katalon.core.webservice/"/>
        </attributes>
    </classpathentry>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/wsdl4j_1.6.2.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.google.oauth-client.google-oauth-client_1.22.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.google.http-client.google-http-client_1.22.0.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/ca.juliusdavies.not-yet-commons-ssl_0.3.11.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.kms.katalon.preferences_1.0.0.202004070746.jar"/>
    <classpathentry kind="lib" path="C:/Program Files/Katalon Studio/plugins/com.kms.katalon.core.windows_1.0.0.202004070746.jar">
        <attributes>
            <attribute name="javadoc_location" value="file:/C:/Program%20Files/Katalon%20Studio/configuration/resources/apidocs/com.kms.katalon.core.windows/"/>
        </attributes>
    </classpathentry>
    <classpathentry kind="lib" path="C:/Users/myUsername/git/myProj/Drivers/ashot-1.5.4.jar"/>
    <classpathentry kind="lib" path="C:/Users/myUsername/git/myProj/Drivers/javax.mail.jar"/>
    <classpathentry kind="lib" path="C:/Users/myUsername/git/myProj/Drivers/jcommon-1.0.23.jar"/>
    <classpathentry kind="lib" path="C:/Users/myUsername/git/myProj/Drivers/jfreechart-1.5.0.jar"/>
    <classpathentry exported="true" kind="con" path="GROOVY_SUPPORT"/>
    <classpathentry kind="output" path="bin"/>
</classpath>

(Failed) Attempts

I have tried using GroovyDocTool, GroovyClassLoader, GroovyShell by experimenting with the source code, but admittedly I have no idea what I'm doing, and there is very little documentation to go off of. What I ended up trying was a compilation of other answers on this site, and may not be correct.

What I tried (may be wrong):

Note - in Katalon, classes are referred to as "Keywords"

String[] sources =
                [
                        '/git/myProj/Keywords',
                        '/git/myProj/Scripts',
                ]
GroovyDocTool docTool = new GroovyDocTool(sources)
GroovyRootDoc rootDoc = docTool.getRootDoc()
println rootDoc.specifiedPackages()
println rootDoc.specifiedClasses()
for (GroovyPackageDoc p in rootDoc.classes()) {
    // This loop is never entered
    GroovyClassDoc[] allClasses = p.allClasses()
    for (GroovyClassDoc c in allClasses) {
        log(c.getFullPathName() + '\n')
    }
}

Output (all empty):

refactoringtool.script.ScriptParser.par  []
refactoringtool.script.ScriptParser.par  null

Attempts with GroovyClassLoader and GroovyShell are below. They run, but they can't get past the import statements.

File classDir = new File(baseDir + '/git/myProj/.classPath')
String classPath = classDir.toString()
println "classpath: ${classPath}"

File binDir = new File(baseDir + '/git/myProj/bin')
String binPath = binDir.toString()
println "binDir: ${binPath}"

File keywordDir = new File(baseDir + '/git/myProj/bin/keyword')
String keywordPath = keywordDir.toString()
println "keywordDir: ${keywordPath}"

GroovyClassLoader classLoader = new GroovyClassLoader()
classLoader.addClasspath(classPath)
classLoader.addClasspath(binPath)
println "Added classpath:\n${classLoader.getClassPath()}"
println "Loaded classes:\n${classLoader.getLoadedClasses()*.getName()}"
GroovyShell shell = new GroovyShell(classLoader)
Script script = shell.parse(fileText)

Exception:

org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Script1.groovy: 2: unable to resolve class com.myClass
 @ line 2, column 1.
   import com.myClass
   ^
Solution

  • Solution

    All listed attempts failed, but ended up building a script with a raw file parser that mapped out new->old syntax conversions and created new files with the converted content. Can be slow even when multithreaded, but it worked!