Initialize Grails application with 10k records

Question

I'm trying to initialize a new Grails application with 10k records for one domain class / table as an initial data set.

The challenge:

I do not want to inject them over the database as I'd like to have them run through all the Grails validators.

What I've done so far:

For the initial data set, on my dev PC, I've injected them into a database (via Talend Open Studio for Data Integration) and then generated 10k New statements like this:

New Entity(prop1:'val1',prop2:'val2');

I've tried with hundreds of them in Bootstrap, that works fine. Then, I've tried with 10k of them and the Groovy Compiler sometimes stacktraces Class file too large and sometimes it just stays quiet and stalls there.

The basic idea is to have something like

if (Entity.count == 0) {
    loadDataService.loadData()
}

I hardly want to admit, but I've also tried to create 10 of these services with 1000 records each and load them one after each other but then the memory consumption in my IDE is reaching into space and everything stalls and sometimes it comes through and says java.lang.RuntimeException: Method code too large! (64kb limit in Java Methods, which makes sense).

What else I thought about:

It seems that using Rest-API will be the only way to get the 10k records in smoothly. But for that feature I'd have to to put in the effort to develop that and before I encounter another failure, I'd take the advice from you.

Question:

How should I proceed from here? What is the best way to get the application initialized?

Answer 1

I've got it resolved, thanks to the help of @Daniel and @elixir On the way to resolution, I've encountered another problem: The runtime heavily increased record after each record so I've had to clear the hibernate session. With that, the runtime was like "100 records every 2-3 seconds", so it was linear. This article on grails.asia provided this part of the solution.

So I ended up with this code:

Bootstrap.groovy:

    if (Entity.count == 0) {
        println "No Entity found - Launch initial import"
        loadDataService.loadData()
        println "Initial import finished"
    }

and inside loadDataService:

SessionFactory sessionFactory
def loadData() {
    def inputFile = new File(grailsApplication.config.getProperty('importentitypath'))
    def InputJSON = new JsonSlurper().parseFile(inputFile, 'UTF-8')
    int mycounter = 0
    InputJSON.each{
        if (it.key=="data"){
            it.value.each {myrow ->
                mycounter+=1
                if ((mycounter % 100) == 0 ) {
                    println mycounter
                }
                Entity.withNewTransaction {
                    Entity.findByRowkey(myrow.rowkey)?:new Entity(
                            rowkey:myrow.rowkey,
                            property1: myrow.property1,
                            property2: myrow.property2,
                            ....
                    ).save(failOnError:true, flush:true)
                }
                sessionFactory.currentSession.clear()
            }
        }
    }
}

Thank you again for leading me to that solution!