Search code examples
javascriptmarklogicjava

Getting XDMP-EXPNTREECACHEFULL error in MarkLogic 9

Actually i have a set of json files(i.e.Total : 1M with Size 500MB). Each json file has 18 Keys. I tried to implement Envelope pattern using below Javascript 

'use strict';
declareUpdate()
var docs = fn.collection("input");
for(var doc of docs) {
  var transformed = {};

  transformed.Metadata = { "Last Used" : ""};
  transformed.Updated = { "University" : "UCLA"}
  transformed.Source = doc; //Sending original data under Source section
  xdmp.nodeReplace(doc,transformed)
}

I tried invoking this JS.sjs using JAVA API of marklogic 9. But i encountered below error :

Exception in thread "main" com.marklogic.client.FailedRequestException: Local message: failed to apply resource at invoke: Internal Server Error. Server Message: XDMP-EXPNTREECACHEFULL: for(var doc of docs) { -- Expanded tree cache full on host localhost uri file.json-0-968991
    at com.marklogic.client.impl.OkHttpServices.checkStatus(OkHttpServices.java:4317)
    at com.marklogic.client.impl.OkHttpServices.postIteratedResourceImpl(OkHttpServices.java:3831)
    at com.marklogic.client.impl.OkHttpServices.postEvalInvoke(OkHttpServices.java:3768)
    at com.marklogic.client.impl.ServerEvaluationCallImpl.eval(ServerEvaluationCallImpl.java:164)
    at com.marklogic.client.impl.ServerEvaluationCallImpl.eval(ServerEvaluationCallImpl.java:153)
    at com.marklogic.client.impl.ServerEvaluationCallImpl.evalAs(ServerEvaluationCallImpl.java:144)
    at bulkimport.Tsm.main(Tsm.java:19)

I went through MarkLogic Documentation where they had mentioned ways to resolve this error. Following that i had increased expanded tree cache size* to 2048 but still i am facing same error.

How can i optimize by above code (i.e.JS.sjs) to avoid this error ?

Any help is appreciated.

Solution

  • You are trying to update all documents at once. That won't scale. You need to break the workload into smaller pieces somehow, and process your files in batches of roughly 500 to 1000 files.

    There are several ways of doing that, but as suggested in this email reply, DMSDK is probably one of the nicest ways of doing that at the moment: http://markmail.org/message/ua6fyaztctx6syrw.

    For a tutorial on DMSDK, see: http://developer.marklogic.com/learn/data-movement-sdk

    For the guide, see: https://docs.marklogic.com/guide/java/data-movement

    HTH!