I am developing an app on WildFly 10 running on OpenShift, utilizing Nashorn to do server-side rendering of a React app.
The app runs ok on my local machine (i don't like perf levels), but when I deploy it to OpenShift, something mysterious happens.
After a couple of requests, code that ran find on previous requests suddenly starts throwing
java.lang.ClassCastException: jdk.nashorn.internal.objects.NativeRegExpExecResult cannot be cast to jdk.nashorn.internal.objects.NativeArray
The stack trace reveals this coming from a line of code in React-Router... I added a couple of lines of logging to react-router manually and the strange thing is that the parameters that react-router is working on in the failing scenario. This is what the altered code looks like:
if (match != null) {
if (captureRemaining) {if (typeof global != 'undefined') {global.log.warn('match.length=' + match.length);}
remainingPathname = match.pop();
var matchedPath = match[0].substr(0, match[0].length - remainingPathname.length);
if (typeof global != 'undefined') {global.log.warn('remainingPathname=' + remainingPathname + ', matchedPath=' + matchedPath);}
// If we didn't match the entire pathname, then make sure that the match
// we did get ends at a path separator (potentially the one we added
// above at the beginning of the path, if the actual match was empty).
(note the calls to global.log.warn... I added those)
If you look at the full logs you can see that for the first requests, things seem to be working fine, but then, suddenly, it starts throwing this ClassCastException and won't stop anymore. All my app does is return 503 service not available for any request.
I messed around with the code, rewriting it multiple times to get correct behavior but I'm kinda stuck. In the end I stuck in a synchronized block to try to eliminate threading issues but the issue remains. Weird thing is, that if I set max-worker-threads to 1 in WildFly, the issue seems to disappear.
I say seems to because i find it very hard to pin down the problem, what with OpenShift's long deployment times and the 'random' behavior of the issue.
Below is the relevant code for my ReactRenderFilter. Full code on pastebin.
public class ReactRenderFilter implements Filter {
private static final Object LOCK = new Object();
private static final ScriptEngine engine = new ScriptEngineManager().getEngineByMimeType("text/javascript");
private static final List<CompiledScript> scripts = new ArrayList<>();
private static final ThreadLocal<RenderEngine> renderEngine = new ThreadLocal<>();
private FilterConfig config;
private String bundlePath;
private String jspPath;
public static class RenderEngine {
private final ScriptContext context;
private final ReactRenderer renderer;
private final long lastModified;
public RenderEngine(File bundle) throws ScriptException, UnsupportedEncodingException, FileNotFoundException {
context = new SimpleScriptContext();
Bindings global = engine.createBindings();
context.setBindings(global, ScriptContext.ENGINE_SCOPE);
global.put("global", global);
for (CompiledScript script : scripts) {
script.eval(context);
}
engine.eval(new InputStreamReader(new FileInputStream(bundle), "utf-8"), context);
lastModified = bundle.lastModified();
LOG.finer("Getting renderer");
renderer = ((ScriptObjectMirror) engine.eval("global.render", context)).to(ReactRenderer.class);
}
String render(String path, String initialDataJSON) {
return renderer.render(path, initialDataJSON);
}
boolean isOutdated(File bundle) {
return lastModified != bundle.lastModified();
}
}
public ReactRenderFilter() {super();}
@Override public void destroy() {}
@Override public void init(FilterConfig filterConfig) throws ServletException {
config = filterConfig;
try {
String[] paths = ...
for (String path : paths) {
if (path.trim().isEmpty()) {continue;}
File file = new File(config.getServletContext().getRealPath(path.trim()));
scripts.add(((Compilable) engine).compile(new InputStreamReader(new FileInputStream(file), "utf-8")));
}
bundlePath = config.getServletContext().getRealPath(config.getInitParameter(PARAM_APP_BUNDLE_PATH).trim());
jspPath = config.getInitParameter(PARAM_MARKUP_JSP_PATH).trim();
} catch (UnsupportedEncodingException | FileNotFoundException | ScriptException e) {
throw new ServletException("Unable to initialize ReactRenderServlet.", e);
}
}
@Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
File bundle = new File(bundlePath);
HttpServletRequest req = (HttpServletRequest) request;
HttpServletResponse res = (HttpServletResponse) response;
String path = req.getRequestURI().substring(req.getContextPath().length());
String initialDataJSON = "{}";
@SuppressWarnings("unchecked")
Map<String, Object> initialData = (Map<String, Object>) req.getAttribute("initialData");
if (initialData != null) {
ObjectMapper mapper = new ObjectMapper();
initialDataJSON = mapper.writeValueAsString(initialData);
req.setAttribute("initialDataJSON", initialDataJSON);
}
String renderResult = null;
try {
if (renderEngine.get() == null || renderEngine.get().isOutdated(bundle)) {
// prevent multiple render engines to be instantiated simultaneously
synchronized (LOCK) {
renderEngine.set(new RenderEngine(bundle));
}
}
// I sure hope there is a way around this... locking on central object
// during rendering can't be good for performance... But it beats having
// only one worker thread
synchronized (LOCK) {
renderResult = renderEngine.get().render(path, initialDataJSON);
}
if (renderResult.startsWith(MARKUP)) {
String markup = renderResult.substring(MARKUP.length());
req.setAttribute("markup", markup);
int maxAge = 60 * 60; // 60 minutes
res.addHeader("Cache-Control", "public, max-age=" + maxAge);
res.addDateHeader("Expires", new Date().getTime() + maxAge);
req.getRequestDispatcher(jspPath).forward(request, response);
}
else if (renderResult.startsWith(REDIRECT)) {
String url = renderResult.substring(REDIRECT.length());
res.sendRedirect(url);
}
else if (renderResult.startsWith(NOTFOUND)) {
int maxAge = 365 * 24 * 60 * 60; // 365 days
res.addHeader("Cache-Control", "public, max-age=" + maxAge);
res.addDateHeader("Expires", new Date().getTime() + maxAge);
chain.doFilter(request, response);
}
else {
String msg = renderResult.substring(ERROR.length());
throw new ServletException("Unable to generate response for route [" + path + "]: " + msg);
}
} catch (ScriptException e) {
throw new ServletException(e);
}
}
}
As you can see, I have one static ScriptEngine and a separate ScriptContext + Bindings for each thread (in a ThreadLocal)... I thought (based on docs I found) that this should be thread-safe... In desperation I added a LOCK and synchronized blocks on this lock, but it doesn't seem to help.
I am not sure it is even related to threading, but it sure does so.
Does the above code look like the right way to create multiple script contexts to be used concurrently?
Any tips to get rid of this issue, or even to debug it?
This issue appears similar to https://bugs.openjdk.java.net/browse/JDK-8145550 which has been fixed in jdk9 and backported to jdk8u-dev ( http://hg.openjdk.java.net/jdk8u/jdk8u-dev/nashorn/rev/fa7dce1af94e ). It'd be great If you can pull http://hg.openjdk.java.net/jdk8u/jdk8u-dev/nashorn and build nashorn.jar to test against your app. Thanks.