I am developing Grails application with aim of Grails Mongo plugin. When I run application locally with run-app command everything works good and application opens 2 connections to Mongo instance.
But when I package application into war file and deploy it to remote Tomcat server the startup application behavior becomes pretty strange.
Here is the startup log:
??? 12, 2015 1:15:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/myapp] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
??? 12, 2015 1:15:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/myapp] appears to have started a thread named [cluster-1-myapp.server:27017] but has failed to stop it. This is very likely to create a memory leak.
??? 12, 2015 1:15:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/myapp] appears to have started a thread named [MongoCleaner1332131781] but has failed to stop it. This is very likely to create a memory leak.
??? 12, 2015 1:15:59 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deployment of web application archive /data/tomcat3/webapps/myapp.war has finished in 40,759 ms
??? 12, 2015 1:16:04 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load org.bson.io.PoolOutputBuffer. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1612)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
at org.bson.BasicBSONDecoder.<init>(BasicBSONDecoder.java:599)
at com.mongodb.DefaultDBDecoder.<init>(DefaultDBDecoder.java:44)
at com.mongodb.DefaultDBDecoder$DefaultFactory.create(DefaultDBDecoder.java:33)
at com.mongodb.DBPort.<init>(DBPort.java:90)
at com.mongodb.ServerMonitor$ServerMonitorRunnable.run(ServerMonitor.java:87)
at java.lang.Thread.run(Thread.java:745)
??? 12, 2015 1:16:29 PM org.apache.catalina.core.StandardContext startInternal
SEVERE: Error listenerStart
??? 12, 2015 1:16:29 PM org.apache.catalina.core.StandardContext startInternal
SEVERE: Context [/app] startup failed due to previous errors
So during startup application tries to obtain MongoConnection in separate Java thread and fails but doesn't close this thread.
So every ~5 seconds Application opens new connections and MongoDB Server expends all available connections:
> db.serverStatus().connections
{ "current" : 6, "available" : 813, "totalCreated" : NumberLong(800) }
> db.serverStatus().connections
{ "current" : 45, "available" : 774, "totalCreated" : NumberLong(839) }
Mongo DB Server is available from that Tomcat server instance and I can connect to it and retrieve data.
Also stacktrace.log contains next error:
Caused by: com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting for a server that matches AnyServerSelector{}. Client view of cluster state is {type=Unknown, servers=[{address=mongodb.host:27017, type=Unknown, state=Connecting}]
at com.mongodb.BaseCluster.getServer(BaseCluster.java:82)
at com.mongodb.DBTCPConnector.getServer(DBTCPConnector.java:650)
at com.mongodb.DBTCPConnector.access$300(DBTCPConnector.java:39)
at com.mongodb.DBTCPConnector$MyPort.getConnection(DBTCPConnector.java:499)
at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:447)
at com.mongodb.DBTCPConnector.authenticate(DBTCPConnector.java:620)
at com.mongodb.DBApiLayer.doAuthenticate(DBApiLayer.java:195)
at com.mongodb.DB.authenticateCommandHelper(DB.java:765)
at com.mongodb.DB.authenticate(DB.java:721)
at org.grails.datastore.gorm.mongo.bean.factory.GMongoFactoryBean.afterPropertiesSet(GMongoFactoryBean.java:149)
... 5 more
Anybody faced the same issue?
Thank you!
The issue was the incorrect setup of MongoDB driver connection. Our MongoDB server is installed on the same instance with Tomcat container. But we use full URL to connect to MongoDB, e.g. app.server.com:27017. Driver was not able to establish connection and Tomcat could not kill the connection thread pool.
As a solution of this issue you 2 ways: