I'm running a web server with jetty 9.4.7.v20170914 and connecting to a database using Apache Derby 10.14.1.0. Most of my transactions work fine, but when the database is during heavy load the queries start to fail, hanging up the database until they are successfully completed. I'm running a script that crawls a website and batch inserts thousands of objects in the database and this issue is causing it to take a huge amount of time to complete.
Unit tests work fine for every database operation and I've checked my EntityManager usage to assure that I'm correctly opening, commiting, and closing the transactions. I've tried other versions of Eclipselink, Jetty and Derby, but it didn't solve the issue.
The problem occurs in different queries but I believe it always happens in SELECT statements. Please note that I'm checking if objects exists before creating them, but they are two separate transactions.
[EL Info]: query: 2018-06-13 16:03:48.344--UnitOfWork(1800307564)--Communication failure detected when attempting to perform read query outside of a transaction. Attempting to retry query. Error was: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.5.v20170607-b3d05bd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLNonTransientConnectionException: java.net.BindException : Error connecting to server localhost on port 1,527 with message Address already in use: connect.
Error Code: 40000
Call: SELECT t1.ID, t1.PARENTADDITIVECLASSID FROM ADDITIVE t0, ADDITIVE_ADDITIVECLASS t2, ADDITIVECLASS t1 WHERE ((t0.ID = ?) AND ((t2.additiveClasses_ID = t1.ID) AND (t0.ID = t2.additives_ID)))
bind => [1 parameter bound]
Query: ReadAllQuery(name="entity.Additive.GET_ADDITIVECLASSES" referenceClass=AdditiveClass sql="SELECT t1.ID, t1.PARENTADDITIVECLASSID FROM ADDITIVE t0, ADDITIVE_ADDITIVECLASS t2, ADDITIVECLASS t1 WHERE ((t0.ID = ?) AND ((t2.additiveClasses_ID = t1.ID) AND (t0.ID = t2.additives_ID)))").
I also get the following exceptions, although not nearly as frequently.
and
The problem is gone after migrating the database to a native Ubuntu 18.04 installation (running the same image on VMware Workstation 14 didn't solve the issue). As @Chris pointed out in the comments, this is likely a problem with how Windows allocates outbound TCP ports. I'm okay with using Linux for this, but if anyone runs into these issues on Windows be sure to check this guide. Please note that I haven't tested those steps myself.