ConcurrentModificationException on a object wrapped with lock

Question

I've left my application up for a night and noticed such a stacktrace in the morning:

java.util.ConcurrentModificationException: null
at java.base/java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:756) ~[na:na]
    at java.base/java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:788) ~[na:na]
    at java.base/java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:786) ~[na:na]
    at ****.MyClass.cleanup(MyClass.kt:150) ~[main/:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-6.0.12.jar:6.0.12]
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-6.0.12.jar:6.0.12]
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:96) ~[spring-context-6.0.12.jar:6.0.12]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

I tried to find a root cause but I was not able to. So let me provide simplified version of my code:

@Configuration
@EnableConfigurationProperties(MyConnectionProperties::class)
class MyClass(
    private val myProperties: MyConnectionProperties,
) {

    private val logger: KLogger = KotlinLogging.logger {}
    private val myLock = ReentrantLock()
    private val myHolder: MutableMap<String, MyConnectionExtension> = mutableMapOf()
     
    @Scheduled(cron = "...")
    fun cleanup() {
        val connectionsToClose: MutableList<LDAPConnection> = mutableListOf()

        myLock.withLock {
            myHolder.forEach { (userDn, connectionExtension) ->
                val lastTouchTime = max(
                    connectionExtension.connection.lastCommunicationTime,
                    connectionExtension.lastPullingTime,
                )
                val connectionName = connectionExtension.connection.connectionName

                

                if (lastTouchTime + ldapConnectionProperties.lifetimeMs < Instant.now().toEpochMilli()) {
                    myHolder.remove(userDn)
                    connectionsToClose.add(connectionExtension.connection)

                   
                }
            }
        }

        connectionsToClose.forEach { connection ->
            try {
                connection.close()               
            } catch (e: Exception) {
                logger.warn(e) {"....."}
            }
        }

        logger.trace { "job finished" }
    }
    
    private fun getConnection(userId: String, password: String): LDAPConnection {
        myHolder.withLock {
            val connectionExtension = connectionHolder[userId]
            val connection = connectionExtension?.connection

            return if (connection == null || !connection.isConnected) {
                createConnection(userId, password).also {
                    connectionHolder[userId] = LdapConnectionExtension(
                        connection = it,
                        lastPullingTime = Instant.now().toEpochMilli(),
                    )
                }
            } else {                
                connectionExtension.lastPullingTime = Instant.now().toEpochMilli()
                connection
            }
        }
    }
    ...
}

There are only 2 functions which use myHolder so I provided only them. From my point of view access to myHolder is covered with lock so no concurrent issue is expected even although map is not concurrent here.

What is wrong here ?

Answer 1

As the comments said, you are removeing from a map, while iterating over it with forEach. forEach spins up an iterator, if you change the map while iterating over it, a ConcurrentModificationException is thrown.

Solution would be to record all elements you want to remove in a separate collection, then go over those after the forEach iteration and remove them from the map.