After updating Infinispan from version 9.4.11.Final to 10.0.1.Final the starting of multiple caches produces this error when trying to start the fourth cache. With the previous version it runs smoothly.
Caused by: java.lang.RuntimeException: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.persistence.manager.PersistenceManager
at com.metaproc.core.datagrid.impl.InfinispanDatagridServiceImpl.<init>(InfinispanDatagridServiceImpl.java:139) ~[?:?]
... 71 more
Caused by: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.persistence.manager.PersistenceManager
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:555) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:605) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:569) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:605) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:569) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:605) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:569) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:605) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:569) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:605) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:569) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:561) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:605) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:569) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:344) ~[?:?]
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:240) ~[?:?]
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:207) ~[?:?]
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1091) ~[?:?]
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:513) ~[?:?]
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:693) ~[?:?]
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:632) ~[?:?]
at org.infinispan.manager.DefaultCacheManager.startCache(DefaultCacheManager.java:1024) ~[?:?]
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[?:?]
at java.lang.Thread.start(Thread.java:717) ~[?:?]
at org.infinispan.persistence.sifs.Index.start(Index.java:75) ~[?:?]
at org.infinispan.persistence.sifs.SoftIndexFileStore.startIndex(SoftIndexFileStore.java:242) ~[?:?]
at org.infinispan.persistence.sifs.SoftIndexFileStore.start(SoftIndexFileStore.java:161) ~[?:?]
at org.infinispan.persistence.support.ComposedSegmentedLoadWriteStore.startNewStoreForSegment(ComposedSegmentedLoadWriteStore.java:258) ~[?:?]
at org.infinispan.persistence.support.ComposedSegmentedLoadWriteStore.start(ComposedSegmentedLoadWriteStore.java:244) ~[?:?]
at org.infinispan.persistence.manager.PersistenceManagerImpl.lambda$startWriter$42(PersistenceManagerImpl.java:1304) ~[?:?]
at org.infinispan.persistence.manager.PersistenceManagerImpl.startStore(PersistenceManagerImpl.java:1333) ~[?:?]
at org.infinispan.persistence.manager.PersistenceManagerImpl.startWriter(PersistenceManagerImpl.java:1296) ~[?:?]
at org.infinispan.persistence.manager.PersistenceManagerImpl.lambda$start$0(PersistenceManagerImpl.java:193) ~[?:?]
at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:?]
at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:193) ~[?:?]
at org.infinispan.persistence.support.DelegatingPersistenceManager.start(DelegatingPersistenceManager.java:39) ~[?:?]
at org.infinispan.persistence.support.CorePackageImpl$1.start(CorePackageImpl.java:30) ~[?:?]
at org.infinispan.persistence.support.CorePackageImpl$1.start(CorePackageImpl.java:24) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:587) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:584) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:578) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:605) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:569) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:605) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:569) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:605) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:569) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:605) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:569) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:605) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:569) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:561) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:605) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:569) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) ~[?:?]
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770) ~[?:?]
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:344) ~[?:?]
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:240) ~[?:?]
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:207) ~[?:?]
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1091) ~[?:?]
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:513) ~[?:?]
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:693) ~[?:?]
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:632) ~[?:?]
at org.infinispan.manager.DefaultCacheManager.startCache(DefaultCacheManager.java:1024) ~[?:?]
We are using this configuration:
final SoftIndexFileStoreConfigurationBuilder persistentStoreConfiguration =
new SoftIndexFileStoreConfigurationBuilder(configurationBuilder)
.indexLocation(indexPath)
.dataLocation(dataPath)
.maxNodeSize(8192)
.fetchPersistentState(true);
new ConfigurationBuilder()
.locking()
.isolationLevel(IsolationLevel.REPEATABLE_READ)
.lockAcquisitionTimeout(30000)
.useLockStriping(false)
.transaction()
.autoCommit(true)
.completedTxTimeout(60000)
.lockingMode(LockingMode.OPTIMISTIC)
.transactionMode(TransactionMode.TRANSACTIONAL)
.notifications(true)
.reaperWakeUpInterval(30000)
.cacheStopTimeout(30000)
.transactionManagerLookup(new GenericTransactionManagerLookup())
.clustering()
.cacheMode(CacheMode.DIST_SYNC)
.stateTransfer()
.fetchInMemoryState(true)
.hash().numOwners(2)
.l1()
.lifespan(-1)
.memory()
.size(1)
.persistence()
.passivation(true)
.addStore(persistentStoreConfiguration).build();
It looks like something of the configuration of the soft index file store is not correct anymore, I just cannot figure out what it might be. I tried to decrease the maxNodeSize, without any success. Can someone point me to the part where I need to adjust the configurations after the upgrade or is there anything else I'm missing?
I am pretty certain this is caused by the SoftIndexFileStore being segmented and creating a bunch of extra threads. There should have been a WARN message logged to your console output stating something to that extent.
You can also read about this at Upgrading Guide.
You can explicitly define the store to not be segmented by setting the segmented="false"
attribute on your xml configuration. Since you are using programmatic configuration it would just be adding .segmented(false)
to your code. Source Reference