Good day,
Currently I have a web application deploy Jboss Server 7.1.4.GA. And I have a db2 database server with a database name mydb, and ip address 10.8.12.167. Db2 version is 10.8.5. And I create the mydb database by running initial script, for example, create table, insert pre-load data and so on.
In Jboss Console, I configure the server to point to mydb, and the web application is working fine.
Now there is a new database server, ip address 10.8.112.63. I install same version of db2 database to it. The different is, I didnt run those initial script, I just do db backup from 10.8.12.167, and restore in here.
In Jboss console, I change the database ip to 10.8.112.63, when test connection, its successful. However, when I want to enable my server group's deployment, it will run until some point and hit timeout. The following is my server.log from my Jboss server:
18:33:29,938 INFO [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] (ServerService Thread Pool -- 87) Building JPA container EntityManagerFactory for persistence unit 'IBSJPA'
2020-03-18 18:38:23,332 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 25) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'deploy' at address '[("deployment" => "IBSApiEAR.ear")]'
18:38:23,332 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 25) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'deploy' at address '[("deployment" => "IBSApiEAR.ear")]'
2020-03-18 18:38:23,336 ERROR [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0021: Deploy of deployment "IBSApiEAR.ear" was rolled back with the following failure message: "WFLYCTL0344: Operation timed out awaiting service container stability"
18:38:23,336 ERROR [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0021: Deploy of deployment "IBSApiEAR.ear" was rolled back with the following failure message: "WFLYCTL0344: Operation timed out awaiting service container stability"
2020-03-18 18:38:28,332 INFO [org.jboss.as.protocol] (ServerService Thread Pool -- 89) WFLYPRT0057: cancelled task by interrupting thread Thread[ServerService Thread Pool -- 25,5,ServerService ThreadGroup]
18:38:28,332 INFO [org.jboss.as.protocol] (ServerService Thread Pool -- 89) WFLYPRT0057: cancelled task by interrupting thread Thread[ServerService Thread Pool -- 25,5,ServerService ThreadGroup]
2020-03-18 18:38:28,331 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 25) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$1@6ec64843 for operation deploy at address [("deployment" => "IBSApiEAR.ear")] failed handling operation rollback -- java.lang.InterruptedException: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:515)
at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1485)
at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1439)
at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1412)
at org.jboss.as.controller.AbstractOperationContext$Step.access$400(AbstractOperationContext.java:1286)
at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:859)
at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:709)
at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:450)
at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1402)
at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:418)
at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:263)
at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:229)
at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler.internalExecute(TransactionalProtocolOperationHandler.java:249)
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler.doExecute(TransactionalProtocolOperationHandler.java:190)
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:143)
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:139)
at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:263)
at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:229)
at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:198)
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:162)
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:158)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2.execute(TransactionalProtocolOperationHandler.java:158)
at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$1.doExecute(ManagementRequestContextImpl.java:70)
at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$AsyncTaskRunner.run(ManagementRequestContextImpl.java:160)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
18:38:28,331 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 25) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$1@6ec64843 for operation deploy at address [("deployment" => "IBSApiEAR.ear")] failed handling operation rollback -- java.lang.InterruptedException: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:515)
at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1485)
at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1439)
at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1412)
at org.jboss.as.controller.AbstractOperationContext$Step.access$400(AbstractOperationContext.java:1286)
at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:859)
at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:709)
at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:450)
at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1402)
at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:418)
at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:263)
at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:229)
at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler.internalExecute(TransactionalProtocolOperationHandler.java:249)
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler.doExecute(TransactionalProtocolOperationHandler.java:190)
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:143)
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:139)
at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:263)
at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:229)
at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:198)
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:162)
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:158)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2.execute(TransactionalProtocolOperationHandler.java:158)
at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$1.doExecute(ManagementRequestContextImpl.java:70)
at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$AsyncTaskRunner.run(ManagementRequestContextImpl.java:160)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
2020-03-18 18:38:33,336 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 25) WFLYCTL0349: Timeout after [5] seconds waiting for service container stability while finalizing an operation. Process must be restarted. Step that first updated the service container was 'deploy' at address '[("deployment" => "IBSApiEAR.ear")]'
18:38:33,336 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 25) WFLYCTL0349: Timeout after [5] seconds waiting for service container stability while finalizing an operation. Process must be restarted. Step that first updated the service container was 'deploy' at address '[("deployment" => "IBSApiEAR.ear")]'
Anyone know what is going on and how to solve this? Or there is another log I can check further on this?
I found out the root cause at the end.
It is because of the firewall blocking between my app and the new database.
Even I can ping the database ip and telnet the port from my application, but its still some firewall rule blocking the access.
After I ask network team to turn off the rule, then it will run smoothly.