I am working on a Java application which deploys web artifacts in Apache Tomcat Docker Containers with the use of Google Kubernetes. I am using https://github.com/spotify/docker-client in order to carry out Docker Image and Container handling activities and https://github.com/fabric8io/fabric8/tree/master/components/kubernetes-api for Kubernetes related functionalities.
In this application, I have added a functionality which enables the user to remove a web artifact which the user deploys.
When removing I,
delete the Kubernetes replication controller which I use to generate the desired number of pod replicas
separately delete off the replica pods (as pods are not deleted automatically when the replication controller is deleted in the corresponding method in the Java API)
delete off the corresponding Service created
delete off the Docker Containers corresponding to the pods deleted off
finally, remove the Docker Image used for the deployment
Following code shows the removal functionality implemented:
public boolean remove(String tenant, String appName) throws WebArtifactHandlerException {
String componentName = generateKubernetesComponentName(tenant, appName);
final int singleImageIndex = 0;
try {
if (replicationControllerHandler.getReplicationController(componentName) != null) {
String dockerImage = replicationControllerHandler.getReplicationController(componentName).getSpec()
.getTemplate().getSpec().getContainers().get(singleImageIndex).getImage();
List<String> containerIds = containerHandler.getRunningContainerIdsByImage(dockerImage);
replicationControllerHandler.deleteReplicationController(componentName);
podHandler.deleteReplicaPods(tenant, appName);
serviceHandler.deleteService(componentName);
Thread.sleep(OPERATION_DELAY_IN_MILLISECONDS);
containerHandler.deleteContainers(containerIds);
imageBuilder.removeImage(tenant, appName, getDockerImageVersion(dockerImage));
return true;
} else {
return false;
}
} catch (Exception exception) {
String message = String.format("Failed to remove web artifact[artifact]: %s",
generateKubernetesComponentName(tenant, appName));
LOG.error(message, exception);
throw new WebArtifactHandlerException(message, exception);
}
}
Implementation of the Docker Container deletion functionality is as follows:
public void deleteContainers(List<String> containerIds) throws WebArtifactHandlerException {
try {
for (String containerId : containerIds) {
dockerClient.removeContainer(containerId);
Thread.sleep(OPERATION_DELAY_IN_MILLISECONDS);
}
} catch (Exception exception) {
String message = "Could not delete the Docker Containers.";
LOG.error(message, exception);
throw new WebArtifactHandlerException(message, exception);
}
}
In the above case although the execution of the desired function takes place without any sort of issue, at certain instances I tend to get the following exception.
Sep 11, 2015 3:57:28 PM org.apache.poc.webartifact.WebArtifactHandler remove
SEVERE: Failed to remove web artifact[artifact]: app-wso2-com
org.apache.poc.miscellaneous.exceptions.WebArtifactHandlerException: Could not delete the Docker Containers.
at org.apache.poc.docker.JavaWebArtifactContainerHandler.deleteContainers(JavaWebArtifactContainerHandler.java:80)
at org.apache.poc.webartifact.WebArtifactHandler.remove(WebArtifactHandler.java:206)
at org.apache.poc.Executor.process(Executor.java:222)
at org.apache.poc.Executor.main(Executor.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
Caused by: com.spotify.docker.client.DockerRequestException: Request error: DELETE unix://localhost:80/v1.12/containers/af05916d2bddf73dcf8bf41c6ea7f5f3b859c90b97447a8248ffa7b5b3968691: 409
at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:1061)
at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:1021)
at com.spotify.docker.client.DefaultDockerClient.removeContainer(DefaultDockerClient.java:544)
at com.spotify.docker.client.DefaultDockerClient.removeContainer(DefaultDockerClient.java:535)
at org.wso2.carbon6.poc.docker.JavaWebArtifactContainerHandler.deleteContainers(JavaWebArtifactContainerHandler.java:74)
... 8 more
Caused by: com.spotify.docker.client.shaded.javax.ws.rs.ClientErrorException: HTTP 409 Conflict
at org.glassfish.jersey.client.JerseyInvocation.createExceptionForFamily(JerseyInvocation.java:991)
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:975)
at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:795)
at org.glassfish.jersey.client.JerseyInvocation.access$500(JerseyInvocation.java:91)
at org.glassfish.jersey.client.JerseyInvocation$5.completed(JerseyInvocation.java:756)
at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:189)
at org.glassfish.jersey.client.ClientRuntime.access$300(ClientRuntime.java:74)
at org.glassfish.jersey.client.ClientRuntime$1.run(ClientRuntime.java:171)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:320)
at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:201)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I searched a large number of sources for any help for this but still I wasn't able to avoid it in all instances, I execute this functionality.
At the beginning I tended to get this issue more often than now, but allowing the executing thread to sleep at the end of deleting each Docker Container and before deleting any Docker Containers, gradually reduced the number of instances I am getting this issue.
Is sleeping the thread the ultimate solution for this issue or is there any other reason for this issue to pop and a solution that can help me to avoid this exception? Any help is greatly appreciated.
Unfortunately I'm not familiar with the Java client library.
My suggestion would be to try using the regular command-line client (kubectl). If that works, then you know the problem is in the Java client library or your usage of it. If using the command line client doesn't work, then there will be more people who can help you (since a lot more people are familiar with the command-line client than with the Java client library).
In other words % kubectl delete pods ... # --cascade=true by default % kubectl delete services ...
I'm curious why you need step (4) and (5). Step (4) should happen automatically when you delete the pod, and step (5) should happen automatically in the background.
If the two lines of "kubectl delete" work, then the problem is with the Java client library or your usage of it. As a starting point I would suggest remove calling deleteContainers() and removeImage() from your Java code and see if that helps. I think those steps are unnecessary.