docker pull from Artifactory results in "net/http: request canceled" inconsistently

We are running Artifactory 5.11.0 (just update to 6.0.2 today and haven't yet seen this) in a docker container and when our automation executes a docker pull from Artifactory, 9/10 times it is successful. Sometimes, even when running the docker pull from the machine hosting Artifactory, the docker pull fails with:

Pulling 'docker.{artifactory url}/staging:latest'... Error response from daemon: Get http://docker.{artifactory url}/v2/staging/manifests/latest: Get http://docker.{artifactory url}:80/artifactory/api/docker/docker/v2/token?account=admin&scope=repository%3Astaging%3Apull&service=docker.{artifactory url}%3A80: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Like I said, most of the time this is working perfect, but that 1/10 (probably less) we get the above error during our automated builds. I tried running the docker pull in a while loop over night until it hit a failure and there was no failure. Ran ping overnight and no packets were lost.

OS: Debian 9 x64
Docker version 17.09.0-ce, build afdb6d4 and seems to happen more frequently with Docker version 18.03.1~ce-0~debian, but I have no direct evidence to suggest the client is at fault.

Solution

Here is what JFrog provided me to try to resolve this issue. (Note: we were on an older version of Artifactory at the time and they did recommend that we update it to the latest as there were several updates that could help).

The RAM value -Xmx 2g was the default value provided by Artifactory. We can increase that value by going into the Docker container "docker exec -it artifactory bash"
and then $Artifactory_Home/bin/artifactory.default ( Mostly: - /opt/jfrog/artifactory/bin/artifactory.default) and we can change the RAM value accordingly. Please follow this link for more information.

We should also change the access max threads count and we can do that by going to $Artifactory_Home/tomcat/config/server.xml and change it to:

        <Connector port="8040" sendReasonPhrase="true" maxThreads="<200>"/>

Also add below line in /var/opt/jfrog/artifactory/etc/artifactory.system.properties

        artifactory.access.client.max.connections=200

To deal with heavy loads we need to append the below line  in /var/opt/jfrog/artifactory/etc/db.properties.Please follow this link for more information.

        pool.max.active=200

Also, they told me to be sure that we were using the API Key when authenticating the docker client with Artifactory instead of user/pass login since the latter will go through our ldap authentication and the former will not:

One thing to try would be to use an API Key instead of the plain text password, as using an API key will not reach out to the LDAP server.

We were already doing this, so this had no impact on the issue.

Also posted here: https://www.jfrog.com/jira/browse/RTFACT-17919?focusedCommentId=66063&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-66063

I hope this helps as it helped us.