Uploading files in CKAN but received 'Could not connect to DataPusher' error

Question

I'm trying to upload a file to CKAN. However, I received this error from the log:

{'message': 'Could not connect to DataPusher.', 'details': "HTTPConnectionPool(host='localhost', port=8800): Max retries exceeded with url: /job (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 99] Cannot assign requested address',))"}

After CKAN retried few times, the uploading finally finished. I was able to download my resource and track the view.

I don't understand why the connection failed the first couple of times. I tried curl to test the connection by using: curl http://localhost:8800 but I received the error:

curl: (56) Recv failure: Connection reset by peer.

Then I tried to modify my ckan.ini and now the settings are specified as below:

ckan.plugins = datastore datapusher
ckan.datapusher.url = http://localhost:8800

Also in datapusher.conf my host is set to 0.0.0.0 and port is set to 8800:

Listen 8800
<VirtualHost 0.0.0.0:8800>
    ServerName ${CKAN_SERVER_NAME}
    ServerAlias www.${CKAN_SERVER_NAME}

    <Directory "${DATAPUSHER_CONFIG}">
      Require all granted
    </Directory>

    WSGIScriptAlias / ${DATAPUSHER_CONFIG}/datapusher.wsgi
    WSGIPassAuthorization On

    ErrorLog /var/log/apache2/datapusher.error.log
    CustomLog /var/log/apache2/datapusher.custom.log combined

    <IfModule mod_headers.c>
      Header set X-UA-Compatible "IE=edge"
    </IfModule>
</VirtualHost>

In datapusher_settings.py, the Host and Port are also set to '0.0.0.0' and 8800.

I read this post but I didn't find anything different from this setting. Then I changed host in ckan.ini from localhost name space to the actual 127.0.0.1, and restart apache service in CKAN docker container, but it still returns

Could not connect to Datapusher

I tried to upload files but end with a different error number like below:

{'message': 'Could not connect to DataPusher.', 'details': "HTTPConnectionPool(host='127.0.0.1', port=8800): Max retries exceeded with url: /job (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',))"}

and surprisingly after a few times of retry connection in CKAN, I was able to view the resource file and download it. I'm confused because I'm using the same settings for months and I haven't update any configurations and this problem just started to occur recently.

Answer 1

I found a way to resolve the problem manually for development. The problem is caused by datapusher not running on the machine. I manually started datapusher by

python datapusher/main.py deployment/datapusher_settings.py

After datapusher is running without interruption, ckan can connect to datapusher again.