Search code examples
python-3.xrabbitmqceleryworkerbroker

Why Celery worker gives "OSError: Socket closed"


My celery worker that works with rabbitMQ keeps giving me a socket error -- see below -- after working for a few minutes. I'd like to know what could be the main cause of the problem? I thought that it could be the firewall. But, disabling the firewall did not solve the problem. I am working on a Windows 10 machine.

C:\Users\user_\Desktop\Aida>celery -A tasks worker -l info -P eventlet

 -------------- celery@user-RazerBlade v4.2.0 (windowlicker)
---- **** -----
--- * ***  * -- Windows-10-10.0.17134-SP0 2018-07-13 12:27:03
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         tasks:0x4b95370
- ** ---------- .> transport:   amqp://guest:**@localhost:5672//
- ** ---------- .> results:     redis://
- *** --- * --- .> concurrency: 4 (eventlet)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> queueA           exchange=(direct) key=tasks.task_1
                .> queueB           exchange=(direct) key=tasks.task_2
                .> queueC           exchange=(direct) key=tasks.task_3
                .> queueD           exchange=(direct) key=tasks.task_4

[tasks]
  . tasks.task_1
  . tasks.task_2
  . tasks.task_3
  . tasks.task_4

[2018-07-13 12:27:03,119: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2018-07-13 12:27:03,133: INFO/MainProcess] mingle: searching for neighbors
[2018-07-13 12:27:04,166: INFO/MainProcess] mingle: all alone
[2018-07-13 12:27:04,212: INFO/MainProcess] pidbox: Connected to amqp://guest:**@127.0.0.1:5672//.
[2018-07-13 12:27:04,217: INFO/MainProcess] celery@user-RazerBlade ready.
[2018-07-13 12:30:04,270: WARNING/MainProcess] Traceback (most recent call last):
[2018-07-13 12:30:04,271: WARNING/MainProcess] File "c:\users\user_\appdata\local\programs\python\python37-32\lib\site-packages\eventlet\hubs\selects.py", line 55, in wait
    listeners.get(fileno, noop).cb(fileno)
[2018-07-13 12:30:04,276: WARNING/MainProcess] File "c:\users\user_\appdata\local\programs\python\python37-32\lib\site-packages\celery\worker\pidbox.py", line 120, in loop
    connection.drain_events(timeout=1.0)
[2018-07-13 12:30:04,277: WARNING/MainProcess] File "c:\users\user_\appdata\local\programs\python\python37-32\lib\site-packages\kombu\connection.py", line 301, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
[2018-07-13 12:30:04,278: WARNING/MainProcess] File "c:\users\user_\appdata\local\programs\python\python37-32\lib\site-packages\kombu\transport\pyamqp.py", line 103, in drain_events
    return connection.drain_events(**kwargs)
[2018-07-13 12:30:04,279: WARNING/MainProcess] File "c:\users\user_\appdata\local\programs\python\python37-32\lib\site-packages\amqp\connection.py", line 491, in drain_events
    while not self.blocking_read(timeout):
[2018-07-13 12:30:04,281: WARNING/MainProcess] File "c:\users\user_\appdata\local\programs\python\python37-32\lib\site-packages\amqp\connection.py", line 496, in blocking_read
    frame = self.transport.read_frame()
[2018-07-13 12:30:04,285: WARNING/MainProcess] File "c:\users\user_\appdata\local\programs\python\python37-32\lib\site-packages\amqp\transport.py", line 243, in read_frame
    frame_header = read(7, True)
[2018-07-13 12:30:04,286: WARNING/MainProcess] File "c:\users\user_\appdata\local\programs\python\python37-32\lib\site-packages\amqp\transport.py", line 426, in _read
    raise IOError('Socket closed')
[2018-07-13 12:30:04,287: WARNING/MainProcess] OSError: Socket closed
[2018-07-13 12:30:04,288: WARNING/MainProcess] Removing descriptor: 756

Solution

  • I'm not windows user but I'll try to give you some hints.

    (1) - check if your rabbitmq configuration matches your connection string (specifically in linux I can configure rabbitmq to only listen on specific interface so it would not accept connections on, say, loop back interface); The configuration option I would check is NODE_IP_ADDRESS

    (2) - assuming you are 100% certain your rabbitmq is running I'd check if NODENAME is matching your factual system configuration (i.e. check for host name correctness)

    (3) - the most obvious but still worth checking is if your rabbitmq user is correct (I see you have guest user, did you create this user yourself? did you give it permissions to your specific vhost?)

    (4) - and this leads to another obvious question if your connection string is correct. My connection string looks something like this BROKER_URL = 'amqp://my_user:very_secret_password@localhost:5672/my_vhost_name (and on this one from your log it seems you have not configured your connection string to use vhost)