I'm trying to use sidekiq to process some requests in background (it's integrated with padrino framework).
My expectation was that once the worker got called, the request handler would return immediately the answer to user.
In order to be sure it was working properly I implemented a worker that would print some messages and sleep for some 44s prior to finish it's processing (to simulate a long processing in background).
For my surprise, the request got stuck until the worker had the job finished. Only after that the request handler could return an answer to user.
At first I thought that the sleep function could be the problem, so I replaced the sleep function by a busy while but I have the same behavior: the request handler got hanged until the worker finished its task.
Any idea why is this happening?
You can see the following :
request handler:
get :hardworker, map: '/hardworker' do
logger.info "I'll call HardWorker"
HardWorker.perform_async(44)
logger.info "HardWorker was called"
return "The job should still be running in background."
end
Sidekiq worker:
class HardWorker
include Sidekiq::Worker
def perform(count)
logger.info "Doing hard work"
Sidekiq.redis { |c| logger.info "redis location: [#{c.client.location }]" }
redis_info = Sidekiq.redis { |conn| conn.info }
logger.info "connected clients: [#{redis_info['connected_clients']}]"
sleep count
logger.info "hard work was done"
end
end
The redis server is running:
ps -ef | grep redis
returns
redis 1232 1 0 16:54 ? 00:00:09 /usr/bin/redis-server 127.0.0.1:6379
as well as sidekiq server:
bundle exec sidekiq -r ./config/boot.rb -v
2015-06-06T20:31:26.207Z 3343 TID-8614g INFO: Booting Sidekiq 3.3.4 with redis options {:url=>"redis://127.0.0.1:6379/0", :concurrency=>25}
Also, from logs I put in the worker we can see that client apparently is connected to redis:
INFO - redis location: [127.0.0.1:6379]
INFO - connected clients: [3]
sidekiq version: 3.3.4 redis server: 2.8.4 ruby: 2.1.2p95
Note: I mounted sidekiq web tool on my padrino and it shows 0 for all stats (Processed, Failed, Busy, Enqueued, Retries, Scheduled, Dead) both prior and after the worker has executed.
The sample project can be downloaded from here: https://github.com/plicatibu/sidekiq-problem.git
Well I found your bug, in your boot.rb file you have this line:
require 'sidekiq/testing/inline' if RACK_ENV == 'development'
This bit of code, uses Sidekiq's Testing framework which bypasses redis and runs it almost as if it's a ruby class (ie it doesn't ever get queued on redis). Remove that line and only use it in your test suite (if you need it).
You'll also need to remove that pid file from your YML file, you don't need that and if you do, it's probably only for production. My 2cents - remove it.
Once you do that, you're good to go:
Here's your rackup with a get request to /hardworker:
[2015-06-07 07:54:25] INFO WEBrick 1.3.1
[2015-06-07 07:54:25] INFO ruby 2.2.2 (2015-04-13) [x86_64-darwin14]
[2015-06-07 07:54:25] INFO WEBrick::HTTPServer#start: pid=74128 port=9292
INFO - I'll call HardWorker
INFO - HardWorker was called
DEBUG - GET (0.0131s) /hardworker - 200 OK
::1 - - [07/Jun/2015:07:55:32 -0400] "GET /hardworker HTTP/1.1" 200 46 0.0291
Here's sidekiq processing that job in the background:
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
ss
sss sss ss
s sss s ssss sss ____ _ _ _ _
s sssss ssss / ___|(_) __| | ___| | _(_) __ _
s sss \___ \| |/ _` |/ _ \ |/ / |/ _` |
s sssss s ___) | | (_| | __/ <| | (_| |
ss s s |____/|_|\__,_|\___|_|\_\_|\__, |
s s s |_|
s s
sss
sss
INFO - Running in ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-darwin14]
INFO - See LICENSE and the LGPL-3.0 for licensing details.
INFO - Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org/pro
INFO - Starting processing, hit Ctrl-C to stop
DEBUG -
INFO -
INFO - Doing hard work
INFO - redis location: [127.0.0.1:6379]
INFO - connected clients: [3]
INFO - hard work was done
INFO -