Search code examples
rubysinatraconfigracksidekiq

SIdekiq doesn't perform async (Sinatra & Rack)

I am really newbie to Ruby. I understand the language, but it is really hard for me to get the right way to structure project, there are a lot of articles and tutorials suggest running not with simple ruby command but for example rackup. I cannot get an idea of using other commands if my app using multiple gems, there are used as a wrappers ?

As for my project. I am creating simple API with Sinatra & Rack & Sidekiq, I am starting my app as follows.

rackup -p1600 --host 192.168.0.130  config.ru

But I've just started using sidekiq and it required Redis server I've installed it, now everything works no errors.

But the problem is that my task is not being processed.

Here is my example

My endpoint 

  post '/items' do
    item_url = params[:item_url]
    halt(400, {error: 'Item url is not provided'}.to_json) if item_url.nil?
    begin
      item_handler = ItemHandler.new item_url
      item_handler.start_processing
      item_handler.item_status.to_json
    rescue APIErrors::AlreadyExistsError => e
      halt(409, {error: e.message}.to_json)
    rescue APIErrors::InvalidPayloadError => e
      halt(400, {error: e.message}.to_json)
    end
  end

And my ItemHandler

Sidekiq.configure_server do |config|
  config.redis = {password: 'password'}
end
Sidekiq.configure_client do |config|
  config.redis = {password: 'password'}
end

class ItemHandler
...
...
...
  def start_processing
    ItemWorker.perform_async(@item.id)
  end

And finnally ItemWorker

require 'sidekiq'

class ItemWorker
  include Sidekiq::Worker
  attr_accessor :item

  def perform(id)
    # Get item model
    @item = ItemModel.where(_id: id)
    logger.info "Doing hard work"
    puts @item.status
    # Start item processing
    process_item
  end

  def process_item
    logger.info "Doing hard work"
    puts 'Start processing'
    @item.status = ItemModel::STATUS[:downloading]
    @item.save
    result = download_item_file
    if result > RESULT_OK
      @item.status = ItemModel::STATUS[:failed_download]
      @item.save
      puts 'Failed to download file'
    else
      puts 'File downloaded'
      @item.status = ItemModel::STATUS[:downloaded]
      @item.save
    end
  end

  def download_item_file
    return if @item.nil?
    dl_command = EXTERNAL_DOWNLOAD.dup
    dl_command['|url|'] = @item.url
    system dl_command
    $?.exitstatus
  end
end

And nothing happens, no output in console, nothing. Previously I've use fork instead of sidekiq and it worked fine.

Please help me to find the problem.

P.S. If I am doing something other in a wrong way (don't follow best practices & guidelines) please let me know.

Solution

  • Here is a complete example app using sidekiq/redis and sinatra:

    ~/sinatra_projects$ tree myapp/    
myapp/
├── config.ru
├── item_handler.rb
├── item_worker.rb
└── sinatra_app.rb

    sinatra_app.rb:

    require 'sinatra'
require_relative 'item_handler' #=>look for item_handler.rb in the same directory as this file

get '/' do
  "hello world"
end

post '/item' do
  puts "--->#{params[:item_url]}<---"

  item_handler = ItemHandler.new params[:item_url]
  item_handler.start_processing

  "Thanks for the work\n"
end

    item_handler.rb:

    require_relative 'item_worker'  #look for item_worker.rb in the same directory as this file

class ItemHandler
  def initialize(url)
    @url = url
  end

  def start_processing
    ItemWorker.perform_async @url
  end
end

    item_worker.rb:

    require 'sidekiq'

Sidekiq.configure_server do |config|
  #Not much documentation on what you can/should do in here
end

Sidekiq.configure_client do |config|
  #Not much documentation on what you can/should do in here
end

class ItemWorker
  include Sidekiq::Worker

  def perform(url)
    logger.info "Things are happening"

    case url
    when /joe_blow.com/
      sleep 5
      logger.info "joe_blow.com took a really long time" 
    when /twitter_clone.com/
      sleep 3
      logger.info "twitter_clone.com was pretty slow"
    else /google.com/
      sleep 1
      logger.info "google.com response was the quickest"
    end

  end

end

    config.ru:

    require_relative 'sinatra_app'  #=>look for sinatra_app.rb in the same directory as this file 

run Sinatra::Application

    To get things up and running:

    1) Download and install redis(not the same as the redis gem), then in terminal_window1 start redis:

    $ ~/Downloads/redis-3.2.6/src$ ./redis-server

    2) Install the sidekiq gem (which will also install the redis gem because it's a dependency):

    $ gem install sidekiq

    3) Start sidekiq in terminal_window2:

    ~/sinatra_projects/myproj$ sidekiq -r ./item_worker.rb

    4) Launch the Sinatra app in terminal_window3:

    ~/sinatra_projects/myproj$ rackup config.ru

    5) Send a post request to your app in terminal_window4:

    $ curl --data "item_url=http://joe_blow.com" http://localhost:9292/item

    Immediately, the curl window will display the output:

    Thanks for the work

    and the sidekiq window will display the output:

    2016-12-28T02:34:59.149Z 12387 TID-oxt2yycrk ItemWorker JID-b9190f121541d82f21483497 INFO: start
2016-12-28T02:34:59.149Z 12387 TID-oxt2yycrk ItemWorker JID-b9190f121541d82f21483497 INFO: Things are happening

    Five seconds later, the sidekiq window will display:

    2016-12-28T02:35:04.153Z 12387 TID-oxt2yycrk ItemWorker JID-b9190f121541d82f21483497 INFO: joe_blow.com took a really long time
2016-12-28T02:35:04.154Z 12387 TID-oxt2yycrk ItemWorker JID-b9190f121541d82f21483497 INFO: done: 5.004 sec

    And if you send three curl requests in rapid succession(up_arrow + Return):

    $ curl --data "item_url=http://joe_blow.com" http://localhost:9292/item
Thanks for the work
$ curl --data "item_url=http://joe_blow.com" http://localhost:9292/item
Thanks for the work
$ curl --data "item_url=http://joe_blow.com" http://localhost:9292/item
Thanks for the work

    then in the sideqik window you will see:

    2016-12-28T02:38:01.218Z 12387 TID-oxt2yycrk ItemWorker JID-3c8db3dc597789eefaf3d7e6 INFO: start
2016-12-28T02:38:01.218Z 12387 TID-oxt2yycrk ItemWorker JID-3c8db3dc597789eefaf3d7e6 INFO: Things are happening
2016-12-28T02:38:01.983Z 12387 TID-oxt319uyc ItemWorker JID-d5a4f4fa5388b2fdb94b8549 INFO: start
2016-12-28T02:38:01.983Z 12387 TID-oxt319uyc ItemWorker JID-d5a4f4fa5388b2fdb94b8549 INFO: Things are happening
2016-12-28T02:38:02.602Z 12387 TID-oxt2y8eec ItemWorker JID-dd8b2ce2e558f5a88a1836fa INFO: start
2016-12-28T02:38:02.602Z 12387 TID-oxt2y8eec ItemWorker JID-dd8b2ce2e558f5a88a1836fa INFO: Things are happening

    ...time goes by...

    2016-12-28T02:38:06.220Z 12387 TID-oxt2yycrk ItemWorker JID-3c8db3dc597789eefaf3d7e6 INFO: joe_blow.com took a really long time
2016-12-28T02:38:06.220Z 12387 TID-oxt2yycrk ItemWorker JID-3c8db3dc597789eefaf3d7e6 INFO: done: 5.003 sec
2016-12-28T02:38:06.985Z 12387 TID-oxt319uyc ItemWorker JID-d5a4f4fa5388b2fdb94b8549 INFO: joe_blow.com took a really long time
2016-12-28T02:38:06.985Z 12387 TID-oxt319uyc ItemWorker JID-d5a4f4fa5388b2fdb94b8549 INFO: done: 5.002 sec
2016-12-28T02:38:07.603Z 12387 TID-oxt2y8eec ItemWorker JID-dd8b2ce2e558f5a88a1836fa INFO: joe_blow.com took a really long time
2016-12-28T02:38:07.603Z 12387 TID-oxt2y8eec ItemWorker JID-dd8b2ce2e558f5a88a1836fa INFO: done: 5.001 sec

    Instead of taking a total of 15 seconds to run all three workers, after 5 seconds all three workers finished.