Trouble Rescuing from Bunny Connection Net::ReadTimeout

Question

I have a ruby script that uses the Bunny Gem to connect to a rabbitmq instance. The script works for a while, but eventually will die because of a Net::ReadTimeout

E, [2017-08-13T08:48:09.671988 #21351] ERROR -- #<Bunny::Session:0x39eca20 [email protected]:5672, vhost=/, addresses=[104.196.154.25:5672]>: Uncaught exception from consumer #<Bunny::Consumer:32353120 @channel_id=1 @queue=sc_link_queue> @c
onsumer_tag=bunny-1502631967000-46739673895>: #<Net::ReadTimeout: Net::ReadTimeout> @ /home/rails/.rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/net/protocol.rb:158:in `rbuf_fill'
E, [2017-08-13T08:48:32.468023 #23205] ERROR -- #<Bunny::Session:0x42202a0 [email protected]:5672, vhost=/, addresses=[104.196.154.25:5672]>: Uncaught exception from consumer #<Bunny::Consumer:36695920 @channel_id=1 @queue=sc_link_queue> @c
onsumer_tag=bunny-1502631972000-482787698591>: #<Net::ReadTimeout: Net::ReadTimeout> @ /home/rails/.rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/net/protocol.rb:158:in `rbuf_fill'

My script looks like this

module Sc
  class Worker
    def initialize
      init()
    end

    def self.start_headless(type)
      Headless.new(display: 50, destroy_at_exit: false, resuse: true).start
      worker = new
      worker.send(type)
    end

    def init
      $conn ||= Bunny.new($rabbitmq_opts)
      $conn.start
      @browser = Sc::Browser.new()
    rescue Timeout::Error, Net::ReadTimeout, Selenium::WebDriver::Error::UnknownError, Errno::ECONNREFUSED, Selenium::WebDriver::Error::JavascriptError, Exception, StandardError => e
      LOGGER.error("[x] Trouble connecting to rabbitmq, retrying...")
      LOGGER.error("[x] #{e}")
      LOGGER.error("[x] #{e.backtrace}")
      retry
    end

    def listen_for_searches
      channel = $conn.create_channel
      channel.prefetch(1)
      queue = channel.queue($rabbitmq_search_queue, durable: true)
      exchange = channel.default_exchange
      queue.subscribe(:manual_ack => true, :block => true) do |delivery_info, properties, payload|
        LOGGER.info "[x] Received #{payload}"
        payload = JSON.parse(payload)
        scrape = Sc::Search.new(browser: @browser.browser, county: payload["name"], type: payload["type"], date_type: payload["date_type"])
        scrape.run
        scrape.close
        channel.ack(delivery_info.delivery_tag)
      end
    rescue Timeout::Error, Net::ReadTimeout, Selenium::WebDriver::Error::UnknownError, Errno::ECONNREFUSED, Selenium::WebDriver::Error::JavascriptError, Exception, StandardError => e
      LOGGER.error("[x] #{e}")
      LOGGER.error("[x] #{e.backtrace}")
      LOGGER.error("[x] Trouble with scrape, retrying...")
      retry
    end
  end
end

As you can see I'm trying to rescue from pretty much everything that could happen. I still can't seem to get it to recover from the Net::ReadTimeout error. Once the worker dies you can still see that it is connected to rabbitmq, but the last item it took from the queue is unacknowledged, it is essentially hung.

Answer 1

I have solved this. The issue was that everything that runs inside the Bunny subscribe block is handled in a different thread, so you need to add the rescue statements to inside that block.