Search code examples
rubyconcurrencycelluloid

Why do I get random errors by using celluloid?

I need to make API calls to a webservice in order to retrieve date. For this purpose I created a sample in order to get familiar with celluloid. Here I ll user the openweather API for "training" purpose. The ultimate goal is to run multiple requests in concurrently.

my booking class (booking.rb) fetches data

    require 'celluloid'
require 'open-uri'
require 'json'

    class Booking
      include Celluloid


      def initialize

      end

      def parse(url)
        p "back again"
        begin
          buffer = open(url).read
          p JSON.parse(buffer)['cod']
        rescue => e
          "fuck"
        end

      end

    end

and this is how I run it:

require 'celluloid'
Celluloid.shutdown_timeout = 10

begin

  pool = Booking.pool

  %W(
http://api.openweathermap.org/data/2.5/weather?q=London,uk
http://api.openweathermap.org/data/2.5/weather?q=Berlin,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
).each_with_index do |weather, i|
    p i
    #Booking.new.async.parse(weather)
    pool.future.parse(weather)
  end
rescue => e
  p "ex #{e}"
end

p "start"

Now I do get different error messages when I run it several times:

ruby run_booking.rb
0
1
2
3
4
5
6
7
8
9
"start"
D, [2015-06-11T21:20:06.351274 #33316] DEBUG -- : Terminating 9 actors...
E, [2015-06-11T21:20:16.356649 #33316] ERROR -- : Couldn't cleanly terminate all actors in 10 seconds!
➜  booking  ruby run_booking.rb
0
1
2
3
4
5
6
7
8
9
"start"
D, [2015-06-11T21:22:19.172770 #33344] DEBUG -- : Terminating 9 actors...
W, [2015-06-11T21:22:19.173145 #33344]  WARN -- : Terminating task: type=:finalizer, meta={:method_name=>:__shutdown__}, status=:receiving
    Celluloid::TaskFiber backtrace unavailable. Please try `Celluloid.task_class = Celluloid::TaskThread` if you need backtraces here.

So I am wondering what's going on here? Help is appreciated. Thanks in advance

Solution

  • 0. Your program is finishing before Celluloid does its work.

    You need to use the method shown in #2 to avoid that. You need to join each Future to make sure all the information is retrieved and parsed... Otherwise you just fired off a ton of asynchronous calls that do not block ... which means nothing stops the program from exiting.

    When in doubt, just add sleep to the end of your program, until you can figure out how you want to gracefully finish. So far the code you showed is incomplete.

    1. Use the new version Celluloid like this:

    In your Gemfile:

    gem 'celluloid', github: 'celluloid/celluloid', branch: '0.17.0-prerelease', submodules: true

    Then when you require the Celluloid library, use this:

    require 'celluloid/current'

    2. Just use Futures. You don't need a pool at all, it will slow you down:

    results = []
booking = Booking.new

%W(
http://api.openweathermap.org/data/2.5/weather?q=London,uk
http://api.openweathermap.org/data/2.5/weather?q=Berlin,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
http://api.openweathermap.org/data/2.5/weather?q=Munich,de
).each_with_index do |weather, i|
  p i
  results << booking.future.parse(weather)
end

#de Turn the futures into actual results by calling `.value` on each future:
results = results.map(&:value)

    3. Use http.rb with Celluloid support:

    Add Celluloid::IO to your Gemfile like this:

    gem 'celluloid-io', github: 'celluloid/celluloid-io', branch: '0.17.0-dependent', submodules: true

    Then use HTTP instead, and pass in Celluloid::IO socket types. Here's the example from http.rb itself:

    require "celluloid/io"
require "http"

class HttpFetcher
  include Celluloid::IO

  def fetch(url)
    HTTP.get(url, socket_class: Celluloid::IO::TCPSocket)
  end
end

    That call right there uses evented TCP, which goes great with your actor which has concurrent outbound gathering calls.