Goliath breaks em-synchrony/em-hiredis when multiple with_api() tests are called in one RSpec suite

Question

I'm just experiencing weird behavior while testing an Goliath API with RSpec. One of my tests looks like this:

require 'helper'

describe Scales::Dispatch do

  it "should return a 404 if resource was not found" do
    with_api(Scales::Server) do
      get_request(:path => '/') do |client|
        client.response_header.http_status.should == 404
      end
    end
  end

  it "should return a resource" do
    Scales::Storage::Sync.set "/existing", "some content"

    with_api(Scales::Server) do
      get_request(:path => '/existing') do |client|
        client.response_header.http_status.should == 200
        client.response.should == "some content"
      end
    end

    Scales::Storage::Sync.del "/existing"
  end

end

The API basically just looks up a key in a redis with the help of em-synchrony/em-hiredis like this:

module Scales
  module Lookup
    class << self

      def request(env)
        response = Storage::Async.get(path(env))
        response.nil? ? render_not_found : render(response)
      end

      private

      def path(env)
        env["REQUEST_URI"]
      end

      def render_not_found
        [404, {}, ""]
      end

      def render(response)
        [200, {}, response]
      end

    end
  end
end

Both tests run individually, but not together. After the first is executed, the whole system stalls about 10 seconds. The second with_api is then called but the get_request is never executed - and i think it is running in some sort of timeout.

I discovered the same behavior at another, pretty similar test which is pushing and popping a queue like this:

describe Scales::Queue::Async do

  [Scales::Queue::Async::Request, Scales::Queue::Async::Response].each do |queue|
    context queue.name.split("::").last do

      it "should place a few jobs" do
        async do
          queue.add "job 1"
          queue.add "job 2"
          queue.add "job 3"
        end
      end

      it "should take them out blocking" do
        async do
          queue.pop.should == "job 1"
          queue.pop.should == "job 2"
          queue.pop.should == "job 3"
        end
      end

    end
  end

end

The contents of second async do .. is also not executed at all. Without goliath loaded a pretty similar test runs perfectly:

require 'eventmachine'
require 'em-synchrony'
require 'em-synchrony/em-hiredis'

module Helpers

  def async
    if EM.reactor_running?
      yield
    else
      out = nil
      EM.synchrony do
        out = yield
        EM.stop
      end
      out
    end
  end

end

RSpec.configure do |config|
  config.include Helpers
  config.treat_symbols_as_metadata_keys_with_true_values = true
end

describe "em-synchrony/em-hiredis" do

  it "should lpush a job" do
    async do
      redis = EM::Hiredis.connect
      redis.lpush("a_queue", "job1")
    end
  end

  it "should block pop a job" do
    async do
      redis = EM::Hiredis.connect
      redis.brpop("a_queue", 0).last.should == "job1"
    end
  end

end

The async do .. for the previous task is the same RSpec helper.

I was searching the whole day like crazy, but to me it doesn't make any sense. Because the last test is running completely fine, I guess its neither a em-synchrony nor a em-synchrony/em-hiredis thing.

Maybe goliath is not stopping, occupying the EM somewhat too long or so?

Thanks for your help, this is driving me nuts!

Answer 1

Okay, I found the solution.

I checked to connection right before every request and if it was there I didn't reestablish it. But it appears as every stopping of the eventmachine closes the connection, so basically for every new request there was a connection timeout that failed silently.

Thanks for your time!