Users get disconnected during test runs

Question

I work on a Ruby on Rails app that has many test in its deployment process (~3000) We use capybara and selenium for feature tests. We recently migrated CI from Heroku CI to CircleCi. We run 10 docker instances to run our test suite.

For many feature tests, we use this block of code to sign in admin users:

def admin_signs_in
  admin_user = FactoryBot.create(:admin_user, :for_signin)
  sign_in admin_user
  return admin_user
end

But sometimes, randomly, one test doesn't pass because users get disconnected and cannot access to the page needed for the test.

An example of failing test: require 'rails_helper'

RSpec.describe "Admin sets client budget to project", js: true do
  let(:client)  { create(:client) }
  let(:project) { create(:project, client: client) }
  let(:timeslots_proposal) { create(:timeslots_proposal_with_client, project: project) }

  before do
    admin_signs_in
    # this path needs an authenticated user
    visit advanced_admin_timeslots_proposal_path(timeslots_proposal)

    within "#client-budget" do
      find(".fa-pen").click
    end
  end

  describe "negative amount" do
    before do
      within "#some_container" do  
        # this block doesn't fail, meaning that at this point, user is authenticated          
        expect(find("#some_field")["placeholder"]).to eq "Enter amount"
        fill_in "some_field", with: "-235.99"
        find('.btn-primary').click
      end
    end

    it "raises an error" do
      # this fails, logs indicate that user is redirected to signin page
      expect(page).to have_content "Amount must be positive"
    end
  end
end

We tried to use rspec-retry gem to try those tests several times, but when it fails once, it fails for all retries.

It can happen to any test in our suite, I'd say it happens in 0.1% of authenticated tests, randomly.

I suspect a problem with Capybara cookies, but I don't know how to solve this.

Answer 1

Thanks to Thomas Walpole, I finally found the issue. First thing is obviously to get logs from test runs.

I managed to save Selenium driver logs for failing tests using this in my config:

config.after(:each) do |example|
  if example.metadata[:js] && example.exception
    File.write("#{path}/driver_logs.txt", Capybara.page.driver.browser.manage.logs.get(:driver).map(&:to_s).join("\n"))
  end
end

I had in mind that the problem was probably due to the _session_id cookie that Rails use to store user session.

So in my huge 2M logs file I searched for this and found this line

"Set-Cookie": "visitor_uuid=1234567890987654321; path=/; SameSite=None\n_session_id=1234567890987654321azertyu; path=/; expires=Sun, 04 Apr 2021 22:00:01 GMT; HttpOnly",

NB: We are in 2022 at the time I'm writing this post. And Selenium chrome is aware of that, so my cookie immediately expires.

I know we use Timecop to set current time for some date related tests, so I searched for a bad Timecop usage.

I found this line

before { Timecop.travel(Time.parse("2021-01-05")) }

Which is never released. I changed it to :

before { Timecop.freeze(Time.parse("2021-01-05")) }
after  { Timecop.return }

So my issue was that my test server and test browser had a different time.