Docker Rails works on dev but not on staging

Question

My development environment is working successfully which consists of Ubuntu LTS 22.04, Docker, Rails 7.1, and Puma v6.4.2

My staging server environment is not working. This environment is using Ubuntu LTS 22.04, Docker, Rails 7.1 & Puma v6.4.2, and Apache. The error I am getting is below. The CalsShibGem is a custom written gem which lives in '/lib/' directory of Rails app. I will paste below the files that will hopefully be helpful. Has anyone encountered this issue before? Dev environment (Docker container) works without issue but moving the app to staging server (running Docker staging container) causes the below error...

Puma starting in single mode...
* Puma version: 6.4.2 (ruby 3.2.4-p170) ("The Eagle of Durango")
*  Min threads: 5
*  Max threads: 5
*  Environment: staging
*          PID: 132
! Unable to load application: NameError: uninitialized constant CalsShibGem::Lib::CalsShib
bundler: failed to load command: puma (/usr/local/bundle/bin/puma)
/usr/local/bundle/gems/zeitwerk-2.6.16/lib/zeitwerk/loader/eager_load.rb:180:in `const_get': uninitialized constant CalsShibGem::Lib::CalsShib (NameError)

            queue << [abspath, namespace.const_get(cname, false)]
                                        ^^^^^^^^^^
    from /usr/local/bundle/gems/zeitwerk-2.6.16/lib/zeitwerk/loader/eager_load.rb:180:in `block in actual_eager_load_dir'
    from /usr/local/bundle/gems/zeitwerk-2.6.16/lib/zeitwerk/loader/helpers.rb:47:in `block in ls'
    from /usr/local/bundle/gems/zeitwerk-2.6.16/lib/zeitwerk/loader/helpers.rb:25:in `each'
    from /usr/local/bundle/gems/zeitwerk-2.6.16/lib/zeitwerk/loader/helpers.rb:25:in `ls'
    from /usr/local/bundle/gems/zeitwerk-2.6.16/lib/zeitwerk/loader/eager_load.rb:168:in `actual_eager_load_dir'
    from /usr/local/bundle/gems/zeitwerk-2.6.16/lib/zeitwerk/loader/eager_load.rb:17:in `block (2 levels) in eager_load'

Thank you, Chris.

Gemfile

source "https://rubygems.org"

ruby "3.2.4"

# Bundle edge Rails instead: gem "rails", github: "rails/rails", branch: "main"
gem "rails", "~> 7.1.3", ">= 7.1.3.4"

# The original asset pipeline for Rails [https://github.com/rails/sprockets-rails]
gem "sprockets-rails"

# Use mysql as the database for Active Record
gem "mysql2", "~> 0.5"

# Use the Puma web server [https://github.com/puma/puma]
gem "puma", ">= 5.0"

# Bundle and transpile JavaScript [https://github.com/rails/jsbundling-rails]
gem "jsbundling-rails"

# Hotwire's SPA-like page accelerator [https://turbo.hotwired.dev]
gem "turbo-rails"

# Hotwire's modest JavaScript framework [https://stimulus.hotwired.dev]
gem "stimulus-rails"

# Bundle and process CSS [https://github.com/rails/cssbundling-rails]
gem "cssbundling-rails"

# Build JSON APIs with ease [https://github.com/rails/jbuilder]
gem "jbuilder"

# Use Redis adapter to run Action Cable in production
# gem "redis", ">= 4.0.1"

# Use Kredis to get higher-level data types in Redis [https://github.com/rails/kredis]
# gem "kredis"

# Use Active Model has_secure_password [https://guides.rubyonrails.org/active_model_basics.html#securepassword]
# gem "bcrypt", "~> 3.1.7"

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem "tzinfo-data", platforms: %i[ windows jruby ]

# Reduces boot times through caching; required in config/boot.rb
gem "bootsnap", require: false

# Use Active Storage variants [https://guides.rubyonrails.org/active_storage_overview.html#transforming-images]
# gem "image_processing", "~> 1.2"

gem 'exception_notification'

# Date/time parse written in pure Ruby.
gem 'chronic'

# Add email validation gem
# Ex: validates_format_of :email, :with => RFC822::EMAIL
gem 'rfc-822'

# HTTP Client - https://github.com/lostisland/faraday
gem 'faraday'

# Role Auth Gem - https://github.com/varvet/pundit
gem "pundit"

# Custom Gem/Plugin: Capi2
gem 'capi2', path: 'lib/capi2'

# Cals::Shib  Plugin
#   * rspec file lives in /spec/cals_shib/
gem 'cals_shib', path: 'lib/cals_shib_gem'

# Working with Excel Files
# ... read Excel files
#gem "roo", "~> 2.8.0"
# ...  generating Excel files.
gem 'caxlsx'
# https://github.com/caxlsx/caxlsx
gem 'caxlsx_rails'


group :development, :test do
  # See https://guides.rubyonrails.org/debugging_rails_applications.html#debugging-with-the-debug-gem
  gem "debug", platforms: %i[ mri windows ]

  # RSpec - https://github.com/rspec/rspec-rails
  gem 'rspec-rails', '~> 6.1.0'
end

group :test do
  # https://github.com/thoughtbot/shoulda-matchers
  gem 'shoulda-matchers', '~> 6.0'
end

group :development do
  # Use console on exceptions pages [https://github.com/rails/web-console]
  gem "web-console"

  # Add speed badges [https://github.com/MiniProfiler/rack-mini-profiler]
  # gem "rack-mini-profiler"

  # Speed up commands on slow machines / big apps [https://github.com/rails/spring]
  # gem "spring"
end

config/puma.rb

# This configuration file will be evaluated by Puma. The top-level methods that
# are invoked here are part of Puma's configuration DSL. For more information
# about methods provided by the DSL, see https://puma.io/puma/Puma/DSL.html.

# Puma can serve each request in a thread from an internal thread pool.
# The `threads` method setting takes two numbers: a minimum and maximum.
# Any libraries that use thread pools should be configured to match
# the maximum value specified for Puma. Default is set to 5 threads for minimum
# and maximum; this matches the default thread size of Active Record.
max_threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
min_threads_count = ENV.fetch("RAILS_MIN_THREADS") { max_threads_count }
threads min_threads_count, max_threads_count

# Specifies that the worker count should equal the number of processors in production.
if ENV["RAILS_ENV"] == "production"
  require "concurrent-ruby"
  worker_count = Integer(ENV.fetch("WEB_CONCURRENCY") { Concurrent.physical_processor_count })
  workers worker_count if worker_count > 1
end
# if ENV["RAILS_ENV"] == "staging"
#   require "concurrent-ruby"
#   worker_count = Integer(ENV.fetch("WEB_CONCURRENCY") { Concurrent.physical_processor_count })
#   workers worker_count if worker_count > 1
# end

# Specifies the `worker_timeout` threshold that Puma will use to wait before
# terminating a worker in development environments.
worker_timeout 3600 if ENV.fetch("RAILS_ENV", "development") == "development"

# Specifies the `port` that Puma will listen on to receive requests; default is 3000.
port ENV.fetch("PORT") { 3000 }

# Specifies the `environment` that Puma will run in.
environment ENV.fetch("RAILS_ENV") { "development" }

# Specifies the `pidfile` that Puma will use.
pidfile ENV.fetch("PIDFILE") { "tmp/pids/server.pid" }

# Allow puma to be restarted by `bin/rails restart` command.
plugin :tmp_restart

Dockerfile.staging

FROM ruby:3.2.4
LABEL maintainer="Chris "

# The base image is based on Debian, and we use apt to install packages.  Apt
# will use the DEBIAN_FRONTEND environment variable to allow limited control
# in its behavior.  In this case, we don't want it to ask interactive questions
# as that will make the docker build command appear to be hung.
ENV DEBIAN_FRONTEND noninteractive

# Download latest package information and install packages.
# -y option says to answer yes to any prompts.
# -qq option enables quiet mode to reduce printed output.
# Note: it is always recommended to combine the apt-get update and
#       apt-get install commands into a single RUN instruction.
# apt-transport-https = allow apt to work with https-based sources
# RUN apt-get update -yqq
# rm -rf /var/lib/apt/lists/* == removes nodejs package lists.
RUN apt-get update -y && apt-get --force-yes install -y --no-install-recommends  \
    build-essential \
    vim \
    curl \
    less \
    libmariadb-dev \
    logrotate \
    git && \
    rm -rf /var/lib/apt/lists/*
# redis-tools && \    THE 2nd to last line needs appersands.


# Change some environment variables from the defaults set in the official Docker image for Ruby
#RUN echo $PATH

# Install Nodejs
COPY scripts/install_nodejs.sh ./
RUN ./install_nodejs.sh && rm ./install_nodejs.sh
RUN echo "NODE Version:" && node --version

# Create and define the node_modules's cache directory.
RUN mkdir /usr/src/cache
WORKDIR /usr/src/cache

# Install the application's dependencies into the node_modules's cache directory.
COPY package.json ./
COPY package-lock.json ./
RUN npm install
RUN echo "NPM Version:" && npm --version

# Install Yarn globally
RUN npm install --global yarn

# Make this the current working directory for the image. So we can execute Rails \
# cmds against image.
RUN mkdir -p /usr/src/app

# Gemfile Caching Trick
# Note: When using COPY with more than one source file, the destination must
#       be a directory and end with a /
# 1. This creates a separate, independent layer. Docker's cache for this layer
#    will only be busted if either of these two files (Gemfile & Gemfile.lcok) change.
COPY Gemfile* /usr/src/app/

# Copy logrotate app rotate configuration. Used logrotate to rotate all
# logs within /log directory.
COPY extras/logrotate.d/all_logs_in_log_folder.conf /etc/logrotate.d/
# Need to change .conf file permissions in order for logrotate to accept .conf file.
RUN chmod 644 /etc/logrotate.d/all_logs_in_log_folder.conf
# Add the cron job. Runs on the 14 min of every hour.
RUN crontab -l | { cat; echo "14 * * * * /usr/sbin/logrotate /etc/logrotate.d/all_logs_in_log_folder.conf"; } | crontab -

# Need to copy custom gem files over b/c 'bundle install' looks for those files.
COPY lib/cals_shib_gem /usr/src/app/lib/cals_shib_gem
COPY lib/capi2 /usr/src/app/lib/capi2

# CD or change into the working directory.
WORKDIR /usr/src/app

# Set timezone. Which conflicted with trying to connect to campus Oracle database.
# The Oracle error is: ORA-01805: possible error in date/time operation
ENV TZ=America/Chicago
#RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
#RUN timedatectl set-timezone America/Chicago

RUN echo "gem: --no-document" >> ~/.gemrc && \
  bundle install

# ADD/COPY app files from local directory into container so they are baked into the image.
# The source path on our local machine is always relative to where the Dockerfile is located.
ADD . /usr/src/app

# Add a script to be executed every time the container starts.
# Entrypoint files are used to set up or configure a container at runtime.
# Below file needs to be executable: $ sudo chmod +x docker_entrypoint_staging.sh
ENTRYPOINT ["./entrypoints/docker_entrypoint_staging.sh"]

docker-compose.staging.yml

version: '3.8'

# compose project name - this is needed in order to successfully run
#                        eops_staging and eops_prod on the same server.
#                        Project name, distinguishes staging vs production
#                        services => app.
name: bldg_access_staging

services:
  app:
    logging:
      driver: awslogs
      options:
        awslogs-region: us-west-2
        awslogs-group: bldg_access_stag
        # Use container name else by default will be container ID.
        awslogs-stream: bldg_access_stag
    image: registry.wisc.edu/bldg_access_v2/app_v3/bldg_access_staging
    container_name: bldg_access_staging
    #restart: always
    environment:
      # Represent Puma
      PIDFILE: /tmp/pids/server.pid
      # Represent Passenger
      #PIDFILE: /tmp/pids/passenger.3000.pid
      RAILS_ENV: staging
      RACK_ENV: staging
      RAILS_LOG_TO_STDOUT: true
      TZ: "America/Chicago"
    env_file:
      - app.env
    tmpfs:
      # tmpfs mount is temporary and persists in the host memory.
      # When container stops, the tmpfs mount is removed, and all files in it will be gone.
      - /tmp/pids/
    ports:
      - "3025:3000"
networks:
  default:
    name: nonsensitive-network
    external: true

docker_entrypoint_staging.sh

#!/bin/bash

# Entrypoint files are used to set up or configure a container at runtime.

# Tells shell that runs the script to fail fast if there are any problems later in the script.
set -e

# Run multiple services in a container.
# https://docs.docker.com/config/containers/multi-service_container/
# Using Bash Job Controls
# --turn on bash's job control
set -m

cp -r /usr/src/cache/node_modules/. /usr/src/app/node_modules/

# Compile Rails Assets at runtime.
RAILS_ENV=staging bundle exec rake assets:precompile

# Start the primary process and put it in the background
#bundle exec passenger start &
bundle exec puma -C config/puma.rb &

service cron start &

# Start the helper process
#bundle exec sidekiq -C config/sidekiq.yml

# now we bring the primary process back into the foreground
# and leave it there
fg %1

# Then exec the container's main process (what's set as CMD in the Dockerfile).
#exec "$@"
#service cron start && \
#  bundle exec passenger start

Answer 1

You have your gems in lib/ directory which is configured to be autoloaded in rails v7.1, but you're not following zeitwerk file structure. This works fine in development because you're requiring the files, so does the bundler with the gem method.

In production or staging, you're eager loading your app, that means a file like lib/cals_shib_gem/lib/cals_shib.rb is expected to define a CalsShibGem::Lib::CalsShib. File structure relative to lib/ has to correspond to class/module names.

Solution, is to move your gems out of lib/ directory.

You can also stop autoloading lib/ or ignore specific directories under it:

# config/application.rb

# comment this out or add subdirectories to ignore
config.autoload_lib(ignore: %w(assets tasks cals_shib_gem capi2))

You can run bin/rails zeitwerk:check in development to verify your app can be eager loaded.

https://guides.rubyonrails.org/autoloading_and_reloading_constants.html#config-autoload-lib-ignore