I'm deploying a Ruby on Rails application on AWS Beanstalk. The application also needs a sidekiq process for background jobs. There's also a sneakers process running to listen on messages from a RabbitMQ instance.
I created an upstart process for sidekiq using ebextensions from the process outlined here. Using the same outline, I created another upstart process for running the sneakers rake task. All the config files are here in this gist.
The deploy runs fine and I can see the sidekiq and sneaker processes running, but after a few deploys I began seeing a number of rake processes being spawned which are taking up database connections.
[root@ip-XXX ec2-user]# ps aux | grep '[/]opt/rubies/ruby-2.0.0-p648/bin/rake'
webapp 13563 0.0 2.2 1400644 184988 ? Sl 01:41 0:00 /opt/rubies/ruby-2.0.0-p648/bin/rake
webapp 13866 0.7 2.3 694804 193620 ? Sl 01:42 0:10 /opt/rubies/ruby-2.0.0-p648/bin/rake
webapp 14029 0.0 2.2 1400912 183700 ? Sl 01:42 0:00 /opt/rubies/ruby-2.0.0-p648/bin/rake
webapp 14046 0.0 2.2 1400912 183812 ? Sl 01:42 0:00 /opt/rubies/ruby-2.0.0-p648/bin/rake
webapp 14048 0.0 2.2 1400912 183804 ? Sl 01:42 0:00 /opt/rubies/ruby-2.0.0-p648/bin/rake
webapp 14073 0.0 2.2 1400912 183712 ? Sl 01:42 0:00 /opt/rubies/ruby-2.0.0-p648/bin/rake
webapp 14158 0.0 2.2 827056 187972 ? Sl Nov23 4:23 /opt/rubies/ruby-2.0.0-p648/bin/rake
webapp 19139 0.9 2.3 694744 193388 ? Sl 01:47 0:10 /opt/rubies/ruby-2.0.0-p648/bin/rake
webapp 19273 0.0 2.2 1400852 183680 ? Sl 01:47 0:00 /opt/rubies/ruby-2.0.0-p648/bin/rake
webapp 19290 0.0 2.2 1400852 183732 ? Sl 01:47 0:00 /opt/rubies/ruby-2.0.0-p648/bin/rake
[root@ip-XXX ec2-user]# ps auxf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
webapp 14158 0.0 2.2 827056 187972 ? Sl Nov23 4:24 /opt/rubies/ruby-2.0.0-p648/bin/ rake
webapp 13563 0.0 2.2 1400644 185700 ? Sl 01:41 0:00 \_ /opt/rubies/ruby-2.0.0-p648/bin/ rake
webapp 13866 0.4 2.3 694804 193620 ? Sl 01:42 0:11 /opt/rubies/ruby-2.0.0-p648/bin/ rake
webapp 14029 0.0 2.2 1400912 184412 ? Sl 01:42 0:00 \_ /opt/rubies/ruby-2.0.0-p648/bin/ rake
webapp 14046 0.0 2.2 1400912 184372 ? Sl 01:42 0:00 \_ /opt/rubies/ruby-2.0.0-p648/bin/ rake
webapp 14048 0.0 2.2 1400912 184516 ? Sl 01:42 0:00 \_ /opt/rubies/ruby-2.0.0-p648/bin/ rake
webapp 14073 0.0 2.2 1400912 184540 ? Sl 01:42 0:00 \_ /opt/rubies/ruby-2.0.0-p648/bin/ rake
webapp 19139 0.4 2.3 694876 193428 ? Sl 01:47 0:11 /opt/rubies/ruby-2.0.0-p648/bin/ rake
webapp 19273 0.0 2.2 1400852 184288 ? Sl 01:47 0:00 \_ /opt/rubies/ruby-2.0.0-p648/bin/ rake
webapp 19290 0.0 2.2 1400852 184472 ? Sl 01:47 0:00 \_ /opt/rubies/ruby-2.0.0-p648/bin/ rake
webapp 19293 0.0 2.2 1400852 184488 ? Sl 01:47 0:00 \_ /opt/rubies/ruby-2.0.0-p648/bin/ rake
webapp 19333 0.0 2.2 1400852 184420 ? Sl 01:47 0:00 \_ /opt/rubies/ruby-2.0.0-p648/bin/ rake
root 21038 0.0 0.0 217276 3460 ? Ssl 01:55 0:00 PassengerWatchdog
webapp 21041 0.1 0.0 704036 5652 ? Sl 01:55 0:02 \_ PassengerHelperAgent
webapp 21047 0.0 0.0 243944 7840 ? Sl 01:55 0:00 \_ PassengerLoggingAgent
root 21056 0.0 0.0 56404 1016 ? Ss 01:55 0:00 PassengerWebHelper: master process / var/lib/passenger/standalone/4.0.60/webhelper-1.8.1-x86_64-linux/PassengerWebHelper -c /tmp/ passenger-standalone.e022jt/config -p /tmp/passenger-standalone.e022jt/
webapp 21057 0.0 0.0 56812 4436 ? S 01:55 0:00 \_ PassengerWebHelper: worker process
webapp 21058 0.0 0.0 56812 4436 ? S 01:55 0:00 \_ PassengerWebHelper: worker process
root 21063 0.0 0.0 8552 1104 ? Ss 01:55 0:00 /var/lib/passenger/standalone/4.0.60/ support-x86_64-linux/agents/TempDirToucher /tmp/passenger-standalone.e022jt --cleanup --daemonize --pid-file /tmp/passenger-standalone.e022jt/temp_dir_toucher.pid --log-f
root 21078 0.0 0.0 11600 2748 ? Ss 01:55 0:00 /bin/bash
root 21102 0.0 0.0 54764 2556 ? S 01:55 0:00 \_ su -s /bin/bash -c bundle exec sidekiq -L /var/app/current/log/sidekiq.log -P /var/app/support/pids/sidekiq.pid
root 21103 8.1 2.6 1452872 212932 ? Sl 01:55 2:27 \_ sidekiq 4.1.2 current [0 of 25 busy]
root 21118 0.0 0.0 54768 2644 ? Ss 01:55 0:00 su -s /bin/bash -c bundle exec rake sneakers:run >> /var/app/current/log/sneakers.log 2>&1 webapp
webapp 21146 0.0 0.0 9476 2336 ? Ss 01:55 0:00 \_ bash -c bundle exec rake sneakers:run >> /var/app/current/log/sneakers.log 2>&1
webapp 21147 0.6 2.3 693604 193232 ? Sl 01:55 0:11 \_ /opt/rubies/ruby-2.0.0-p648/ bin/rake
webapp 21349 0.0 2.2 1400608 184160 ? Sl 01:55 0:00 \_ /opt/rubies/ ruby-2.0.0-p648/bin/rake
webapp 21411 0.0 2.2 1400608 183812 ? Sl 01:55 0:00 \_ /opt/rubies/ ruby-2.0.0-p648/bin/rake
webapp 21414 0.0 2.2 1400608 183988 ? Sl 01:55 0:00 \_ /opt/rubies/ ruby-2.0.0-p648/bin/rake
webapp 21475 0.0 2.2 1400608 183976 ? Sl 01:55 0:00 \_ /opt/rubies/ ruby-2.0.0-p648/bin/rake
webapp 21720 0.3 3.5 1311928 293968 ? Sl 01:55 0:07 Passenger RackApp: /var/app/current
I'm not sure what spawned these processes (if it was sidekiq or sneakers or passenger). With each deploy the number seems to grow until the postgres connections are maxed out.
Is my beanstalk configuration incorrect? Can anybody help me debug this so I can figure out what's creating these processes?
Looks like every time sneakers rake task was killed, it left behind orphan processes. To rectify I added this as a pre deploy hook:
files:
"/opt/elasticbeanstalk/hooks/appdeploy/pre/04_mute_sneakers.sh":
mode: "000755"
content: |
#!/bin/bash
initctl stop sneakers 2>/dev/null
kill $(ps aux | grep '[/]opt/rubies/ruby-2.0.0-p648/bin/rake' | awk '{print $2}') 2>/dev/null
echo "Killed Sneakers Process"