I'm dealing with a frustrating bug - sometimes sidekiq doesn't start processing jobs. It's annoyingly intermittent.
When it happens, sidekiq.log shows no change (and it is running). The queue size remains at 0. I'm guessing it's getting lost in Redis.
In case it matters, I've got three environments of the same app running on the same machine (different environments), which I just put in (but the problem preceded them). Only one Redis running on the default port.
I know about redis-cli MONITOR but it spews out stuff so fast, and there's so much, that it's hard for me to figure out what is going on.
1491479154.190069 [0 127.0.0.1:38763] "brpop" "queue:default" "2"
1491479154.289507 [0 127.0.0.1:38753] "brpop" "queue:default" "2"
1491479154.289544 [0 127.0.0.1:38752] "brpop" "queue:default" "2"
/// snip... many lines of these
1491479154.364985 [0 127.0.0.1:38772] "multi"
1491479154.365090 [0 127.0.0.1:38772] "incrby" "stat:processed" "0"
1491479154.365107 [0 127.0.0.1:38772] "incrby" "stat:processed:2017-04-06" "0"
1491479154.365128 [0 127.0.0.1:38772] "incrby" "stat:failed" "0"
1491479154.365133 [0 127.0.0.1:38772] "incrby" "stat:failed:2017-04-06" "0"
1491479154.365139 [0 127.0.0.1:38772] "del" "server.name:32533:6befea0f66c0:workers"
1491479154.365149 [0 127.0.0.1:38772] "expire" "server.name:32533:6befea0f66c0:workers" "60"
1491479154.365160 [0 127.0.0.1:38772] "exec"
1491479154.365896 [0 127.0.0.1:38772] "multi"
1491479154.365962 [0 127.0.0.1:38772] "sadd" "processes" "server.name:32533:6befea0f66c0"
1491479154.365977 [0 127.0.0.1:38772] "exists" "server.name:32533:6befea0f66c0"
1491479154.365987 [0 127.0.0.1:38772] "hmset" "server.name:32533:6befea0f66c0" "info" "{\"hostname\":\"server.name\",\"started_at\":1491373877.8700902,\"pid\":2217,\"tag\":\"demo\",\"concurrency\":25,\"queues\":[\"default\"],\"labels\":[],\"identity\":\"server.name:32533:6befea0f66c0\"}" "busy" "0" "beat" "1491479154.3655496" "quiet" "false"
1491479154.366035 [0 127.0.0.1:38772] "expire" "server.name:32533:6befea0f66c0" "60"
1491479154.366046 [0 127.0.0.1:38772] "rpop" "server.name:32533:6befea0f66c0-signals"
1491479154.366058 [0 127.0.0.1:38772] "exec"
1491479154.389484 [0 127.0.0.1:38751] "brpop" "queue:default" "2"
How would one debug Redis and Sidekiq to solve this issue? I wanted to pipe with grep but I can't do it in the redis-cli interactive shell...
Ideas?
There were two instances with different environments running. As soon as I removed staging, production worked just fine. I guess it wasn't set up correctly.