I'm using Sidekiq 4.1.2 as background job processor on Heroku (with Heroku Redis 3.2.0).
Problem is, Sidekiq process consumes more memory with every minute and never goes down. Sometimes it takes just 10 minutes to grow from 300Mb to 2Gb (then it dies, horribly). Of course, my first thought was "Oh, looks like I've screwed up my background job's memory management!".
The nature of my background jobs is that they take a long time to complete, and they create a lot of objects, which are saved to DB and not used again. So I've added periodic manual garbage collector calls: GC.start after every 1000 objects processed.
I tested it on my local machine and was relieved: Sidekiq process with 10 threads was staying under 500Mb. Then I deployed this code on Heroku, but, alas, Sidekiq's insatiable hunger for memory was still in place: memory consumption only increases and never decreases, despite periodic manual starts of garbage collector.
Therefore, problem is following: same code does not leak on my machine, but does leak on Heroku. What may cause such behavior? What should I do to pinpoint it?
It was one of the gems in the :production group. I was using it for profiling, and looks like it did a poor job of it. I've replaced it with a similar gem and now there's no memory leak.