I am running 3 dynos in Heroku platform for my Django application. It was working properly(1.5 years). Since 2 weeks, I am getting Error R14 (Memory quota exceeded) error frequently.
What should I do in order to avoid this error? How can I monitor the problem?
2015-01-27T10:34:01.855731+00:00 app[web.3]: Starting development server at http://0.0.0.0:43181/
2015-01-27T10:34:02.042166+00:00 heroku[web.3]: State changed from starting to up
2015-01-27T10:34:15.626327+00:00 heroku[web.2]: Error R14 (Memory quota exceeded)
2015-01-27T10:34:15.626241+00:00 heroku[web.2]: Process running mem=662M(129.4%)
2015-01-27T10:34:28.151622+00:00 heroku[router]: at=info method=GET path="/api/shop/651/?format=json&&account=(null)" request_id=2d904167-3a7d-4c8c-9b2c-ae845d0fffa9 fwd="88.247.106.124" dyno=web.1 connect=0ms service=3009ms status=200 bytes=282437
2015-01-27T10:34:28.146392+00:00 app[web.1]: [27/Jan/2015 12:34:28] "GET /api/shop/651/?format=json&&account=(null) HTTP/1.1" 200 282077
2015-01-27T10:34:35.480951+00:00 heroku[web.2]: Process running mem=662M(129.4%)
2015-01-27T10:34:35.481269+00:00 heroku[web.2]: Error R14 (Memory quota exceeded)
2015-01-27T10:34:55.511625+00:00 heroku[web.2]: Process running mem=662M(129.4%)
2015-01-27T10:34:55.511625+00:00 heroku[web.2]: Error R14 (Memory quota exceeded)
These are the logs.
And guppy results:
>>> hp.setref()
>>> hp.heap()
Partition of a set of 40 objects. Total size = 6632 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 5 12 3160 48 3160 48 unicode
1 25 62 2200 33 5360 81 __builtin__.weakref
2 6 15 496 7 5856 88 list
3 1 2 488 7 6344 96 types.FrameType
4 2 5 184 3 6528 98 tuple
5 1 2 104 2 6632 100 urlparse.SplitResult
>>> hp.heap()
Partition of a set of 24479 objects. Total size = 12695072 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 7212 29 7558176 60 7558176 60 dict of tastypie.fields.CharField
1 265 1 866008 7 8424184 66 dict (no owner)
2 232 1 777664 6 9201848 72 dict of 0x7fe18acb9360
3 696 3 729408 6 9931256 78 dict of tastypie.fields.DecimalField
4 567 2 594216 5 10525472 83 dict of tastypie.fields.BooleanField
5 517 2 541816 4 11067288 87 dict of tastypie.fields.IntegerField
6 7212 29 461568 4 11528856 91 tastypie.fields.CharField
7 260 1 272480 2 11801336 93 dict of tastypie.fields.DateTimeField
8 1255 5 223952 2 12025288 95 unicode
9 53 0 96248 1 12121536 95 dict of tastypie.fields.ToManyField
It's likely you have either a memory leak, or you're running too many concurrent processes with your server. Are you using gunicorn? If so, look at your procfile and see how many workers you're running -- then lower it by one.
To track the issue, try running:
$ heroku logs --tail
On the command line to view your web logs and see when the memory errors start kicking in.