I am running a background task in my django app with RQ, as it is supposed to be one of the simplest ways to get the job done. The task consists in checking in some APIs if any information has been updated and inserting any new information in my own database.
Until a few days ago, it was working fine, but I am now getting an error that I am not able to solve.
From the last lines of the error message:
OOM command not allowed when used memory > 'maxmemory'
I thought at the beginning that I was passing too much data to the worker. However, I have eventually reduced the data passed to a single dictionary with 5 key-value pairs and I am still getting the error (see the complete message at the bottom). Until last week, however, I was passing more than 20 dictionaries, each of them with more elements and it was just working fine.
I have checked here and here, but it does not seem to be the same problem as mine.
Any idea of why I am getting this error and how to solve it?
Error message:
Internal Server Error: /en/results/
Traceback (most recent call last):
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2518, in _execute_transaction
response = self.parse_response(connection, '_')
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2584, in parse_response
self, connection, command_name, **options)
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response
response = connection.read_response()
File "/my/path/lib/python3.5/site-packages/redis/connection.py", line 582, in read_response
raise response
redis.exceptions.ExecAbortError: Transaction discarded because of previous errors.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/my/path/lib/python3.5/site-packages/django/core/handlers/base.py", line 149, in get_response
response = self.process_exception_by_middleware(e, request)
File "/my/path/lib/python3.5/site-packages/django/core/handlers/base.py", line 147, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/my/path/app/results/views.py", line 259, in results_list
price_update_to_queue(outbound_trips)
File "/my/path/app/results/views.py", line 331, in price_update_to_queue
queue.enqueue(update_prices, queries)
File "/my/path/lib/python3.5/site-packages/rq/queue.py", line 274, in enqueue
job_id=job_id, at_front=at_front, meta=meta)
File "/my/path/lib/python3.5/site-packages/django_rq/queues.py", line 60, in enqueue_call
return self.original_enqueue_call(*args, **kwargs)
File "/my/path/lib/python3.5/site-packages/django_rq/queues.py", line 56, in original_enqueue_call
return super(DjangoRQ, self).enqueue_call(*args, **kwargs)
File "/my/path/lib/python3.5/site-packages/rq/queue.py", line 227, in enqueue_call
job = self.enqueue_job(job, at_front=at_front)
File "/my/path/lib/python3.5/site-packages/rq/queue.py", line 298, in enqueue_job
pipe.execute()
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2626, in execute
return execute(conn, stack, raise_on_error)
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2523, in _execute_transaction
raise errors[0][1]
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2510, in _execute_transaction
self.parse_response(connection, '_')
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 2584, in parse_response
self, connection, command_name, **options)
File "/my/path/lib/python3.5/site-packages/redis/client.py", line 585, in parse_response
response = connection.read_response()
File "/my/path/lib/python3.5/site-packages/redis/connection.py", line 582, in read_response
raise response
redis.exceptions.ResponseError: Command # 3 (HMSET b'rq:job:8df23896-d52d-4585-aa60-9f5f9a39292a' created_at 2017-06-26T13:13:33Z timeout 500 origin default data b'\x80\x04\x95\xff\x00\x00\x00\x00\x00\x00\x00(\x8c\x1bresults.tasks.update_prices\x94N]\x94}\x94(\x8c\x03arr\x94\x8c\x0bDE-CGN-00-0\x94\x8c\x03dep\x94\x8c\x0bDE-BER-02-0\x94\x8c\x03rtn\x94K\x00\x8c\x06seller\x94\x8c\x04KIWI\x94\x8c\x06dep_dt\x94\x8c\x08datetime\x94\x8c\x08datetime\x94\x93\x94C\n\x07\xe1\x07\x18\x00\x00\x00\x00\x00\x00\x94\x8c\x0bpsycopg2.tz\x94\x8c\x13FixedOffsetTimezone\x94\x93\x94KxN\x86\x94R\x94}\x94\x8c\x07_offset\x94\x8c\x08datetime\x94\x8c\ttimedelta\x94\x93\x94K\x00M \x1cK\x00\x87\x94R\x94sb\x86\x94R\x94ua\x85\x94}\x94t\x94.' description results.tasks.update_prices([{'arr': 'DE-CGN-00-0', 'dep': 'DE-BER-02-0', 'rtn': 0, 'seller': 'KIWI', 'dep_dt': datetime.datetime(2017, 7, 24, 0, 0, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=120, name=None))}]) enqueued_at 2017-06-26T13:13:33Z status queued) of pipeline caused error: OOM command not allowed when used memory > 'maxmemory'.
[26/Jun/2017 13:13:33] "GET /en/results/?From=Berlin&To=Cologne&OutboundDate=24-07-2017&ReturnDate= HTTP/1.1" 500 174583
And this is how I've been passing the tasks
#views.py
import django_rq
from .tasks import update_prices
...
queue = django_rq.get_queue('default')
queue.enqueue(update_prices, queries)
and
#tasks.py
from django_rq import job
@job
def update_prices(queries):
...
return queries_to_background(queries)
Had a similar issue before. My Flask app is deployed onto Heroku and I was using redistogo on Heroku and the memory was very little for the free plan, which caused me the exceeding maxmemory error. Then I switched to heroku-redis and the problem was solved.