I'm using a recursive method in Google App Engine to loop through paginated pages. There can only be one task in the queue at the same time because you can only get the next page with the current page, i.e. there's no way of knowing all pages to add to the queue at the same time.
The recursive function works fine, however, it does not abide by the instructions in the queue.yaml file.
I've created a test example that illustrations the issue. I'd expect this process to take five minutes. Reason is the queue.yaml file should only execute one task a minute and I'm sending five tasks to the queue. However, all of the tasks are executing at once.
I can hack something together with a sleep function within the test() method to slow it down, but that defeats the purpose of using the rate limiting in the queue.yaml file.
Any help would be appreciated!
main.py
@app.route('/test',methods=['POST'])
def test():
data = json.loads(request.data)
current_num,max_num = data['current_num'],data['max_num']
if current_num <= max_num:
current_num += 1
task = taskqueue.add(
queue_name='test-queue',
method='POST',
url='/test',
payload=json.dumps({'current_num':current_num,
'max_num':max_num}))
message = "current_num: {}, max_num: {}".format(current_num,max_num)
print message
return message
else:
print 'finished'
return 'finished'
@app.route('/task')
def test_task():
max_num = 5
task = taskqueue.add(
queue_name='test-queue',
method='POST',
url='/test',
payload=json.dumps({'current_num':0,'max_num':max_num})
)
print "current_num: {}, max_num: {}".format(0,max_num)
return 'submitted task'
queue.yaml
queue:
- name: test-queue
target: app.main
rate: 1/m
bucket_size: 1
max_concurrent_requests: 1
retry_parameters:
min_backoff_seconds: 5
task_retry_limit: 1
Log file:
WARNING 2018-04-18 12:24:19,824 urlfetch_stub.py:555] Stripped prohibited headers from URLFetch request: ['content-length']
current_num: 0, max_num: 5
INFO 2018-04-18 12:24:20,299 module.py:835] default: "GET /task HTTP/1.1" 200 14
WARNING 2018-04-18 12:24:20,338 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 1, max_num: 5
INFO 2018-04-18 12:24:20,350 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING 2018-04-18 12:24:20,350 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 2, max_num: 5
INFO 2018-04-18 12:24:20,362 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING 2018-04-18 12:24:20,363 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 3, max_num: 5
INFO 2018-04-18 12:24:20,380 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING 2018-04-18 12:24:20,381 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 4, max_num: 5
INFO 2018-04-18 12:24:20,399 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING 2018-04-18 12:24:20,400 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 5, max_num: 5
INFO 2018-04-18 12:24:20,415 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING 2018-04-18 12:24:20,416 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
current_num: 6, max_num: 5
INFO 2018-04-18 12:24:20,441 module.py:835] default: "POST /test HTTP/1.1" 200 26
WARNING 2018-04-18 12:24:20,441 dispatcher.py:785] Ignoring instance/version in app.main; multiple versions are not supported in devappserver.
finished
INFO 2018-04-18 12:24:20,490 module.py:835] default: "POST /test HTTP/1.1" 200 8
Notice the execution times:
2018-04-18 12:24:20,299
2018-04-18 12:24:20,350
2018-04-18 12:24:20,362
2018-04-18 12:24:20,380
2018-04-18 12:24:20,399
2018-04-18 12:24:20,415
2018-04-18 12:24:20,441
It's expected behaviour with the dev server, as described in the documentation:
The development server doesn't respect the rate and bucket-size attributes of your queues. As a result, tasks are executed as close to their ETA as possible. Setting a rate of 0 doesn't prevent tasks from being executed automatically.