I had recently asked question regarding huge amount of connections at same time, and one of suggestions i got was that i should use BATCH operations. Thanks for that it helped a lot.
Now i have strange situation, i know what error mean and what is causing it but i have no idea how that happening in my case.
So i have 2000 entities, i split them into 500 chunks ( i know i could split on 1000 or even little more but for now i stick with 500 )
Then i proceed to create them into 4 batch operations, with is huge improvement from my previous logic of opening 2000 connections. I am using request promise library for that operations, and they are successfully created, but here thing become strange.
Even they are created successfully my request promise returning this error,
{
"name": "StatusCodeError",
"statusCode": 413,
"message": "413 - {\"error\":\"RequestEntityTooLarge\",\"description\":\"payload size: 1487638, max size supported: 1048576\"}",
"error": {
"error": "RequestEntityTooLarge",
"description": "payload size: 1487638, max size supported: 1048576"
},
"options": {
"method": "POST",
"headers": {
"Content-Type": "application/json",
"Fiware-Service": "waste4think",
"Fiware-ServicePath": "/deusto/w4t/zamudio/test"
},
"uri": "http://localhost:1026/v2/op/update?options=keyValues",
"body": {
"actionType": "UPDATE",
"entities": [ 2000 entities ]},
"json": true,
"simple": true,
"resolveWithFullResponse": false,
"transform2xxOnly": false
},
"response": {
"statusCode": 413,
"body": {
"error": "RequestEntityTooLarge",
"description": "payload size: 1487638, max size supported: 1048576"
},
"headers": {
"connection": "close",
"content-length": "100",
"content-type": "application/json",
"fiware-correlator": "48689168-6ef7-11e8-a270-0242ac110003",
"date": "Wed, 13 Jun 2018 10:48:18 GMT"
},
"request": {
"uri": {
"protocol": "http:",
"slashes": true,
"auth": null,
"host": "localhost:1026",
"port": "1026",
"hostname": "localhost",
"hash": null,
"search": "?options=keyValues",
"query": "options=keyValues",
"pathname": "/v2/op/update",
"path": "/v2/op/update?options=keyValues",
"href": "http://localhost:1026/v2/op/update?options=keyValues"
},
"method": "POST",
"headers": {
"Content-Type": "application/json",
"Fiware-Service": "waste4think",
"Fiware-ServicePath": "/deusto/w4t/zamudio/test",
"accept": "application/json",
"content-length": 1487638
}
}
}
}
I console log my flow
Sending: 0 500
Sending: 1 500
Sending: 2 500
Sending: 3 500
Sending: 4 134
[ undefined, undefined, undefined, undefined, undefined ]
As u can see it start with first, second batch etc... Undefined here is not problem since batch return only 204 when successful and i console log body instead of code...
But after final batch is completed i get that error ( entities are created ).
As u can see it show error like i put all entities in one huge batch with is not true, also this thing is shown even my entities are successfully created. Now what im missing here and why would Orion return this response but proceed to create entities in small batch operation normally?
Any suggestion would help.
Thanks
I think the problem is exactly the one that Orion is telling you:
"payload size: 1487638, max size supported: 1048576"
Thus, you are sending 1487638 payload bytes in your POST /v2/op/update while Orion allows 1MB as much as described in documentation.
Assuming that 500 entities take 1487638 (on average), then 250 should take a half, around ~700000 bytes. Given that ~700000 bytes is less than 1MB, that should suffice. In fact, it seems you last batch has 134 entities (<250 entities) so probably that one has been correctly processed by Orion.
The approach typically used in client logic in this case is to use some kind of accumulator in your code (e.g. a JSON array). You can implement a loop to fill the array, accumulating the size of each each entity. When you are close to the limit (e.e. 800KB) then you send the batch, reset the array and start with the next one.