Amazon Product API error 429: how to optimize performance and reliability for batch requests?

Question

I'm getting this annoying error 429 for too many requests while sending multiple requests. What's even more disgusting is it limits amount of product to 10 per request as it seems.

So I have a code that breaks down my asin array into groups of 10s and chains it into multiple requests however, when I set 1 second wait after start of every request before making a new one, it doesn't work reliably and still returns an error, increasing the number to 2 sec per request solves this but makes it too slow. (Because usually it takes 0.5 sec per request and waits the remaining 1.5 seconds).

Amazon doesn't have any documentation on how exactly those limits work so we can only guess.

Is there way to improve it further or make something different with the queuing?

$all_posts = get_posts(array(
    'posts_per_page'   => -1
));
$serialized = serialize($all_posts);
preg_match_all ( "/]([^\]]*?)\[\/asa\]/" , $serialized , $matches);
$amazon_items=$matches[1];  //here we get an array of asins

$time_end=microtime(true);
$time_start=0;

$out=array();
for ($i=0;$i<count($amazon_items);$i+=10){
    $arr=array();
    for ($j=0;$j<10&&$i+$j<count($amazon_items);$j++){
        $arr[]=$amazon_items[$i+$j];
    }
    if ($time_end-$time_start<2) {
        echo 'sleeping '.(2-($time_end-$time_start)). 'sec; ';
        sleep (2-($time_end-$time_start));
    }
    $time_start = microtime(true);
    $list = GetItems($arr);
    $time_end = microtime(true);
    echo $time_end-$time_start.' sec, ',PHP_EOL;

    $out = array_merge($out, $list); 
}

Answer 1

I think I got the solution, it seems paapi 5 is especially sensitive to the very first #1 and #2 request. There a good pause should be maintained, after which the request speed can be increased up to the nominal 1 per sec.

So the loop should go something like this: 1st request, pause to make total of 2 seconds since the start of request, 2nd request, pause to make total of 1 seconds since the start of request, 3rd request, pause to make total of 1 seconds since the start of request, 4th request, ...etc

I suspect it might also work with holding 1 whole second AFTER the end of the first request instead of 2 seconds total, before switching to 1 sec total wait loop counted from the beginning of the requests, but I am yet to try this out in order to verify. Obviously if the first request exceeds 1 second and it throws an error then I'll know this is the case.

Edit: it worked for an hour then stopped working again, I am going to change to 1 sec after every request just to be sure. This is really retarded because there is no way to tell what time exactly does it record on their server side, it is neither the starting time of the request nor the end time of the request. It seems as though it can arbitrary fluctuate between the two, and whenever 1 second rule is violated (which happens at random) it starts throwing countless 429 errors. Ok I'm going to simply wait 1 second and hope my requests are slightly faster than 1 sec themselves, which should give me some small advantage and reliability vs waiting full 2 seconds every time, like in my initially posted example.