How, or best way to run while loop to avoid API limitations

Question

I am using an API request and this API has a limit of 2 calls a second and 250 records per request. That's the gist.

I have created this background job that also has the option of a second background job. This may be overkill.

Flow:

1. Order webhooks on creates from shopify
2. Cron job once per day for that days orders in case webhooks fail.

Goal for request:

If there are >= 250 records/orders in the first API request, to then create a second background job in a new worker to fetch page 2 in about 3 minutes, and if page 2 has >= 250, to then create a new background job in the same worker has page 2 (after 2 completes) 3 minutes after page 2 jobs started fetch page 3, and so on.. I use n for the page and add 1 to n if the 250 statement is true.

Cron Job:

 shops = Shop.all
  shops.map do |shop|
    if shop.company.present?
      ShopifyOrderUpdatesWorker.perform_later(shop)
    end 
  end

Background job 1: (for first API call)

      def perform(shop)
        n = 1
        orders = ShopifyAPI::Order.find(:all, params: {created_at_min: 1.day.ago.beginning_of_day.iso8601}, limit: 250, page: n )
        while (orders.count >= 250) || n == 1
          unless n =< 1 
            while n > 1 && orders.count >= 250
              orders = ShopifyAPI::Order.find(:all, params: {created_at_min: 1.day.ago.beginning_of_day.iso8601 }, limit: 250, page: n)
              #while orders.count >= 250 || n == 2
                t = 3
ShopifyOrderUpdatesLimitWorker.delay_for(t.minutes).perform_later(orders)
                n += 1 #add page to API call request
                t += +3 #add 3 minutes for each loop to buffer the api call queue to avoid api limits to be safe
              #end
            end
          end
          if n == 1
            orders.map do |order|
              #code here
            end
          end
          n += 1
        end
      end

Background job 2: (for any API call after the first)

  def perform(orders)
    orders.map do |order|
      #code here
    end
  end

This way, all shops can update "quickly" without being in queue behind other shops. Shops that have a lot of orders will wait the same time they would in either case of doing all of this in one action or in 2.

Is this overkill? Done right for code

In reality, it is probably very rare a webhook will fail so the chances of the second background job being called is slim.

Any possible improvements or suggestions for the code?

This may not be the right place to ask this question, but if anyone has experience with shopify or similar api situations, what are you doing?

Answer 1

sleep(30) if ShopifyAPI.credit_maxed?

add this in your loop