How can I send/use/modify data which is the result of a Sidekiq background worker of my rails app?

Question

I have a ruby on rails web application deployed on Heroku.

This web app fetches some job feeds of given URLs as XMLs. Then regulates these XMLs and creates a single XML file. It worked pretty well for a while. However, since the #of URLs and job ads increases, it does not work at all. This process sometimes takes up to 45 secs since there are over 35K job vacancies (Heroku sends timeout after 30 secs). I am having an H12 timeout error. This error led me to read this worker dynos and background processing.

I figured out that I should apply the approach below:

Scalable-approach Heroku

Now I am using Redis and Sidekiq on my project. And I am able to create a background worker to do all the dirty work. But here is my question.

Instead of doing this call in the controller class:

def apply

   send_data Aggregator.new(providers: providers).call,
             type: 'text/xml; charset=UTF-8;',
             disposition: 'attachment; filename=indeed_apply_yes.xml'
end

I am doin this perform_async call.

 def apply

    ReportWorker.perform_async(Time.now)
    redirect_to health_path   #and returns status 200 ok

  end

I implemented this class: ReportWorker calls the Aggregator Service. data_xml is the field that I need to show somewhere or be downloaded automatically when it's ready.

class ReportWorker

    include Sidekiq::Worker
    sidekiq_options retry: false 
    data_xml = nil

    def perform(start_date)

        url_one = 'https://www.examplea.com/abc/download-xml'
        url_two = 'https://www.exampleb.com/efg/download-xml'
        cursor = 'stop'
        providers = [url_one, url_two, cursor]    

        puts "SIDEKIQ WORKER GENERATING THE XML-DATA AT #{start_date}"
        data_xml = Aggregator.new(providers: providers).call
        puts "SIDEKIQ WORKER GENERATED THE XML-DATA AT #{Time.now}"     


    end 
end

I know that It's not recommended to make send_data/file methods accessible out of Controller classes. Well, any suggestions on how to do it?

Thanks in advance!!

Answer 1

Do you can set up some database on your application? And then store record about completed jobs there, also you can save the entire file in database, but i recommend some cloud storage (like amazon s3). And after that you can show current status of queued jobs on some page for user, with button 'download' after job has done