Search code examples
jqueryjobs

fetching file from job server to my frontend server

I need some suggestions on how to fetch dynamically generated files via jobs running on my remote job server to my frontend server running a user panel to manage jobs.

I do not want my job server's IP to be exposed so allowing user to directly download from the job server is undesired.

I would also need some Jquery library to show download speed, a progress bar of some sort.

Solution

  • I think it would help to know the answers to a few questions.

    Off the top of my head:

    • How large can the files be?
    • How long is it likely to take to download a file?
    • What format are the files?
    • Are the files you're fetching from the job server static? (or, are they being regenerated each time?)

    Edit:

    jQuery UI has a progress bar. (docs)

    From your user panel you would have a link like this:

    <a href="remoteTxtFile.php?file_id=100">File 100</a>

    This link should start a jQuery.getJSON() request to remoteTxtFile.php on your frontend server, which:

    • Checks to see if the requested file has already been downloaded.
    • If it has, return a JSON object like this: {'location': '/cache/file_100.txt'}
      • The Ajax success callback can then show a link to the file.
    • If not, start downloading the file using something (ideas further down) which can output progress to a text file. Save the file to a 'cache' folder. Return a JSON object like this: {'progress':0}
    • Periodically (every second or so) fire another getJSON() to a PHP script which reads the download progress txt file and returns a JSON object like this: {'progress': 52} (percent)
    • When the download is complete, from the PHP return JSON like this: {'location': '/cache/file_100.txt'}
      • When the Ajax success callback sees 'location', the file is done, so show a link to it.

    I'm not sure what you should use for the actual downloading of files. The problem with using PHP and stream_notification_callback to do the download (and output progress) is that the entire file would be downloaded into memory, which is no good for a 1 GB file.

    Perhaps using wget to do the actual download would work better.