Search code examples
message-queuesymfonytask-queue

How to design an app that does heavy tasks and show the result in the frontend (ex Google Search Console)

Let's imagine this:

  1. I have to download an XML document from an URL;
  2. I have to elaborate this document and persist its information in the database, creating or updating a lot of entites.

I think the best way is to use queues. Or maybe I can also use cronjobs.

My problem is this: if I use the same app to do the heavy tasks and also to show to the end user the results of those heavy tasks, it may happen that the heavy tasks slow down the main website.

Take a more concrete example from real life: Google Search Console (or whatever other app that does heavy tasks and shows results to the end user).

Google Search Console gets the XML map, then starts downloading each webpage, on each webpage performs a lot of analysis, then saves the results to a database and so the end user can see the errors of his website and other useful information.

So, put as hypothesis I want to build again Google Search Console as a Symfony app, which are the possible approaches?

I think that for sure I have to use queues, but the app that downloads the webpages and the app that processes these webpages and the public frontend that shows the result of these operations are the same app or are two or three separate apps?

That is, have I to create a unique application that does all these things, or I create an app to download webpages, one other to process these webpages and one other to show to the user the results?

I'm thinking a lot at this, and I'm not able to find a good design to follow.

Because my istinct is to create multiple apps for each of those tasks to make the workings but it seems that create multiple Symfony apps isn't a good choice: Symfony 2 multiple apps?

So I really don't know which path to follow: multiple apps or one big app? And if I use one big app, should have I to use cronjobs or I have anyway use queues?

Solution
    1. I will first detail the architecture :

    Architecture

    1. You can do an Service like this :

    namespace SomeBundle\Utils;

use SomeBundle\Entity\MyEntity;
use SomeBundle\myException;

class SomeService 
{
  private $var;

  public function __construct()
  {

  }

  public function doStuff()
  {
    // Do stuff
  }
}

    To debug you service, you can call it from a testController or basic Controller.

    1. You can make Command like this :

    namespace SomeBundle\Command;

use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

use SomeBundle\Utils\SomeService;

class SomeCommand extends ContainerAwareCommand 
{
  protected function configure()
  {
    $this->setName('some:command')
         ->setDescription('My awesome command');
  }

  protected function execute(InputInterface $input, OutputInterface $output)
  {
    $container = $this->getContainer();
    $sevice = new SomeService($container);
    $results = $service->doStuff();
  }
}
    1. You can do an Command Application like this :

    require __DIR__.'/vendor/autoload.php';
require_once __DIR__.'/app/AppKernel.php';

$kernel = new AppKernel('dev', true);

use SomeBundle\Command\SomeCommand;
use Symfony\Bundle\FrameworkBundle\Console\Application;

$application = new Application($kernel);
$application->add(new SomeCommand());
$application->run();

    Hope this helps !