Search code examples
amazon-ec2cronschedulermesos

EC2 Container Service vs Apache Mesos

We are looking to use Docker container to run our batch jobs in a cluster enviroment. We are evaluating to use AWS ECS Container Service/Chronos/Mesos. As far as I know, Apache Mesos has some overlapping features/purpose that EC2 has, like cluster management. Chronos is a distributed scheduler.

I am having dificult to correlate all this technologies to create a architecture!

EC2 service replace Mesos? What about the scheduler?

We are a small team will little experience in cluster development. Which stack is better for our batch processing?

EDIT I make a huge edit, and i think now i understand the architecture:

enter image description here

This is a sample picture with two cluster been managed by Mesos.

Reading the ECS Container Service documentation(http://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduling_tasks.html), AWS is on the way of integrete ECS with the Mesos Apache Framework. So I imagine that using in the future, we can use the mesos framework to manage the resources in the ECS Cluster. So it is going to be possible to use Chronos (for batch scheduling) and Marathon (for long running app.)

EDIT At this moment, we dont have distributed jobs running, like hadoop jobs or sparks jobs. Our job are much simpler, running on single instances of EC2. We are planning to use Docker to run our batch running jobs.

Solution

  • I'd argue it depends on the type of batch jobs, but the Apache Mesos ecosystem is certainly more flexible than ECS to accomodate your needs. The flexibility comes from the fact that Mesos uses a so called two-level scheduling model, which is a fancy name for it outsources the scheduling decision into frameworks (rather than trying to implement each and every existing and future workload scheduling strategy in its core, itself).

    You mentioned one such a framework already, Chronos, which is a good working horse, just maybe don't use the dependencies for jobs, ok? Then there is another great batch framework called Cook. Depending on your needs (for example SQL-based batch report generation) you could use Apache Spark. And so on and so forth.

    BTW, did I mention already that with Mesos you don't risk a vendor lock-in, while being able to deploy it, depending on your needs, fully in one cloud (such as AWS), hybrid cloud (say AWS and GCP/Azure) or on-premises?

    UPDATE: to clarify, of course Mesos has first-class Docker support.