Search code examples
apache-sparkmesosmesosphere

Mesos implementation

I have two Django websites that create a Spark Session to a Cluster which is running on Mesos.

The problem is that whatever Django starts first will create a framework and take 100% the resources permanently, it grabs them and doesn't let them go even if idle.

I am lost on how to make the two frameworks use only the neede resources and have them concurrently access the Spark cluster.

Looked into spark schedulres, dynamic resources for spark and mesos but nothing seems to work. Is it even possible or should I change the approach?

Solution

  • Self solved using dynamic allocation.