Spark can run over Mesos in two modes: coarse-grained (default) and fine-grained (deprecated).
In coarse-grained mode, each Spark executor runs as a single Mesos task. Spark executors are sized according to the following configuration variables:
- Executor memory:
spark.executor.memory - Executor cores:
spark.executor.cores - Number of executors:
spark.cores.max/spark.executor.cores
Executors are brought up when the application starts, until spark.cores.max is reached. If spark.cores.max is not set, the Spark application will reserve all resources offered to it by Mesos, so it is highly recommended to set this variable in any sort of multi-tenant cluster, including those running multiple concurrent Spark applications.
The scheduler will start the executors round-robin on the offers Mesos gives it, but there are no spread guarantees, as Mesos does not provide such guarantees on the offer stream.
The benefit of the coarse-grained mode is a much lower startup overhead...
