Mastering Mesos: The ultimate guide to managing, building, and deploying large-scale clusters with Apache Mesos

Aurora: This is a service scheduler that runs on top of Mesos, enabling you to run long-running services that take advantage of the scalability, fault-tolerance, and resource isolation of Mesos.

Marathon: This is a private PaaS built on Mesos. It automatically handles hardware or software failures and ensures that an app is "always on".

Singularity: This is a scheduler (the HTTP API and web interface) for running Mesos tasks, such as long-running processes, one-off tasks, and scheduled jobs.

SSSP: This is a simple web application that provides a "Megaupload" white label to store and share files in S3.

Cray Chapel is a productive parallel programming language. The Chapel Mesos scheduler lets you run Chapel programs on Mesos.

Dark is a Python clone of Spark, a MapReduce-like framework written in Python and running on Mesos.

Exelixi is a distributed framework used to run genetic algorithms at scale.

Hadoop Running Hadoop on Mesos distributes MapReduce jobs efficiently across an entire cluster.

Hama is a distributed computing framework based on Bulk Synchronous Parallel computing techniques for massive scientific computations—for example, matrix, graph, and network algorithms.

MPI is a message-passing system designed to function on a wide variety of parallel computers.

Spark is a fast and general-purpose cluster computing system that makes parallel jobs easy to write.

Storm is a distributed real-time computation system. Storm makes it easy to reliably process unbounded streams of data, doing for real-time processing what Hadoop does for batch processing.

Chronos is a distributed job scheduler that supports complex job topologies. It can be used as a more fault-tolerant replacement for cron.

Jenkins is a continuous integration server. The Mesos-Jenkins plugin allows it to dynamically launch workers on a Mesos cluster, depending on the workload.

JobServer is a distributed job scheduler and processor that allows developers to build custom batch processing Tasklets using a point and click Web UI.

Cassandra is a performant and highly available distributed database. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data.

Elasticsearch is a distributed search engine. Mesos makes it easy for it to run and scale.

resources='cpus:24;mem:24576;disk:409600;ports:[21000-24000];bugs:{a,b,c}'

attributes='rack:abc;zone:west;os:centos5;level:10;keys:[1000-1500]'

A scalar called cpus with the value 24

A scalar called mem with the value 24576

A scalar called disk with the value 409600

A range called ports with values 21000 through 24000 (inclusive)

A set called bugs with the values a, b, and c

A rack attribute with the text value abc

A zone attribute with the text value west

An os attribute with the text value centos5

A level attribute with the scalar value 10

A keys attribute with range values 1000 through 1500 (inclusive)

Counters: This is used to measure discrete events, such as the number of finished tasks or invalid status updates. The values are always whole numbers.

Gauges: This is used to check the snapshot of a particular metric, such as the number of active frameworks or running tasks at a particular time.

registered: This can be registered via the following code:

void registered(ExecutorDriver driver,
  ExecutorInfo executorInfo,
  FrameworkInfo frameworkInfo,
  SlaveInfo slaveInfo)

This code is invoked once the executor driver is able to successfully connect with Mesos. In particular, a scheduler can pass some data to its executors through the ExecutorInfo.getData() field.

The following are the parameters:

reregistered: This can be reregistered as follows:

void reregistered(ExecutorDriver driver,
  SlaveInfo slaveInfo)

This code is invoked when the executor reregisters with a restarted slave.

The following are the parameters:

disconnected: This can be disconnected via the following code:

void disconnected(ExecutorDriver driver)

The preceding code is invoked when the executor gets "disconnected" from the slave—for example, when the slave is restarted due to an upgrade).

The following is the parameter:

launchTask: Take a look at the following code:

void launchTask(ExecutorDriver driver,
  TaskInfo task)

The preceding code is invoked when a task is launched on this executor (initiated via SchedulerDriver.launchTasks(java.util.Collection<OfferID>, java.util.Collection<TaskInfo>, Filters). Note that this task can be realized with a thread, a process, or some simple computation; however, no other callbacks will be invoked on this executor until this callback returns.

The following are the parameters:

killTask: Run the following code:

void killTask(ExecutorDriver driver,
  TaskID taskId)

This is invoked when a task running within this executor is killed via SchedulerDriver.killTask (TaskID). Note that no status update will be sent on behalf of the executor, and the executor is responsible for creating a new TaskStatus protobuf message (that is, with TASK_KILLED) and invoking ExecutorDriver.sendStatusUpdate (TaskStatus).

The following are the parameters:

frameworkMessage: Run the following code:

void frameworkMessage(ExecutorDriver driver,
  byte[] data)

This is invoked when a framework message arrives for this executor. These messages are the best effort; do not expect a framework message to be retransmitted in any reliable fashion.

The following are the parameters:

shutdown: Execute the following code:

void shutdown(ExecutorDriver driver)

This is invoked when the executor terminates all of its currently running tasks. Note that after Mesos determines that an executor has terminated, any tasks that the executor did not send Terminal status updates for (for example, TASK_KILLED, TASK_FINISHED, TASK_FAILED, and so on), and a TASK_LOST status update will be created.

The following is the parameter:

error: Run the following:

void error(ExecutorDriver driver,
  java.lang.String message)

The previous code is invoked when a fatal error occurs with the executor and/or executor driver. The driver will be aborted BEFORE invoking this callback.

The following are the parameters:

start: Run the following line:

Status start()

The preceding code starts the executor driver. This needs to be called before any other driver calls are made.

The state of the driver after the call is returned.

stop: Run the following line:

Status stop()

This stops the executor driver.

The state of the driver after the call is the return.

abort: Run the following line:

Status abort()

This aborts the driver so that no more callbacks can be made to the executor. The semantics of abort and stop are deliberately separated so that the code can detect an aborted driver (via the return status of join(); refer to the following section) and instantiate and start another driver if desired (from within the same process, although this functionality is currently not supported for executors).

The state of the driver after the call is the return.

join: Run the following:

Status join()

This waits for the driver to be stopped or aborted, possibly blocking the current thread indefinitely. The return status of this function can be used to determine whether the driver was aborted (take a look at mesos.proto for a description of status).

The state of the driver after the call is the return.

run: Take a look at the following line of code:

Status run()

This starts and immediately joins (that is, blocks) the driver.

The state of the driver after the call is the return.

sendStatusUpdate: Here's the code to execute:

Status sendStatusUpdate(TaskStatus status)

This sends a status update to the framework scheduler, retrying as necessary until an acknowledgement is received or the executor is terminated (in which case, a TASK_LOST status update will be sent). Take a look at Scheduler.statusUpdate(org.apache.mesos.SchedulerDriver, TaskStatus) for more information about status update acknowledgements.

The following is the parameter:

The state of the driver after the call is the return.

sendFrameworkMessage: Run the following code:

Status sendFrameworkMessage(byte[] data)

This sends a message to the framework scheduler. These messages are sent on a best effort basis and should not be expected to be retransmitted in any reliable fashion.

The parameters are as follows:

The state of the driver after the call is the return.

registered: This can be registered via the following code:

void registered(SchedulerDriver driver,
  FrameworkID frameworkId,
  MasterInfo masterInfo)

The preceding is invoked when the scheduler successfully registers with a Mesos master. A unique ID (generated by the master) is used to distinguish this framework from others, and MasterInfo with the IP and port of the current master are provided as arguments.

The following are the parameters:

reregistered: The preceding code can be reregistered as follows:

void reregistered(SchedulerDriver driver,
  MasterInfo masterInfo)

The preceding code is invoked when the scheduler reregisters with a newly elected Mesos master. This is only called when the scheduler is previously registered. MasterInfo containing the updated information about the elected master is provided as an argument.

The parameters are as follows:

resourceOffers: Execute the following code:

void resourceOffers(SchedulerDriver driver,
  java.util.List<Offer> offers)

The preceding code is invoked when resources are offered to this framework. A single offer will only contain resources from a single slave. Resources associated with an offer will not be reoffered to this framework until either; (a) this framework rejects these resources (refer to SchedulerDriver.launchTasks(java.util.Collection<OfferID>, java.util.Collection<TaskInfo>, Filters)), or (b) these resources are rescinded (refer to offerRescinded(org.apache.mesos.SchedulerDriver, OfferID)). Note that resources may be concurrently offered to more than one framework at a time, depending on the allocator being used. In this case, the first framework to launch tasks using these resources will be able to use them, while the other frameworks will have these resources rescinded. (Alternatively, if a framework has already launched tasks with these resources, these tasks will fail with a TASK_LOST status and a message saying as much).

The following are the parameters:

offerRescinded: Run the following code:

void offerRescinded(SchedulerDriver driver,
  OfferID offerId)

This is invoked when an offer is no longer valid (for example, the slave is lost or another framework is used resources in the offer). If, for whatever reason, an offer is never rescinded (for example, a dropped message, failing over framework, and so on), a framework that attempts to launch tasks using an invalid offer will receive a TASK_LOST status update for these tasks (take a look at resourceOffers(org.apache.mesos.SchedulerDriver, java.util.List<Offer>)).

The following are the parameters:

statusUpdate: Take a look at the following code:

void statusUpdate(SchedulerDriver driver,
  TaskStatus status)

The preceding code is invoked when the status of a task changes (for example, a slave is lost, so the task is lost; a task is finished, and an executor sends a status update saying so; and so on). If, for whatever reason, the scheduler is aborted during this callback or the process exits, then another status update will be delivered. (Note, however, that this is currently not true if the slave sending the status update is lost or fails during this time.)

The parameters are as follows:

frameworkMessage: Take a look at the following code:

void frameworkMessage(SchedulerDriver driver,
  ExecutorID executorId,
  SlaveID slaveId,
  byte[] data)

The preceding code is invoked when an executor sends a message. These messages are sent on a best effort basis and should not be expected to be retransmitted in any reliable fashion.

The parameters are as follows:

disconnected: Run the following:

void disconnected(SchedulerDriver driver)

This is invoked when the scheduler becomes disconnected from the master (for example, the master fails and another takes over).

The following is the parameter:

slaveLost: Execute the following code:

void slaveLost(SchedulerDriver driver,
  SlaveID slaveId)

This is invoked when a slave is determined unreachable (for example, machine failure or network partition). Most frameworks need to reschedule any tasks launched on this slave on a new slave.

The following are the parameters:

executorLost: Run the following:

void executorLost(SchedulerDriver driver,
  ExecutorID executorId,
  SlaveID slaveId,
  int status)

The preceding is invoked when an executor is exited or terminated. Note that any running task will have the TASK_LOST status update automatically generated.

The following are the parameters:

error: Run the following code:

void error(SchedulerDriver driver,
  java.lang.String message)

The preceding is invoked when there is an unrecoverable error in the scheduler or driver. The driver will be aborted before invoking this callback.

The following are the parameters:

start: Run the following code:

Status start()

This starts the scheduler driver. It needs to be called before any other driver calls are made.

The preceding returns the state of the driver after the call.

stop: Execute the following code:

Status stop(boolean failover)

This stops the scheduler driver. If the failover flag is set to false, it is expected that this framework will never reconnect to Mesos. So, Mesos will unregister the framework and shut down all its tasks and executors. If failover is true, all executors and tasks will remain running (for some framework-specific failover timeout), allowing the scheduler to reconnect (possibly in the same process or from a different process—for example, on a different machine).

The following is the parameter:

This returns the state of the driver after the call.

Stop: Run the following line:

Status stop()

This stops the scheduler driver assuming no failover. This will cause Mesos to unregister the framework and shut down all its tasks and executors.

This returns the state of the driver after the call.

abort: Execute the following code:

Status abort()

This aborts the driver so that no more callbacks can be made to the scheduler. The semantics of abort and stop are deliberately separated so that code can detect an aborted driver (via the return status of join(); refer to the following section) and instantiate and start another driver if desired from within the same process.

This returns the state of the driver after the call.

join: Run the following:

Status join()

This waits for the driver to be stopped or aborted, possibly blocking the current thread indefinitely. The return status of this function can be used to determine whether the driver was aborted (take a look at mesos.proto for a description of Status).

This returns the state of the driver after the call.

run: Execute the following:

Status run()

This starts and immediately joins (that is, blocks) the driver.

It returns the state of the driver after the call.

requestResources: Take a look at the following:

Status requestResources(java.util.Collection<Request> requests)

This requests resources from Mesos (take a look at mesos.proto for a description of Request and how, for example, to request resources from specific slaves). Any resources available are offered to the framework via the Scheduler.resourceOffers(org.apache.mesos.SchedulerDriver, java.util.List<Offer>) callback asynchronously.

The following is the parameter:

It returns the state of the driver after the call.

launchTasks: Use the following code:

Status launchTasks(java.util.Collection<OfferID> offerIds,
  java.util.Collection<TaskInfo> tasks,
  Filters filters)

The preceding code launches the given set of tasks on a set of offers. Resources from offers are aggregated when more than one is provided. Note that all the offers must belong to the same slave. Any resources remaining (that is, not used by the tasks or their executors) will be considered declined. The specified filters are applied on all unused resources (take a look at mesos.proto for a description of Filters). Invoking this function with an empty collection of tasks declines offers in their entirety (refer to declineOffer(OfferID, Filters)).

The following are the parameters:

It returns the state of the driver after the call.

killTask: Execute the following code:

Status killTask(TaskID taskId)

This kills the specified task. Note that attempting to kill a task is currently not reliable. If, for example, a scheduler fails over while it attempts to kill a task, it will need to retry in the future. Likewise, if unregistered/disconnected, the request will be dropped (these semantics may be changed in the future).

The following is the parameter:

It returns the state of the driver after the call.

declineOffer: Run the following code:

Status declineOffer(OfferID offerId,
  Filters filters)

This declines an offer in its entirety and applies the specified filters on the resources (take a look at mesos.proto for a description of Filters). Note that this can be done at any time, and it is not necessary to do this within the Scheduler.resourceOffers(org.apache.mesos.SchedulerDriver, java.util.List<Offer>) callback.

The following are the parameters:

It returns the state of the driver after the call.

reviveOffers: Execute the following:

Status reviveOffers()

This removes all the filters previously set by the framework (via launchTasks(java.util.Collection<OfferID>, java.util.Collection<TaskInfo>, Filters)). This enables the framework to receive offers from these filtered slaves.

It returns the state of the driver after the call.

sendFrameworkMessage: Take a look at the following:

Status sendFrameworkMessage(ExecutorID executorId,
  SlaveID slaveId,
  byte[] data)

This sends a message from the framework to one of its executors. These messages are sent on a best effort basis and should not be expected to be retransmitted in any reliable fashion.

The parameters are:

It returns the state of the driver after the call.

reconcileTasks: Take a look at the following code:

Status reconcileTasks(java.util.Collection<TaskStatus> statuses)

This allows the framework to query the status for nonterminal tasks. This causes the master to send back the latest task status for each task in statuses if possible. Tasks that are no longer known will result in a TASK_LOST update. If statuses is empty, the master will send the latest status for each task currently known.

The following are the parameters:

It returns the state of the driver after the call.

Developers get immediate access to cluster resources, whether it be to scale or introduce new services.

Developers no longer need to understand the process of requisitioning hardware or servers, and it is easier to scale up the resource requirements inside Mesos than it is to recreate servers with more or less CPUs and memory.

Hardware failures are more transparent to developers as services are automatically replaced when tasks are lost or they fail. In other words, developers are no longer paged because of a simple hardware failure.

Scheduled tasks (cron jobs) are now exposed via a web interface and are not tied to a single server, which may fail at any time, taking the cron job with it.