Building your Spark job with something else

If neither sbt nor Maven suits your needs, you may decide to use another build system. Thankfully, Spark supports building a fat JAR file with all its dependencies, which makes it easy to include in the build system of your choice. Simply, run the sbt/sbt assembly in the Spark directory and copy the resulting assembly JAR file at core/target/spark-core-assembly-1.5.2.jar to your build dependencies, and you are good to go. It is more common to use the spark-assembly-1.5.2-hadoop2.6.0.jar file. These files exist in $SPARK_HOME$/lib (if users use a prebuilt version) or in $SPARK_HOME$/ assembly/target/scala-2.10/ (if users build the source code with Maven or sbt).

Tip

No matter what your build system is, you may find yourself wanting to use a patched version of the Spark libraries. In this case, you can deploy your Spark library locally. I recommend giving it a different version number to ensure that sbt/Maven picks up the modified version. You can...