Running local code remotely in the cloud

In previous chapters, we ran all our code in a local laptop environment, and limited our DL fine-tuning step to only three epochs due to the limited power of a laptop. This serves the purpose of getting the code running and testing quickly in a local environment but does not serve to build an actual high-performance DL model. We really need to run the fine-tuning step in a remote GPU cluster. Ideally, we should only change some configuration and still issue the MLflow run command line in a local laptop console, but the actual pipeline will be submitted to a remote cluster in the cloud. Let's see how we can do this for our DL pipeline.

Let's start with submitting code to run in a Databricks server. There are three prerequisites:

• An Enterprise Databricks server: You need to have access to an Enterprise-licensed Databricks server or a free trial version of the Databricks server (https://docs.databricks.com/getting-started/try...