Exercise – deploying a dummy workflow with Vertex AI Pipeline

Before we continue with the hands-on exercise, let's understand what Vertex AI Pipeline is. Vertex AI Pipeline is a tool for orchestrating ML workflows. Under the hood, it uses an open source tool called Kubeflow Pipeline. Similar to the relationship between Airflow and Cloud Composer or Hadoop and DataProc, to understand Vertex AI Pipeline, we need to be familiar with Kubeflow Pipelines.

Kubeflow Pipeline is a platform for building and deploying portable, scalable ML workflows based on Docker containers. Using containers for ML workflows is very important compared to data workflows. For example, in data workflows, it's typical to load BigQuery, GCS, and pandas libraries for all the steps. Those libraries will be used in both the upstream and downstream steps. In ML, the upstream is data loading; the other step is building models that need specific libraries such as TensorFlow or scikit-learn, while the...