In this introductory chapter, you will learn how to set up and run TensorFlow Enterprise in a Google Cloud Platform (GCP) environment. This will enable you to get some initial hands-on experience of how TensorFlow Enterprise integrates with other services in GCP. One of the most important improvements in TensorFlow Enterprise is the integration with the data storage options in Google Cloud, such as Google Cloud Storage and BigQuery.

This chapter starts by covering how to complete a one-time setup for the cloud environment and enable the necessary cloud service APIs. Then we will see how easy it is to work with these data storage systems at scale.

In this chapter, we'll cover the following topics:

• Understanding TensorFlow Enterprise
• Configuring cloud environments for TensorFlow Enterprise
• Accessing the data sources

TensorFlow has become an ecosystem consisting of many valuable assets. At the core of its popularity and versatility is a comprehensive machine learning library and model templates that evolve quickly with new features and capabilities. This popularity comes at a cost, and that cost is expressed as complexity, intricate dependencies, and API updates or deprecation timelines that can easily break the models and workflow that were laboriously built not too long ago. It is one thing to learn and use the latest improvement in your code as you build a model to experiment with your ideas and hypotheses, but it is quite another if your job is to build a model for long-term production use, maintenance, and support.

Another problem associated with early TensorFlow in general concerned its code debugging process. In TensorFlow 1, lazy execution makes it rather tricky to test or debug your code because the code is not executed unless it is wrapped in a session...

Assuming you have a Google Cloud account already set up with a billing method, before you can start using TensorFlow Enterprise, there are some one-time setup steps that you must complete in Google Cloud. This setup consists of the following steps:

1. Create a cloud project and enable billing.
2. Create a Google Cloud Storage bucket.
3. Enable the necessary APIs.

The following are some quick instructions for these steps.

Setting up a cloud environment

Now we are going to take a look at what we need to set up in Google Cloud before we can start using TensorFlow Enterprise. These setups are needed so that essential Google Cloud services can integrate seamlessly into the user tenant. For example, the project ID is used to enable resource creation credentials and access for different services when working with data in the TensorFlow workflow. And by virtue of the project ID, you can read and write data into your...

We will use a simple example of putting data stored in a Google Cloud bucket into a table that can be queried by BigQuery. The easiest way to do so is to use the BigQuery UI. Make sure it is in the right project. We will use this example to create a dataset that contains one table.

You can navigate to BigQuery by searching for it in the search bar of the GCP portal, as in the following screenshot:

Figure 1.13 – Searching for BigQuery

You will see BigQuery being suggested. Click on it and it will take you to the BigQuery portal:

Figure 1.14 – BigQuery and the data warehouse query portal

Here are the steps to create a persistent table in the BigQuery data warehouse:

1. Select Create dataset:

   Figure 1.15 – Creating a dataset for the project

2. Make sure you are in the dataset that you just created. Now click CREATE TABLE:

   Figure 1.16 – Creating a table for the dataset

   In the...

In this section, we are going to see firsthand how easy it is to access data stored in one of the Google Cloud Storage options, such as a storage bucket or BigQuery. To do so, we need to configure an environment to execute some example TensorFlow API code and command-line tools in this section. The easiest way to use TensorFlow Enterprise is through the AI Platform Notebook in Google Cloud:

1. In the GCP portal, search for AI Platform.
2. Then select NEW INSTANCE, with TensorFlow Enterprise 2.3 and Without GPUs. Then click OPEN JUPYTERLAB:

   Figure 1.21 – The Google Cloud AI Platform and instance creation

3. Click on Python 3, and it will provide a new notebook to execute the remainder of this chapter's examples:

Figure 1.22 – A JupyterLab environment hosted by AI Platform

An instance of TensorFlow Enterprise running on AI Platform is now ready for use. Next, we are going to use this platform...

TensorFlow Enterprise can easily access data sources in Google Cloud Storage as well as BigQuery. Either of these data sources can easily host gigabytes to terabytes of data. Reading training data into the JupyterLab runtime at this magnitude of size is definitely out of question, however. Therefore, streaming data as batches through training is the way to handle data ingestion. The tf.data API is the way to build a data ingestion pipeline that aggregates data from files in a distributed system. After this step, the data object can go through transformation steps and evolve into a new data object for training.

In this section, we are going to learn basic coding patterns for the following tasks:

• Reading data from a Cloud Storage bucket
• Reading data from a BigQuery table
• Writing data into a Cloud Storage bucket
• Writing data into BigQuery table

After this, you will have a good grasp of reading and writing data to a Google Cloud...

This chapter provided a broad overview of the TensorFlow Enterprise environment hosted by Google Cloud AI Platform. We also saw how this platform seamlessly integrates specific tools such as command-line APIs to facilitate the easy transfer of data or objects between the JupyterLab environment and our storage solutions. These tools make it easy to access data stored in BigQuery or in storage buckets, which are the two most commonly used data sources in TensorFlow.

In the next chapter, we will take a closer look at the three ways available in AI Platform to use TensorFlow Enterprise: the Notebook, Deep Learning VM, and Deep Learning Containers.