Chapter 12: Boosting Your Confidence as a Data Engineer

In this chapter, we will review and check our understanding of all the topics that have been covered throughout this book. We will do that by simulating the Google Cloud certification's question format. I'll start by talking about the Google Cloud certification so that you become familiar with it.

As the last part of this book, I will share my thoughts on what will be the future of Data Engineering. I hope that, by reaching the final section of this book, you will get all you need to have confidence as a data engineer working with Google Cloud Platform (GCP).

In this chapter, we will cover the following topics:

• Overviewing the Google Cloud certification
• Quiz – reviewing all the concepts you've learned about
• The past, present, and future of Data Engineering
• Boosting your confidence and final thoughts

Let's get started!

Overviewing the Google Cloud certification

Let's start this final chapter by talking about the Google Cloud certification. In my opinion, taking the certification is important. I highly recommend that you take it – not only to get the certificates on paper but also to validate how much you know about the topics that are needed to be a Google Cloud professional. The experience that you get while preparing for the certification is the most important part of your journey. Regardless of whether you pass or fail the exam, I believe that you will get new knowledge along the way.

Google Cloud provides a list of official certifications that you can use to validate your expertise with the Google Cloud technology. There are three main categories available:

• Foundational
• Associate
• Professional

Foundational is intended for anyone with no hands-on experience with Google Cloud technology. Associate is recommended for anyone who has 6+ months of Google Cloud...

Quiz – reviewing all the concepts you've learned about

In this section, I will give you examples of what the Google Cloud Data Engineer certification's questions look like. There are 12 questions in the exam that are grouped into 4 categories.

I will list all the questions first; the key answers will be provided in the next section, after the twelfth question.

Questions

Each question has four options and you need to choose the correct answer.

Designing data processing systems

Please choose one answer for each of the questions in this section:

1. Your company runs a variety of Spark jobs on an on-premises cluster. You are planning to migrate to GCP quickly and easily. What kind of deployment would you recommend?
   1. Deploy a Cloud Dataproc cluster to run the Spark jobs.
   2. Run Spark and Hadoop jobs in Cloud Dataflow.
   3. Migrate HDFS files to Cloud Storage. Load data from Cloud Storage to BigQuery. Convert all the Spark jobs into BigQuery SQL.
   4. Run new jobs on...

The past, present, and future of Data Engineering

The Data Engineering practice has been around since the early internet era in the 1990s. Going back to Chapter 1, Fundamentals of Data Engineering, in the past, data engineers were mostly ETL developers using specific tools. Most of these tools were proprietary tools and located on-premises. The term data engineers itself didn't exist; the more common terms used to be data modelers, database admin, and ETL developer (ETL references the proprietary ETL tool's name). Each of the ETL tools had the necessary expertise and best practices surrounding them.

Now, in the present, Data Engineering has evolved into a more mature and singular role. This means that the practice has a lot more common principles, concepts, and best practices. This is due to two reasons – the rapid improvement in the technologies supporting the practice and the fact that Data Engineering has become a critical and central role to organizations.

...

Boosting your confidence and final thoughts

One thing that I hope you have after reading this book is more confidence – the confidence to design and develop a data pipeline and, more importantly, the confidence as a data engineer using Google Cloud Platform.

I will quote the definition of the word confidence from the physiological aspect by the American Psychological Association:

"Confidence is Self-assurance: trust in one's abilities, capacities, and judgment."

By following all the chapters throughout the book, you've learned about the important GCP services related to Data Engineering and tried them out in your environment by completing the hands-on exercises provided. You've tried 19 new technologies from GCP, including BigQuery, Google Cloud Storage, Cloud Composer, DataProc, DataFlow, Pub/Sub, CloudSQL, Google Data Studio, Vertex AI pipeline, Vision AI, Translate AI, AutoML, IAM, Cloud Shell, Cloud Editor, Cloud Build, Google Container...

Summary

In this chapter, we concluded everything we've learned throughout this book. We started the chapter by overviewing the certification options from Google Cloud. There, we covered most of the GCP services that are part of the certification. But for those that weren't covered, I suggest that you take a look at those services by looking at other resources.

In the next section, we looked at some quiz questions to help you gain a better understanding of the types of questions that will be asked and the format of the certification. I highly suggest that you review the other chapters if you missed some points from any of these questions.

Finally, I shared my thoughts on the past, present, and future of Data Engineering. Do remember that big data and the cloud are the present. You, as a data engineer, are and will be the center of it. Today is the best era for Data Engineering and I'm 100% confident in this. I hope that you enjoyed reading this book and have a...

Why subscribe?

• Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals
• Improve your learning with Skill Plans built especially for you
• Get a free eBook or video every month
• Fully searchable for easy access to vital information
• Copy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [e-mail] for more details.

At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.