Boosting Your Confidence as a Data Engineer

In this chapter, we will review and check our understanding of all the topics that have been covered throughout this book. We will do that by simulating the Google Cloud certification’s question format. I’ll start by talking about the Google Cloud certification so that you become familiar with it.

As the last part of this book, I will share my thoughts on what the future of data engineering is. I hope that, by reaching the final section of this book, you will have all you need to have confidence as a data engineer working with Google Cloud Platform (GCP).

In this chapter, we will cover the following topics:

• Overviewing the Google Cloud certification
• Quiz – reviewing all the concepts you’ve learned about
• The past, present, and future of data engineering
• Boosting your confidence and final thoughts

Let’s get started!

Overviewing the Google Cloud certification

Let’s start this final chapter by talking about the Google Cloud certification. In my opinion, taking the certification is important. I highly recommend that you take it – not only to get the certificate on paper but also to validate how much you know about the topics that are needed to be a Google Cloud professional. The experience that you receive while preparing for the certification is the most important part of your journey. Regardless of whether you passed or failed the exam, I believe that you will gain knowledge along the way.

Google Cloud provides a list of official certifications that you can use to validate your expertise with Google Cloud technology. There are three main categories available:

• Foundational
• Associate
• Professional

Foundational is intended for anyone with no hands-on experience with Google Cloud technology. Associate is recommended for anyone who has 6+ months of Google Cloud...

Quiz – reviewing all the concepts you’ve learned about

In this section, I will give you examples of what the Google Cloud Data Engineer Certification questions look like. The 12 example questions are grouped into 5 categories.

I will list all the questions first; the key answers will be provided in the next section, after the 12th question.

Questions

Each question has four options, and you need to choose the correct answer.

Designing data processing systems

Choose one answer for each of the questions in this section:

1. Your company is implementing a data pipeline to load data from the source system to thousands of BigQuery tables. There is a new regulation that there can’t be phone number data in any datasets. As a data engineer, you need to report how many tables in BigQuery contain phone number information. You need to do it as fast as possible with minimum effort. How are you going to find the information?
   1. Develop a Dataflow script in Java...

The past, present, and future of data engineering

The data engineering practice has been there since the early internet era in the 1990s. Going back to Chapter 1, Fundamentals of Data Engineering, in the Start with knowing the roles of a data engineer section, in the past, data engineers were mostly ETL developers using specific tools. Most of these tools were proprietary tools and located on-premises. The term data engineer itself wasn’t commonplace; the more common terms used to be data modelers, database admin, and ETL developer (ETL references the proprietary ETL tool’s name). Each of the ETL tools had the necessary expertise and best practices surrounding them.

Now, in the present, data engineering has evolved into a more mature and singular role. This means that the practice is receiving a lot more common principles, concepts, and best practices. This is due to two reasons – the rapid improvement in the technologies supporting the practice and the fact...

Boosting your confidence and final thoughts

One thing that I hope you get after reading this book is more confidence – the confidence to design and develop a data pipeline and, more importantly, the confidence as a data engineer to use GCP.

I will quote the definition of the word confidence in a physiological context by the American Psychological Association: “Confidence is Self-assurance: trust in one’s abilities, capacities, and judgment.”

By following all the chapters throughout the book, you’ve learned about the important GCP services related to data engineering and tried them out in your environment by completing the hands-on exercises provided. You’ve tried 19 new technologies from GCP, including BigQuery, GCS, Cloud Composer, Dataproc, Dataflow, Pub/Sub, Cloud SQL, Dataplex, Dataform, Data Loss Prevention or Sensitive Data Protection, Looker Studio, Vertex AI, IAM, Cloud Shell, Cloud Editor, Cloud Build, Google Container Registry...

Summary

In this chapter, we conclude everything we’ve learned throughout this book. We started the chapter by overviewing the certification options from Google Cloud. There, we covered most of the GCP services that are part of the certification. But for those that weren’t, I suggest that you take a look at those services by looking at other resources.

In the next section, we looked at some quiz questions to help you gain a better understanding of the types of questions that will be asked and the format of the certification. I highly suggest that you review the other chapters if you missed some points from any of these questions.

Finally, I shared my thoughts on the past, present, and future of data engineering. Do remember that big data and the cloud are present concepts. You, as a data engineer, are and will be the center of it. Today is the best era for data engineering, and I’m 100% confident in this. I hope that you enjoyed reading this book and have...