Cost Strategy in GCP

This chapter will cover one of the most frequently asked questions from stakeholders – how much does a solution cost running in GCP? Each GCP service has different pricing mechanisms. In this chapter, we will look at what valuable information you will need to calculate the GCP costs.

On top of that, we will have a section dedicated to BigQuery. We will discuss the difference between two options for the BigQuery pricing models – on-demand and editions. Finally, we will revisit the two important features in BigQuery: partitioned tables and clustered tables. Understanding these features can optimize a lot of your future costs in BigQuery.

The following topics will be covered in this chapter:

• Estimating the cost of your end-to-end data solution in GCP
• Tips to optimize BigQuery using partitioned and clustered tables

Technical requirements

For this chapter’s exercises, we will use BigQuery and GCP pricing calculators from the internet, which you can open using any browser.

Estimating the cost of your end-to-end data solution in GCP

While trying out the exercises in this book, we briefly discussed the costs that might be incurred when using various GCP services. You may have already been billed for some of the resources, so you may be wondering, “How much will they cost in a full production system?” This question is important and is often asked by stakeholders.

As a data engineer, it would be great if you could estimate the end-to-end data solution cost upfront. To estimate the cost in GCP, first, we need to understand that not all GCP services use the same pricing calculation.

Note

The pricing model that’s described in this book is based on the latest information at the time of writing. Google Cloud can change the pricing model for any service at any time.

There are three types of pricing models:

Virtual Machine (VM)-based: There are GCP resources that are billed with the VMs. The bills that are generated by the machines...

Tips to optimize BigQuery using partitioned and clustered tables

BigQuery tables can store data from zero bytes to petabytes of data. There is no difference between creating a small-sized table or a large-sized table. To simplify the context and for illustration purposes only, let’s say a small-sized table ranges from KBs to 100 GB. The large-sized tables range from 100 GB to PB of data. Technically, both tables are the same, but if you think about optimizing performance and compute cost, you can configure the tables using two features called BigQuery partitioned tables and BigQuery clustered tables.

These features are helpful for both on-demand and flat-rate pricing. In on-demand pricing, the features will cut the billed bytes and will reduce the overall cost that is calculated from the billed bytes. With flat-rate pricing, it doesn’t affect the cost directly. Remember that the cost of flat-rate pricing is flat per period. However, when you’re using features...

Summary

In this chapter, we learned about two different things. First, we learned about how to estimate an end-to-end data solution cost. Second, we understood how BigQuery partitioned and clustered tables play a significant role in cost.

These two topics are usually needed by data engineers in different situations. Understanding how to calculate cost will help in the early stages of GCP implementation. This is usually a particularly important step for an organization to decide its future solution as a whole.

The second topic usually occurs when you’re designing BigQuery tables, and at a time when you need to evaluate the running BigQuery solution. Even though it’s obvious that using partitioned and clustered tables is beneficial, it’s not a surprise in a big organization, as many tables are not optimized and can be improved.

Lastly, we performed an experiment using three different tables. It proved that using partitioned and clustered tables can reduce...