DevOps and Databases

Just like with applications and other components of software engineering, databases also have a set of operational considerations and requirements. Starting from database setup, replication, backup, user access control configuration, monitoring, updates, patching, scalability, availability, resilience, disaster recovery, privacy, localization, performance, throughput, archival up to CI/CD, orchestration, and more, databases has its own laundry list of administrative demands. With the cloud, a vast majority of this list becomes automatic and fully managed by the service provider.

In this chapter, we will take a look at some of these attributes:

• Upgrades, updates, and patching
• Monitoring
• Security, privacy, and encryption
• Replication and availability
• Scalability
• Performance and throughput
• SLA, SLI, and SLO
• Data federation
• CI/CD
• Database migration service
• Systems, query, and performance insights

We will also...

Upgrades, updates, and patching

Upgrades, updates, and patching are essential for ensuring the security, performance, and availability of cloud databases. By keeping your databases up-to-date, you can help to prevent security vulnerabilities, performance degradation, and outages.

There are a number of different ways to manage upgrades, updates, and patching for cloud databases. Some of the most common methods include the following:

• Automatic updates: This means that the provider will automatically update your databases with the latest security patches and bug fixes. Google Cloud offers automatic updates for many of its databases. You can choose to have Google automatically update your databases on a schedule, or you can choose to be notified before updates are applied.
• Manual updates: If your cloud provider does not offer automatic updates, you will need to manually update your databases. This can be a time-consuming and complex process, but it gives you more control...

Security, privacy, and encryption

Security and privacy should be a major focus of any type of database design. There are some basic areas that need to be taken into account while choosing the design for databases:

• Data breaches, loss, and corruption: This includes design considerations to plan and assess your choices for data protection and back-up to prepare for factors such as hardware failure, software errors, human error, or natural disasters.
• Data privacy: This includes considerations to comply with data privacy regulations such as the General Data Protection Regulation (GDPR).
• Data security: This includes ensuring your databases and data are secure and encrypted.

Here are some tips for mitigating cloud database security and privacy concerns:

• Choose a reputable cloud provider: When choosing a cloud provider, it is important to choose one that has a good reputation for security, and is a solution that encrypts your data and dependencies in transit...

Replication and availability

Database replication is the process of copying data from one database to another. This can be done for a variety of reasons, including the following:

• To improve performance: By replicating data to multiple servers, you can distribute the load and improve performance
• To improve availability: By replicating data to multiple servers, you can ensure that your data is always available, even if one server fails
• To improve disaster recovery: By replicating data to multiple locations, you can protect your data from disasters

The primary reasons behind using data replication are to support high availability, disaster recovery, and to allow your data to scale without affecting the performance.

There are a number of different ways to replicate data for cloud databases depending on your need, some of them are listed as follows. The instance being replicated is referred to here as the primary instance and the replication copies are referred...

Scalability

Scalability is the ability of a system to handle increasing demands for resources. In the context of databases, scalability refers to the ability of a database to handle increasing demands for data storage and processing. There are two fundamental types of scalability that are vastly discussed:

• Vertical scalability: Vertical scalability is the ability of a system to handle increasing demands by increasing the capacity of its resources. For example, a database can be vertically scaled by increasing the amount of memory or storage available to it.
• Horizontal scalability: Horizontal scalability is the ability of a system to handle increasing demands by adding more nodes to the system. For example, a database can be horizontally scaled by adding more database servers to the cluster. Replication can also be considered as one of the ways of performing horizontal scaling, making the data more available within and across regions.

Simply put, horizontal scaling...

Performance and throughput

Performance and throughput are two important factors to consider when choosing a cloud database. Performance refers to how quickly the database can respond to queries, while throughput refers to how much data the database can process per unit of time.

There are a number of factors that can affect the performance and throughput of a cloud database, including the following:

• The type of database: Different types of databases are suited for different workloads. For example, relational databases are well-suited for storing and querying structured data, while NoSQL databases are well-suited for storing and querying unstructured data.
• The size of the database: The larger the database, the more resources it will require and the slower it will be.
• The number of users: The more users that are accessing the database, the more resources it will require and the slower it will be.
• The types of queries: Different types of queries can require different...

SLA, SLI, and SLO

A service-level agreement (SLA) is a contract between a cloud provider and a customer that guarantees a certain level of service. For example, an SLA might guarantee that a database will be available 99.9% of the time.

Imagine you are the owner of a business that relies heavily on a cloud-based customer relationship management (CRM) system to manage interactions with your clients. You sign an SLA with your cloud service provider, and it guarantees that the CRM system will be available and accessible to your team 99.9% of the time over the course of a year. This means that you expect no more than 8 hours and 45 minutes of downtime in a year, ensuring that your business operations run smoothly.

A service-level indicator (SLI) is a metric that is used to measure the performance of a service. For example, an SLI for a database might be the average response time for queries.

Let’s consider a video streaming platform. An SLI for such a platform could be...

Data federation

Data federation is a technique that allows you to query data from multiple sources as if it were all stored in a single database. This can be useful if you have data that is stored in different systems, or if you want to be able to query data from multiple sources without having to move it to a single database.

Federated queries are queries that are executed against multiple data sources. It can be used to join data from different sources or to perform complex queries that would be difficult or impossible to execute against a single data source.

Google Cloud databases support federated queries through the use of the BigQuery Federated Query Service. The BigQuery Federated Query Service allows you to query data from a variety of data sources, including the following:

• Google Cloud BigQuery
• Google Cloud Spanner
• Google Cloud SQL
• Amazon Redshift
• Microsoft Azure SQL Database

The BigQuery Federated Query Service is a powerful tool that...

Continuous integration/continuous delivery (CI/CD)

Continuous integration (CI) and continuous delivery (CD) are two important software development practices that can help to improve the quality and reliability of your software. CI involves automating the process of building, testing, and deploying your software. CD involves automating the process of delivering your software to production. By automating these processes, you can ensure that your software is always up-to-date and of the highest quality. You can also reduce the risk of errors and improve the speed of development.

CI/CD can help to simplify the management of cloud databases by automating the process of deploying database changes. This can help to ensure that database changes are made in a consistent and controlled manner and that they are tested before being deployed to production.

Google Cloud offers a number of CI/CD tools and services that can be used to automate the process of building, testing, and deploying...

Migrating to cloud databases

Migrating databases to the cloud has become a pivotal strategy for organizations across the globe. In an age defined by digital transformation, businesses must be agile, scalable, and resilient to remain competitive. Cloud database migration is the key to achieving these goals and ensuring the long-term success of your enterprise. With all its benefits and tangible impact on business outcomes, it’s only natural that more and more organizations and businesses are moving to cloud databases and the tools or services that they use for enabling this migration come with a lot of responsibilities.

When considering a cloud database migration service or tool, there are several important considerations and requirements to ensure a successful transition. Here’s a comprehensive list of factors to keep in mind:

• Compatibility assessment: The tool or service should provide a compatibility assessment to analyze the existing database and identify...

Database Migration Service

In this section, we’ll take a quick walkthrough of the steps involved in migrating a sample MySQL database to Cloud SQL for a MySQL database instance using Database Migration Service. We won’t go through the full implementation/testing phase of the migration. We will just look at an overview of the configuration steps:

1. Go to the Google Cloud console and make sure your project is selected, or select or create your project, as shown in the instructions here: https://cloud.google.com/resource-manager/docs/creating-managing-projects.
2. In the search bar of the console, search for Database Migration Service.
3. Enable the API if you are prompted to, as shown in the following screenshot:

Figure 8.5 – Database Migration API ENABLE screen

4. Go to the IAM page at https://console.cloud.google.com/iam-admin/iam and make sure your user account has the Database Migration admin role assigned to it, as...

System, query, and performance insights

Understanding the functioning of your database system, application, performance, cost, and security-related factors are key considerations when it comes to database operations and observability. Google Cloud provides insights on system, query, performance, cost, security, and so on. System insights allow you to track important attributes such as system usage, availability, CPU utilization, disk utilization, database load, and so on, and proactively focus on finding patterns and taking actions. Query insights provide help in the detection and diagnosis of issues in the performance of your instance by examining queries in your application. There are more such insights. These insights provide information using historical and near real-time data, and at the same time, provide actionable prescriptive alerts with regard to your services. If you can enable these for your data and application, where available, you will be able to enhance the performance...

Summary

As for any DevOps team’s dreams, the goal of this chapter has been to help you get started with understanding the fundamentals and importance of enabling organizations and teams to just focus on the business priorities and code, and leave the rest to the cloud platform. This way, DevOps teams can focus on more innovative activities that enable business growth while the platform keeps the lights on for you.

In this chapter, we have comprehensively explored the crucial intersection of DevOps practices and database operations with focused information on managing databases in the Google Cloud ecosystem. We have covered operational aspects such as maintenance, monitoring, security, privacy, encryption, replication, availability, consistency, scalability, performance, throughput, SLA, SLI, SLO, data federation, CI/ CD, migration, insights, and how to address some of these with Google Cloud databases and services.

In the upcoming chapters, we will look at data modeling...