You're reading from Learning Google BigQuery

Product typeBook

Published inDec 2017

Reading LevelBeginner

PublisherPackt

ISBN-139781787288591

Edition1st Edition

Languages

Python

Concepts

Database Administration

Authors (3):

Thirukkumaran Haridass

Mikhail Berlyant

Eric Brown

View More author details

Google Cloud SDK

Google Cloud Platform provides an SDK developed in Python to manage the resources in the Cloud. The framework is available for Windows, Linux, and macOS. Python 2.7 is a requisite for installing this SDK. The SDK provides command-line utilities to manage and interact with various services on Google Cloud.

The following are the three command-line utilities available in SDK:

gsutil: This is the command-line utility to interact with Google Cloud Storage
bq: This is the command-line utility to interact with Google BigQuery
gcloud: This is the command-line utility to interact with all other services on Google Cloud

Installing Google Cloud SDK

The installers are available for Windows, Linux, and macOS. Since Linux has various distributions, some manual command execution is needed for installing and configuring the Google Cloud SDK on Linux.

Installing Google Cloud SDK on Windows

Google Cloud SDK for Windows comes with a friendly installer and it also comes with an option to install Python which is a prerequisite to run the commands in Google Cloud SDK:

Download the installer from the link provided: https://cloud.google.com/sdk/docs/quickstart-windows. The installer is a GUI-based utility which will install the requisites for the SDK, and the SDK with default configuration.

In the installation wizard, choose the Bundled Python...

gsutil for Google Cloud Storage

gsutil provides options to manage files, folders, and buckets in Google Cloud Storage. The first step in moving your data to Google Cloud and Google BigQuery is to export the data and upload to Google Cloud Storage:

Manually via the browser if it is small
Automate it for basic scenarios using gsutil, which comes with Google Cloud SDK
The third option will be to use the Google Cloud Storage API to perform advanced automation

Before using the gsutil command, make sure that the project and credentials configured in the Google Cloud SDK are pointing to the project and account which you intend to use by typing the following command:

gcloud info

We will now look at the features available with gsutil:

To see the list of options provided by gsutil, type the following command:

gsutil help

The available commands section shows the command...

Using the bq utility for BigQuery

The bq command-line utility is used to interact with the Google BigQuery service on Google Cloud Platform:

Use the following command to check the version of the bq utility once the SDK is installed:

bq version

Type the following command to confirm which project the bq utility will use. The bq utility shares settings with the gcloud utility. If you wish to change the project, then run the command in the second line and choose the project you want to work on:

gcloud info
gcloud init

If an older version of Google Cloud SDK is installed on the machine, then run the bq init command to choose the default project for the bq utility to use. Use the bq help option to see the complete details of the command, its options, and its switches.

The first step in using BigQuery is to create a dataset in a project and then create tables...

Using the gcloud utility

The gcloud utility is used to interact with the rest of the services on the Google Cloud Platform, other than BigQuery and Google Cloud Storage. The commands in the gcloud utility are grouped for each service. The following are the service groups for some of the services on the Google Cloud Platform:

Service group	Google Cloud service
App	App Engine standard and flexible environment
Compute	Compute Engine to manage virtual machines
Container	Container Engine to manage containers and clusters
Dataflow	Manage Cloud Dataflow services for ETL and data processing
Dataproc	Manage Cloud Dataproc service which consists of Apache Hadoop, Spark, Pig, and Hive
Datastore	Manage Cloud Datastore service which creates entities on a NoSQL database
SQL	Manage Cloud SQL service which consists of MySQL or PostgreSQL databases...

Connecting to Cloud SQL using gcloud

There is more than one way to connect to a Cloud SQL instance from your machine, but the two most frequently used options are in the following list. The listed prerequisites for both the options require the MySQL client libraries to be installed on the local machine:

Adding the client machine IP address to authorized networks in Google Cloud Console
Installing a proxy script on the client machine and using that to connect to Google Cloud SQL

Authorizing the client machine via Google Cloud Console

Get the IP address of the client machine by opening the browser and navigating to this URL: http://ipv4.whatismyv6.com/. Note down the IP address shown on this page for your machine. Open Google...

Deploying to Google App Engine

To deploy to Google App Engine, use the app command group. The App Engine allows us to host websites developed in PHP, Java, Go, or Python in the standard environment. To deploy a sample app to the App Engine instance that was created in Chapter 1, Google Cloud and Google BigQuery, follow the steps listed here:

Download the sample PHP file and app.yaml file from the GitHub URL (https://github.com/hthirukkumaran/Learning-Google-BigQuery/tree/master/chapter2/phpapp) to a folder on the local computer. This PHP code connects to the Cloud SQL instance of the same project, executes a sample query, and displays the result.
Open the index.php file and modify the $dsn variable value to point to your Cloud SQL instance. To get the full qualified Cloud SQL instance name, open the Cloud SQL instance in your project. Copy the value of the instance connection...

Summary

This chapter covered the Google Cloud SDK and the utilities that come with it. The samples in this chapter provided an overview of how to interact with Google Cloud Services and how to manage your resources on Google Cloud Platform using the Google Cloud SDK. Use the utilities to write your batch programs that you run from your on-premises infrastructure.

The next chapter covers data types in Google BigQuery, how to use them when creating custom tables, and how to import and export the data using the bq utility that was covered in this chapter.

The rest of the chapter is locked

You have been reading a chapter from

Learning Google BigQuery

Published in: Dec 2017Publisher: PacktISBN-13: 9781787288591

A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.

undefined

Unlock this book and the full library FREE for 7 days

Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of

Start free trial

Renews at $15.99/month. Cancel anytime

Authors (3)

Thirukkumaran Haridass

Thirukkumaran Haridass currently works as a lead software engineer at Builder Homesite Inc. in Austin, Texas, USA. He has over 15 years of experience in the IT industry. He has been working on the Google Cloud Platform for more than 3 years. Haridass is responsible for the big data initiatives in his organization that help the company and its customers realize the value of their data. He has played various roles in the IT industry and worked for Fortune 500 companies in various verticals, such as retail, e-commerce, banking, automotive, and presently, real estate online marketing.
Read more about Thirukkumaran Haridass

Mikhail Berlyant

Other recommended products

Related to this chapter

Google Cloud Platform for Architects

The Google Cloud Platform is fast emerging as a leading public cloud provider. The GCP, as it is popularly known, is backed by Google’s awe-inspiring engineering expertise and infrastructure and is able to draw upon the goodwill and respect that Google has come to enjoy. The GCP is one of a handful of public cloud providers to offer the full range of cloud computing services, ranging from IaaS (Infrastructure-as-a-Service) to PaaS (Platform-as-a-Service). There is another reason the GCP is fast gaining popularity; genre-defining technologies such as TensorFlow and Kubernetes originated at Google before being open-sourced, and the GCP is a natural choice of cloud on which to run them. If you are a cloud professional today, time spent on mastering the GCP is likely to be an excellent investment.

BookJun 2018372 pages

Machine Learning with BigQuery ML

This book helps you accelerate machine learning model development with BigQuery ML. Throughout the book, you'll use various ML models to learn about BigQuery ML features and discover how to apply them to different business scenarios. This book will help you to extend existing SQL capabilities to leverage the full potential of machine learning.

BookJun 2021344 pages

Google Cloud Platform for Developers

The Google Cloud Platform provides auto-scaling compute power and distributed in-memory cache, task queues and data stores to write, build, and deploy Cloud-hosted applications. This book will help you to learn how to integrate the various services to build optimal solutions for your unique business needs using Google Cloud Platform.

BookJul 2018506 pages

Google Cloud Platform Cookbook

Google Cloud Platform is a cloud computing service that offers hosting on the same supporting technology internally used by Google for its end users. This book follows a recipe-based approach, giving you hands-on experience to make the most out of Google Cloud services.

BookApr 2018280 pages

Cloud Analytics with Google Cloud Platform

This book will deep-dive into the concept of analytics on the cloud with the design and business considerations. You will build an end-to-end analytics engine to perform smart analytics using machine learning and deep learning concepts. From ingestion to processing your data, this book contains the best practices using Google Cloud Platform.

BookApr 2018282 pages

Google Cloud Platform Administration

With Google Cloud Platform, you can build, test and deploy applications on Google’s highly reliable and scalable infrastructure. This book will help you explore a list of different developer tools available to manage and interact with the GCP platform.

BookSep 2018230 pages

Professional Cloud Architect – Google Cloud Certification Guide

This book will help you prepare for Google’s popular Professional Cloud Architect certification from the ground up. You will learn the necessary skills to design, develop, and manage enterprise-grade cloud solutions, and thereby achieve your organization’s business objectives.

BookOct 2019520 pages

Hands-On Serverless Computing

Serverless applications and architectures are gaining momentum and are increasingly being used by companies of all sizes to solve the problems of developers. This book teaches you how to quickly and securely develop applications without the hassle of configuring and maintaining infrastructure on three public cloud platforms.

BookJul 2018350 pages

Hands-On Artificial Intelligence on Google Cloud Platform

This book focuses on the use of powerful AI tools offered by Google Cloud Platform to develop and design intelligent applications on the cloud. You will start with topics that set the foundation for using GCP with various powerful libraries, and then move on to building end to end AI applications using them.

BookMar 2020350 pages

Hands-On Machine Learning on Google Cloud Platform

In this book, you will learn how to create powerful machine learning based applications for a wide variety of problems leveraging different data services from the Google Cloud Platform. Finally, you will know the main difficulties that you may encounter and get appropriate strategies to overcome these difficulties and build efficient systems.

BookApr 2018500 pages

Architecting Google Cloud Solutions

Google Cloud is a powerful and highly scalable cloud platform that has seen rising demand and interest from enterprises seeking digital transformation and looking to modernize their workloads. This book is a comprehensive introduction to solution architecture with Google Cloud that will have you up to speed in no time.

BookApr 2021472 pages

Personalised recommendations for you

Based on your interests and search pattern

Et al.

Ever wonder why speech recognition systems don't understand the Scottish accent, or what would happen if an astronaut only ate mac 'n' cheese, or other spurious reflections you'd have at a bar? We did, then collated those deliberations into absurd research articles with fake figures and methodologies inspired by even more fictionally absurd studies.

BookAug 2023230 pages5

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages4

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages5

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages1

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages5

Mastering Tableau 2023

This book is a comprehensive resource to mastering your Tableau skills and becoming a BI expert. As you progress, you will learn how to build advanced dashboards and improve your storytelling to derive key business insight, as well as make you well-versed with advanced functionalities of Tableau in the business intelligence domain.

BookAug 2023684 pages

Building AI Applications with ChatGPT APIs

This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.

BookSep 2023258 pages5

Building AI Applications with ChatGPT APIs

This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.

BookSep 2023258 pages2

Data Engineering with AWS

Embark on a journey to master data engineering pipelines on AWS! Our book offers a hands-on experience of AWS services for ingesting, transforming, and consuming data. Whether you're an absolute beginner or someone with basic data engineering experience, this guide is an indispensable resource.

BookOct 2023636 pages5

Modern Data Architecture on AWS

Every organization wants an agile, performant, and cost-effective data platform that meets all their current and future business needs. Purpose-built AWS analytics services and their features play a big part in building such a modern data platform. This book brings to you all the design and architectural patterns that’ll help you achieve this goal.

BookAug 2023420 pages5

Practical Guide to Applied Conformal Prediction in Python

Discover the power of Conformal Prediction with the "Practical Guide to Applied Conformal Prediction in Python." Master the latest techniques to quantify uncertainty in machine learning and computer vision models, and seamlessly apply them to your industry applications.

BookDec 2023240 pages

TinyML Cookbook

With over 70 project-based recipes, the TinyML Cookbook is a practical guide that will help you to get the most out of your microcontrollers. It provides a comprehensive understanding of the theoretical foundations while giving you hands-on experience training ML models for deployment on Arduino Nano 33 BLE Sense, Raspberry Pi Pico, and SparkFun RedBoard Artemis Nano microcontrollers.

BookNov 2023664 pages