Spark SQL

In our previous chapter, we learned about DataFrames and datasets and how we can use or write custom encoders to have type-safe operations on datasets. This chapter explains the SQL component of Spark, which helps developers working on Hive or familiar with RDBMS SQL to use a similar style in Spark.

We will be covering the following topics in this chapter:

• Spark metastore
• SQL language manual
• SQL database using Java Database Connectivity (JDBC)

Spark SQL

Spark SQL is an abstraction of data using SchemaRDD, which allows you to define datasets with schema and then query datasets using SQL. To start with, you just have to type spark-sql in the Terminal with Spark installed. This will open a Spark shell for you.

Spark metastore

To store databases, table names, and schema, Spark installs a default database, metastore.db, at the same location from where you started the SQL shell.

Using the Hive metastore in Spark SQL

Spark provides the flexibility to leverage the existing Hive metastore. This will allow users to access...

Summary

In this chapter, we learned how we can connect Spark to the Hive metastore and use the Spark SQL language to perform DDL operations in Spark. Also, we went through how we can connect Spark SQL to different RDBMS datastores and query tables, which provide DataFrames as results. In the next chapter, we will be studying Spark Streaming, machine learning, and graph analysis.