You're reading from Cloud Scale Analytics with Azure Data Services

Product type Book

Published in Jul 2021

Publisher Packt

ISBN-13 9781800562936

Pages 520 pages

Edition 1st Edition

Languages

Concepts

Data Streaming

Author (1):

Patrik Borosch

Table of Contents (20) Chapters

Preface

1. Section 1: Data Warehousing and Considerations Regarding Cloud Computing

2. Chapter 1: Balancing the Benefits of Data Lakes Over Data Warehouses

3. Chapter 2: Connecting Requirements and Technology

4. Section 2: The Storage Layer

5. Chapter 3: Understanding the Data Lake Storage Layer

6. Chapter 4: Understanding Synapse SQL Pools and SQL Options

7. Section 3: Cloud-Scale Data Integration and Data Transformation

8. Chapter 5: Integrating Data into Your Modern Data Warehouse

9. Chapter 6: Using Synapse Spark Pools

10. Chapter 7: Using Databricks Spark Clusters

11. Chapter 8: Streaming Data into Your MDWH

12. Chapter 9: Integrating Azure Cognitive Services and Machine Learning

13. Chapter 10: Loading the Presentation Layer

14. Section 4: Data Presentation, Dashboarding, and Distribution

15. Chapter 11: Developing and Maintaining the Presentation Layer

16. Chapter 12: Distributing Data

17. Chapter 13: Introducing Industry Data Models

18. Chapter 14: Establishing Data Governance

19. Other Books You May Enjoy

Monitoring your Synapse Spark pools

When you're developing your Spark application, you will sometimes need to get your hands deep into the engine to dig into the details of your jobs and the environment you run them in.

To ascertain details regarding your environment, navigate to the Synapse management hub. Your first stop is the Apache Spark pools section. You will see a list of all Spark pools, and by clicking on them, you can get an overview page with information about occupied vCores, allocated memory, and active Spark applications:

Figure 6.21 – Synapse Spark pools overview

The next level of detail to investigate will then be the application itself. You will find the Applications overview in the Management pane in Synapse Studio. You will be taken to a list of all applications that are present in your Synapse environment. By clicking on the application name on the line you're interested in, you'll get to the application details...