Chapter 15: Designing for Logging and Monitoring

In the previous chapter, we explored the options for enabling solutions to be highly available and implementing automatic redundancy to ensure that our systems are always running and healthy.

In this chapter, we begin Section 5, Operations and Monitoring, by looking at how we ensure we are always aware of what is happening to our platform from a perspective of health, security, and costs.

Azure provides a range of options for collecting logs and metrics that we can continually monitor and alert on so that we can be proactive and also keep the systems running optimally. Some tools are enabled by default; others must be specifically activated. For larger organizations with more complex structures and requirements, how we design those systems can greatly impact the efficiency and scalability of them.

This chapter, therefore, explores the different tools available to us and what the architectural options are when using them. Specifically...

Technical requirements

This chapter will use the Azure portal (https://portal.azure.com) for examples.

Understanding logs and storage options

Monitoring and logging in Azure is a key feature of the platform and helps to drive everything from resilience and automated scaling to performance and security.

Monitoring is so important that it is enabled by default. However, this does not mean we don't need to consider monitoring in our designs – each service has its own nuances, and the platform as a whole has a number of options for how you can configure your solutions depending on your organization's needs.

As an example, we will consider two extremes. The first is a start-up company that is developing a modern containerized web service built using a microservice-based architecture. The scalability and health of the service are key, and the solution itself may be built from numerous different components, including storage, databases, service queues, and apps.

As all these components must work harmoniously together, with many interdependencies between them, we need...

Exploring monitoring tools

There are a number of tools available that help us monitor, alert, analyze, and visualize our logs. Some require additional steps to be configured, as we saw in the previous section when we configured activity logs, to also output them to a Log Analytics workspace; others are available automatically.

By default, basic logs and metrics can be visualized in the Azure portal via the Metrics and Activity Log blades either at the subscription or component level.

Activity logs

The following screenshot shows the Activity Log blade when looking at a VM:

Figure 15.7 – Azure activity log

Here we can see the most recent Azure plane events, which are useful for troubleshooting events; for example, if the VM had been shut down, the event would be listed in the Azure activity log.

Events can be filtered by clicking the menu options to change the scope of Management Group, Subscription, Event severity, and Timespan. Additional filters can be added by...

Understanding security and compliance

Security and compliance are important factors in all organizations. As we move our solutions to the cloud, we need to ensure that the resources we build do not expose data to the public network.

In addition, many organizations define governance policies that must be adhered to; sometimes these are aligned to regulatory compliance, and sometimes they match security policy.

We discussed compliance and governance in Chapter 5, Ensuring Platform Governance, and during that chapter, we explored how to create policies and initiatives that encode those needs.

Security Center is an optional addition to your toolset, and when enabled on a subscription, it creates an Azure Security benchmark initiative that gets automatically assigned. The initiative contains audit policies containing industry best practices for ensuring a secure platform.

Azure Security Center

Azure Security Center uses your own policies and the built-in policies to calculate a...

Using cost management and reporting

Using a cloud platform such as Azure provides great flexibility in costs; that is, you only pay for exactly what you need. However, with such a flexible pricing model, costs could easily spiral out of control, resulting in bill shock.

Luckily, Azure provides you with a number of tools to help keep an eye on costs and even recommends changes to your infrastructure if you're underutilizing any services.

The first of these tools is the Cost analysis blade. Select any subscription in the portal and you have a Cost analysis option under Cost Management. The following screenshot shows an example:

Figure 15.19 – Example cost analysis

The Cost analysis panel enables you to tailor your view to show a range of different options. The default view will show the current spend for your billing period and a prediction, based on currently running services, of how much your final bill will be.

The menu options provide the ability to filter on...

Summary

In this chapter, we began Section 5, Operations and Monitoring, by looking at the options available for building robust monitoring and logging solutions.

Azure provides basic capturing of metrics and logs from the platform and its components, including built-in tools for visualizing and exploring them. For more advanced scenarios, we looked at additional options for sending these logs and metrics to other locations, such as storage accounts, Log Analytics workspaces, and event hubs.

We looked at how to also hook into and capture VM operating system logs, and then how we can use tools such as Log Analytics to query across multiple data sources.

We then examined how to use Security Center, Defender, and Sentinel to provide proactive threat detection and remediation, and how each product can work together to enable automation and alerting.

Finally, we looked at the tools available for reporting on and monitoring costs.

In the next chapter, we continue our operations theme...

Exam scenario

MegaCorp Inc. is a multi-national organization spread across many different divisions, including sales and marketing, HR, and IT. Due to its size and complexity, each division within each country manages its own solutions, which are hosted in Azure, and each division has its own Azure subscription for its applications.

IT and the overall health management is controlled by a central team that has responsibility for all divisions and countries, and as part of this, a separate sub-team is responsible for monitoring and responding to security threats.

The IT team also runs proof-of-concept systems with business areas when developing new solutions, and these are often created as needed and then de-commissioned when no longer in use.

As the lead architect, you must recommend a monitoring and logging solution that addresses the following requirements:

• The security team needs overall visibility of potential threats.
• Over time, the security team wishes to implement...

Further reading

For more information on the topics covered in this chapter, refer to the following links:

• Azure Monitor: https://docs.microsoft.com/en-us/azure/azure-monitor/overview
• Kusto Query Language: https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/
• Azure Security Center: https://docs.microsoft.com/en-us/azure/security-center/
• Azure Sentinel: https://docs.microsoft.com/en-gb/azure/sentinel/
• Azure Advisor: https://docs.microsoft.com/en-us/azure/advisor/