SAP HANA

SAP released its new SAP HANA in-memory database in 2010. It was initially supported for use with SAP BW as an additional DBMS alongside the existing supported AnyDB, and over the years support has been extended to most SAP Business Suite applications. In 2015 SAP announced its next generation ERP system called SAP S/4HANA, which only supports the SAP HANA database as the data storage and processing layer, and for which some of the code has been reimplemented to take maximum benefit from SAP HANA's in-memory capabilities. To ensure the best performance, SAP works with hyperscalers to provide certified reference architectures that should be adjusted and implemented by customers. In the next section you'll understand the best practices to deploy SAP HANA on Azure.

Supported platforms

SAP HANA is a certified workload in Microsoft Azure and runs on virtual machines or on bare-metal Azure HANA Large Instance (HLI) servers, which are designed for the largest HANA deployments that require massive amounts of memory; up to 24 TB scale-up, and 60 TB scale-out.

All database revisions starting from SAP HANA 1.0 SP12 are supported, including all releases of SAP HANA 2.0. Customers can choose to use either Red Hat Enterprise Linux or SUSE Linux Enterprise Server operating systems. Both vendors publish OS images for SAP in the Azure Marketplace.

SAP HANA can be used as a standalone database but is more usually deployed to support existing SAP Business Suite applications – ECC, BW, CRM, and SRM – or be used with the latest generation applications such as SAP S/4HANA and SAP BW/4HANA.

SAP HANA is generally deployed as a single-node scale-up solution for OLTP applications, while for OLAP applications such as SAP BW or SAP BW/4HANA, it can be deployed as a scale-out solution. SAP HANA scale-out solutions can be deployed on both HLI and VMs.

Why SAP HANA certified platforms?

SAP HANA database performance is highly dependent on the underlying hardware. SAP publishes a long list of hardware and network requirements that ensure that the database can efficiently process massive amounts of data. SAP provides the SAP HANA Hardware and Cloud Measurement Tools (HCMTs) that vendors must use to validate that their platform meets the required key performance indicators (KPIs).

Microsoft is one of the hyperscale cloud vendors that has successfully certified a wide range of virtual machines and bare-metal servers to run SAP HANA. It ensures all hardware components are compatible and offer the required performance.

However, the requirements to achieve certification are very stringent – some might say onerous – and for non-production environments, it may not always be necessary to choose a fully certified platform. You may require one non-production environment to be fully certified, to allow problem investigation away from production, but for other environments you may wish to choose a more cost-conscious solution. There are two main ways to reduce costs when running HANA on Azure:

• Use non-certified VMs: There are a range of Ev3 series VMs that are adequate to run HANA, are available in smaller sizes than the M series (64 GiB, 128 GiB), and where they offer similar memory to the M series, they are generally cheaper.
• Cost-conscious storage configuration³³: Microsoft has published suggested storage configurations that will provide adequate performance for non-production HANA, but are known not to pass the HCMT KPIs.

If you are currently running SAP HANA on VMware in your on-premises environment, then it is highly likely are you already running cost-conscious solutions in non-production.

SAP HANA sizing

SAP HANA has some very specific requirements regarding CPU, memory, networking, and storage. The rules for these are defined by SAP in order to ensure that a HANA database will achieve its optimum performance. As we will see later, while this is critical for production HANA databases, you may want to consider more cost-optimized solutions for at least some of your non-production environments.

CPU and memory requirements

SAP HANA database sizing is primarily focused on the amount of memory required. Originally, there was no CPU sizing (SAPS) as such as SAP simply use a fixed CPU to memory ratio for HANA, or more accurately two ratios.

One ratio for OLTP systems and a second for OLAP systems, with the OLTP systems allowed twice the memory per CPU when compared with the OLAP systems. This was based on the logic that in general OLAP systems use more CPU power to process their work for a given volume of data.

As with sizing SAP NetWeaver systems, sizing SAP HANA based systems depends on whether this is a new installation or a migration of an existing application to SAP HANA. For new installations, you can use SAP Quick Sizer and SAP Sizing Guidelines, the same as for SAP NetWeaver-based systems. When using SAP Quick Sizer, it is worth noting that there are two versions, one Classic for sizing traditional NetWeaver applications on AnyDB, and a new HANA version, for sizing applications on HANA.

Where you are running existing SAP NetWeaver-based systems and want to migrate them to HANA, then SAP provides two sizing reports that you can run to get an estimate of the HANA database sizing. These sizing reports are published via SAP notes as follows:

• 2610534 – HANA BW Sizing Report (/SDF/HANA_BW_SIZING)
• 1872170 – ABAP on HANA sizing report (S/4HANA, Suite on HANA)

Each of these reports will provide a detailed estimate of the sizing of your existing SAP database when migrated to HANA. It is important to note that these reports do not provide any sizing for the application tier, and in general the starting point for this is to perform a reference sizing for the application tier based on the existing deployed application servers, as described in the general sizing section earlier.

To obtain a rough order of magnitude sizing without running these reports, or where you are currently not in a position to run the reports, it is possible to manually estimate the sizing as well. Assuming that the system is well maintained, and the current database is not compressed, then the recommendation from SAP Note 1793345 Sizing for SAP Suite on HANA can be used. These state that you should:

• Take half of size of your disk-based database
• Add a safety buffer of 20%
• Add 50 GB fixed size for code, stack, and other services

This means that if the database is currently approximately 2000 GB in size (tables plus indexes), the maximum memory consumption will be 1250 GB (2000 GB/2 * 1.2 + 50 GB).

Based on the required memory, customers can choose a certified platform to run the database workload in Azure. Currently, Microsoft offers 12 VM sizes and 12 bare-metal servers that are sized to cover memory requirements of small or medium companies or large enterprises. An up-to-date list is available on the SAP-Certified and Supported SAP HANA Hardware Directory³⁴:

VM Size	Memory	Scale-Out (Clustering)	Certified workload
DS14v2	112 GiB	No	SAP Business One
E64sv3	432 GiB	No	OLTP / OLAP
M32ts	192 GiB	No	OLTP / OLAP SAP Business One
M32ls	256 GiB	No	OLTP / OLAP SAP Business One
M64ls	512 GiB	No	OLTP / OLAP SAP Business One
M64s	1000 GiB	No	OLTP / OLAP SAP Business One
M64ms	1792 GiB	No	OLTP / OLAP
M128s	2000 GiB	Yes	OLTP / OLAP
M128ms	3800 GiB	No	OLTP / OLAP
M208s_v2	2850 GiB	No	OLTP / OLAP
M208ms_v2	5700 GiB	No	OLTP / OLAP
M416s_v2	5700 GiB	No	OLTP / OLAP
M416ms_v2	11400 GiB	No	OLTP / OLAP

Table 2-4: SAP HANA certified Azure VMs

HLI Size	Memory	Scale-Out (Clustering)	Certified workload
S144	1.5 TiB	Yes	OLTP / OLAP
S144m	3 TiB	No	OLTP / OLAP
S192	2 TiB	Yes	OLTP / OLAP
S192m	4 TiB	No	OLTP
S384	4 TiB	Yes	OLTP / OLAP
S384m	6 TiB	No	OLTP
S384xm	8 TiB	Yes	OLTP / OLAP
S576m	12 TiB	Yes	OLTP
S72	768 GiB	No	OLTP / OLAP
S72m	1.5 TiB	No	OLTP
S768m	16 TiB	No	OLTP
S960m	20 TiB	No	OLTP

Table 2-5: SAP-certified Azure HANA large instances

Scale-out deployment is only allowed for OLAP workloads. Scale-out for S/4HANA is technically supported by SAP, but not currently supported on Azure.

During the annual SAP SAPPHIRE NOW conference in May 2019, Microsoft announced two new innovations for SAP HANA:

• Availability of VMs with up to 6 TB of memory imminently, and availability of VMs with 12 TB of memory in Q3 CY2019.
• A plan to launch large bare-metal Azure HANA Large instances using Intel Optane memory, including a 4-socket 9 TB instance and an 8-socket 18 TB instance. These instances will enable customers to benefit from faster load times for SAP HANA data in case of a restart and a reduced TCO when compared with current HANA instances that use only Dynamic Random Access Memory (DRAM).

Network requirements

In the SAP HANA Network Requirements document SAP distinguish three network zones:

1. Client Zone: Client applications that connects to database and execute SQL or MDX queries. In addition, it includes the HTTP communication when using the SAP HANA XS engine.
2. Internal Zone: Internal communication between SAP HANA instances, for example in a System Replication scenario or in Scale-Out deployment.
3. Storage Zone: For accessing data and log files that are stored on SAN or NAS.

SAP NetWeaver-based systems are latency sensitive. Following the SAP Note 2543171 – Latency issue between application server and Database the response time between application server and the database should be lower than 0.7 ms. As described earlier this can be achieved in Azure by using PPG.

Storage requirements

For HANA VMs, you must build the SAP HANA storage configuration manually based on the four disk types available in Azure:

• Standard HDD: A storage solution based on HDD disks, should not be used with SAP HANA.
• Standard SSD: A storage solution based on SSD disks, offers similar throughput as the Standard HDD storage, but improves the data access latency. Not used for production HANA instances but can be used as part of cost-conscious storage configuration for non-production environments if you are not concerned about having a certified configuration.
• Premium SSD: A solution based on high performance SSD, and the recommended storage for M series and Mv2 series VMs when used for SAP HANA and combined with the Write Accelerator.
• Ultra SSD: A solution based on ultra performance SSD, the only supported storage for Ev3 series VMs, and gradually being supported on M series and Mv2 series.

Azure will fulfil the SAP storage performance, throughput, and latency requirements when you follow the recommended storage configuration, based on Premium SSD disks and the Write Accelerator³⁵ – a feature of M series and Mv2 series virtual machines that significantly reduces the latency when writing to the HANA log files. It is essential to realize that the Write Accelerator is not automatically enabled, and you need to ensure that this is enabled manually. One of the most common faults found during pre-GoLive checks for SAP HANA on Azure is that the Write Accelerator has not been enabled.

The table below presents a verified storage layout optimized to run SAP HANA on Azure VMs using premium disks:

VM SKU	RAM (GiB)	Max. VM I/O Throughput (Mbps)	/hana /data	/hana /log	/hana /shared	/root volume	/usr /sap	/hana /backup
M32ts	192	500	3 x P20	2 x P20	1 x P20	1 x P6	1 x P6	1 x P20
M32ls	256	500	3 x P20	2 x P20	1 x P20	1 x P6	1 x P6	1 x P20
M64ls	512	1000	3 x P20	2 x P20	1 x P20	1 x P6	1 x P6	1 x P30
M64s	1000	1000	4 x P20	2 x P20	1 x P30	1 x P6	1 x P6	2 x P30
M64ms	1750	1000	3 x P30	2 x P20	1 x P30	1 x P6	1 x P6	3 x P30
M128s	2000	2000	3 x P30	2 x P20	1 x P30	1 x P10	1 x P6	2 x P40
M128ms	3800	2000	5 x P30	2 x P20	1 x P30	1 x P10	1 x P6	4 x P40
M208s_v2	2850	1000	4 x P30	2 x P20	1 x P30	1 x P10	1 x P6	3 x P40
M208ms_v2	5700	1000	4 x P40	2 x P20	1 x P30	1 x P10	1 x P6	3 x P50
M416s_v2	5700	2000	4 x P40	2 x P20	1 x P30	1 x P10	1 x P6	3 x P50
M416ms_v2	11400	2000	8 x P40	2 x P20	1 x P30	1 x P10	1 x P6	4 x P50

Table 2-6: SAP HANA VM storage layouts using Azure premium SSD

Customers can also implement the recently released Ultra SSD disk, which is already certified for SAP HANA on E64s_v3 and will be certified in the future for further VMs. Certification of the Ev3 series was not possible before as the Write Accelerator feature was only available for M series VMs, as it required a specific hardware feature. Ultra SSDs are not yet available in all Azure regions.

Ultra SSD performance can be customized based on throughput and IOPS requirement. The table below summarize the storage layout for virtual machines using Ultra SSDs:

VM SKU	RAM (GiB)	Max. VM I/O Throughput (Mbps)	/hana /data volume (GB)	/hana /data throughput (Mbps)	/hana /data IOPS	/hana /log volume (GB)	/hana /log throughput (Mbps)	/hana /log IOPS
E64s_v3	432	1200	600	700	7500	512	500	2000
M32ts	192	500	250	500	7500	256	500	2000
M32ls	256	500	300	500	7500	256	500	2000
M64ls	512	1000	600	500	7500	512	500	2000
M64s	1000	1000	1200	500	7500	512	500	2000
M64ms	1750	1000	2100	500	7500	512	500	2000
M128s	2000	2000	2400	1200	9000	512	800	2000
M128ms	3800	2000	4800	1200	9000	512	800	2000
M208s_v2	2850	1000	3500	1000	9000	512	500	2000
M208ms_v2	5700	1000	7200	1000	9000	512	500	2000
M416s_v2	5700	2000	7200	1500	9000	512	800	2000
M416ms_v2	11400	2000	14400	1500	9000	512	800	2000

Table 2-7: SAP HANA VM storage layouts using Azure ultra SSD

Since November 2019, ANF is also now certified as a storage option for SAP HANA running on Azure VMs. In order to use ANF for SAP HANA storage, you need to be aware of the per node throughput requirements:

• Enable read/write on /hana/log of a 250 Mbps with 1 MB I/O sizes
• Enable read activity of at least 400 Mbps for /hana/data for 16 MB and 64 MB I/O sizes
• Enable write activity of at least 250 Mbps for /hana/data with 16 MB and 64 MB I/O sizes

The ANF throughput limits per 1 TiB of volume quota are:

• Premium Storage Tier = 64 MiB/s
• Ultra Storage Tier = 128 MiB/s

To fulfil the SAP HANA requirements the minimal volume should be:

Volume	Size Premium Storage tier	Size Ultra Storage tier	Supported NFS Protocol
/hana/log/	4 TiB	2 TiB	v4.1
/hana/data	6.3 TiB	3.2 TiB	v4.1
/hana/shared	Max (512 GB, 1xRAM) per 4 worker nodes	Max (512 GB, 1xRAM) per 4 worker nodes	v3 or v4.1

Please note that these are the minimal requirements. The recommended values depend on the number of hosts and their sizes. For a landscape with one master node, two workers, and one standby deployed on the M128s VM (with 2 TB of memory), you should consider at least:

Volume	Size Premium Storage Tier	Size Ultra Storage Tier
/hana/log	12 TiB	6 TiB
/hana/data	18.9 TiB	9.6 TiB
/hana/shared	2 TiB	2 TiB

The preceding configurations are the minimum storage configurations required to run SAP HANA on these VMs, and in most cases will be sufficient. However, should you enable some of the data tiering capabilities that SAP are starting to roll out, which do not preload all data into memory, then over time the size of the database on disk may continue to grow, without the need for more memory. In this case, for example, you could require more capacity for /hana/data because the size on disk is continuing to grow.

In order to decrease the cost of storage, customers may wish to adjust the storage configuration for non-production environments. While these storage configurations will not meet all the storage KPIs required to pass the HCMT tests published by SAP, and won't be officially supported, they still perform well and will save costs for non-production workloads. Microsoft has published these cost-conscious storage requirements that provide a sensible balance between performance and cost, by using a mix of Premium and Standard SSD disks.

While they do reduce the cost, because they introduce Standard SSD disk, the VM will not benefit from the Single VM SLA, which is only valid when Premium SSD disks are used. Table 2-8 shows the suggested storage layouts:

VM SKU	RAM (GiB)	Max. VM I/O Throughput (Mbps)	/hana/data and /hana/log striped with LVM or MDADM	/hana /shared	/root volume	/usr /sap	Hana /backup
DS14v2	112	768	3 x P20	1 x E20	1 x E6	1 x E6	1 x E15
E16v3	128	384	3 x P20	1 x E20	1 x E6	1 x E6	1 x E15
E32v3	256	768	3 x P20	1 x E20	1 x E6	1 x E6	1 x E20
E64v3	432	1200	3 x P20	1 x E20	1 x E6	1 x E6	1 x E30
GS5	448	2000	3 x P20	1 x E20	1 x E6	1 x E6	1 x E30
M32ts	192	500	3 x P20	1 x E20	1 x E6	1 x E6	1 x E20
M32ls	256	500	3 x P20	1 x E20	1 x E6	1 x E6	1 x E20
M64ls	512	1000	3 x P20	1 x E20	1 x E6	1 x E6	1 x E30
M64s	1000	1000	2 x P30	1 x E30	1 x E6	1 x E6	2 x E30
M64ms	1750	1000	3 x P30	1 x E30	1 x E6	1 x E6	3 x E30
M128s	2000	2000	3 x P30	1 x E30	1 x E10	1 x E6	2 x E40
M128ms	3800	2000	5 x P30	1 x E30	1 x E10	1 x E6	2 x E50
M208s_v2	2850	1000	4 x P30	1 x E30	1 x E10	1 x E6	3 x E40
M208ms_v2	5700	1000	4 x P40	1 x E30	1 x E10	1 x E6	4 x E40
M416s_v2	5700	2000	4 x P40	1 x E30	1 x E10	1 x E6	4 x E40
M416ms_v2	11400	2000	8 x P40	1 x E30	1 x E10	1 x E6	4 x E50

Table 2-8: SAP HANA VM cost optimised storage layout

Designing the storage layout for SAP HANA is a much more complex process compared to other databases due to SAP storage throughput and latency requirements. To ensure your workload is supported by Microsoft and SAP, you should always follow the preceding recommendations.

System deployment

There are multiple options for how to design the SAP HANA architecture to run in Azure:

1. Standalone HANA deployment
2. HANA MCOS
3. HANA MDC
4. HANA Scale-Up or Scale-Out

In some cases, these are similar to the options available for deploying AnyDB for SAP NetWeaver systems as already described, but there are also some additional options for SAP HANA. We'll discuss the above options individually in the following sections.

Standalone HANA deployment

A single node installation of the SAP HANA DBMS is the most common scenario. The database is installed on either SUSE Linux Enterprise Server or Red Hat Enterprise Linux. For production workloads, only certified Azure VMs must be used, but for non-production workloads, non-certified VMs can be used, which may offer lower cost and also provide options with less memory; the smallest certified VM is currently 192 GB.

HANA multiple components on one system (MCOS)

Instead of hosting a separate server for each database instance, multiple separate databases can be deployed together on a single server. In theory it could simplify the administration, by reducing the total number of systems that need to be administered, but in real life additional effort is required to manage the system and ensure that the performance of one database is not affected by others. The database components may interfere with each other and could require special configuration of the operating system, especially if the database releases are not all the same.

Keeping multiple database instances on the same virtual machine is not recommended for a production workload; however it may be an optimization technique for development and test environments.

This solution was more common in Azure in the early days of HLI, when less granular sizes were available, but with HANA-certified VMs there is a much wider range of sizes available. Rather than deploy many small independent HLI for non-production workloads, some customers would deploy fewer large HLI and host multiple HANA databases on each. This was also useful where you need to have a large HLI for DR purposes, but under normal operation want to use that HLI to host multiple non-production databases to reduce cost.

HANA multitenant database containers (MDC)

With HANA 1.0 SPS09, SAP introduced the concept of HANA MDC. With HANA 2.0, MDC became the default, although you can deploy a single tenant database. While MDC is superficially similar in concept to MCOD, in detail, it is very different. MDC provides much greater isolation between the individual databases, making it much more secure and robust than MCOD.

MDC is SAP's preferred solution for virtualizing HANA. While they support running HANA on-premises on VMware and in a public cloud such as Azure their preference has generally been to use bare-metal-certified appliances, and then if you have many small HANA databases, to use MDC so they can easily share a single appliance. However, while MDC does allow you to stack multiple databases, all the databases share one set of binaries, so all the databases run on a common release. It is not possible to patch or upgrade the HANA version for a single database container, and if the HANA instance needs to be restarted, it means that all database containers will suffer downtime.

There are a few use cases where MDC makes good sense:

• Where you have many fairly small HANA databases, then MDC can be a cost-efficient way to host them. You might for instance consider using MDC for EP, PI, or Fiori FES and combine them with one of the smaller SAP components such as SRM.
• In certain non-production environments, such as Sandbox and possible Development, then again you may have many small databases and MDC can reduce the hosting costs.
• Where you have a very small landscape and the SAP components are tightly connected with each other. For example, if all you are running is ECC and Portal on HANA, then you might be happy to have them share a common HANA instance and use MDC.

In SAP NetWeaver workloads, such deployment is popular in Disaster Recovery scenarios, where the main use of the of the secondary system is non-production workloads, but in case of the failover it can overtake the production operations.

It decreases the cost of running the disaster recovery systems.

Figure 2-31: Multi components one system versus multitenant database containers

Figure 2-31 represents in a graphical way the difference between MCOS and MDC deployments of SAP HANA. In the example on the left, each tenant database is isolated in a separate SAP HANA instance with its own binaries and system database. It allows independent maintenance of each database including upgrades. The MDC deployments uses the same common binaries for all databases, which simplify the management but doesn't allow independent upgrades.

HANA scale-up and scale-out

HANA supports two main deployment options, Scale-up and Scale-out. With scale-up, the whole HANA instance must fit into a single HANA node, and as the database grows you will need to scale-up the host, be it a VM or a physical bare-metal server. With scale-out, in theory you can scale the size of the HANA instance as the database grows, by adding extra nodes.

While HANA scale-up can be used to support all scenarios, HANA scale-out is only supported in certain scenarios, so will not address all requirements. Supported scenarios for HANA scale-out are:

• SAP BW on HANA, BW4/HANA
• SAP HANA used for OLAP outside SAP applications
• SAP S/4HANA

The basic architecture of SAP HANA scale-out is shown in Figure 2-32:

Figure 2-32: SAP HANA scale-out architecture

The architecture comprises:

• A master node.
• One or more worker nodes that hold the majority of the data. As the database grows you can add additional worker nodes to increase the available capacity.
• Optional standby node(s), there can be one or more, which provide a high availability solution. In the event of the failure of the master node or one of the worker nodes, then the standby node can take over its role.

Both scale-up and scale-out are supported in Azure on VMs and also on HLI. Most customers prefer to use VMs, and since the release of ANF, it is now possible to deploy SAP HANA scale-out on VMs with a standby node, which previously was only possible on HLI:

• HLI do still have their place as they address certain requirements that cannot currently be addressed with VMs. The main use case for Azure HLI is where your application requires a single HANA instance of more than 12 TB (11.4 TB); a HLI environment is certified up to 20 TB and can support up to 24 TB under the HANA TDIv5 rules.

SAP HANA resilience

When you are using SAP HANA as the database for SAP applications, then you need to consider high availability and disaster recovery in the same way as for any other database. There are some different options, as SAP HANA has some additional built-in capability, but in many cases the solutions are similar to those we have already examined. In this section, we will look at these topics again, for both SAP HANA running on VMs and for SAP HANA running on Azure HLI.

SAP HANA high availability

There are different solutions for high availability (HA) depending on whether you are using HANA scale-up or HANA scale-out, and we will examine these in more detail. As a minimum, when running on VMs, then you can simply rely on Azure VM auto-restart to provide HA for HANA VMs, just as you can for VMs running an other database or the SAP application tier, and this will probably be sufficient for most non-production environments. If you are using HLI there is no auto-restart capability and no spare servers for you to use, so it is even more important that you design for HA. In either case, you need to be aware that the larger the HANA database size, the longer the database will take to start and load its data into memory. For small databases, the restart time may be acceptable, but for larger production databases, the restart time is very unlikely to meet the required SLA. In these situations, HANA-level HA is required.

SAP HANA scale-up HA

To provide HA for SAP HANA scale up, the recommended solution from SAP is to use HANA System Replication (HSR) in synchronous mode. This effectively keeps two HANA instances in sync, with updates applied to the primary and the secondary at the same time. This allows rapid failover of the service in the event of a failure of the primary node. This is the solution that is supported in Azure.

This solution is shown in Figure 2-33. The SBD devices deployed to an availability set are used as the fencing mechanism for the database.

Figure 2-33: SAP HANA system replication synchronous for high availability

You can configure essentially the same solution whether using Azure VMs or HLI.

Failover of the HANA database from primary to secondary is once again handled by Linux Pacemaker. Pacemaker requires a cluster quorum to ensure that both nodes cannot be active at the same time. Currently available options for quorum devices are the Azure Fencing Agent and SBD. Red Hat Enterprise Linux does not support SBD but has implemented the latest Azure Fencing Agent feature that supports fast failover, by forcing the VM shutdown that reduces the time required for failover. For SUSE Enterprise Linux, the SBD device remains the best solution for cluster quorum, but the Azure Fencing Agent fast shutdown option should be supported soon.

You may ask whether it is possible to use the standby node to host another non-production HANA database, but this is not advised in an HA cluster such as this. Using a hot standby node, failover of the HANA instance can happen in seconds. If you use a warm standby where the data is not preloaded into memory, because the memory is being used to host another database, then it will potentially take tens of minutes to load the standby database into memory from disks, too long for HA. If you are running HANA scale-up on VMs, you might as well wait for the failed VM to auto-restart.

While it is technically possible to create a SAP HANA Scale-Out solution with just two nodes, one active and one standby, this is not generally a recommended solution. Once again, the challenge with SAP HANA is that restarting a HANA instance requires that the row store is reloaded into memory, and this can take 5-10 minutes per TB. Using DBMS replication with memory preload means that the standby node is ready to take over processing virtually immediately.

SAP HANA scale-out HA

For HANA Scale-Out, there are two options for HA:

• HANA host auto-failover using a standby node
• HANA System Replication synchronous

In the following sub-sections, we'll describe these options in further detail.

HANA scale-out HA with host auto-failover

HANA host auto-failover has been a native capability of SAP HANA since the early versions, and allows for a standby node to be created that can take over the work of any of the other nodes, either master or workers. This requires shared disk storage as the standby node needs to mount the /hana/data and /hana/log disks from the failed node.

Azure has supported SAP HANA host auto-failover on HLI since the beginning, as HLI uses Azure NetApp storage to provide the required storage for HANA. The NetApp volumes are mounted to the HLI servers using NFS, and the standby node can simply take over the volumes from a failed node.

However, until ANF was certified for use with SAP HANA running on Azure VMs in November 2019 it had not been possible to have a standby node when deploying SAP HANA on Azure VMs. By using ANF storage the same host auto-failover option is now available for VMs as on Azure HLI. This is shown in Figure 2-34:

Figure 2-34: SAP HANA scale-out HA with standby node using Azure NetApp files

HANA scale-out HA with HANA system replication

HANA System Replication synchronous (HSR sync) essentially works the same on HANA scale-out as it does with HANA scale-up. This requires you to have two HANA scale-out systems of the same size, and to replicate data synchronously from the primary to the standby. This solution will provide the fastest failover times, as the HANA database is preloaded in memory in the standby instance.

However, it doubles the cost as it requires twice the hardware.

Figure 2-35: HANA scale-out HA using HSR

As shown in Figure 2-35 each node of the SAP HANA scale-out has its own storage for data and log volumes. To ensure high availability the data is replicated using HSR. A common storage for the /hana/shared is required and, as the high performance is not required for the binaries, it can use a basic NFS cluster deployed on Linux virtual machines.

SAP HANA scale-out HA on VMs

Before the availability of ANF storage, you could still deploy SAP HANA scale-out on Azure VMs but without a standby node. This option is still relevant as you may not always need a standby node, because Azure VM auto-restart will restart a failed node for you. This will not be as fast a recovery as using a standby node but does avoid the additional cost of the standby node.

If you deploy SAP HANA scale-out on Azure VMs using Azure Premium or Ultra SSD storage, then the architecture looks like this:

Figure 2-36: SAP HANA scale-out on Azure VMs

Those familiar with SAP HANA scale-out will notice that Figure 2-36 does not show a standby node. When using Azure Premium or Ultra SSD storage, a standby node cannot be configured as it needs to be able to take over the disks from the failed master or worker node, which is not possible with Azure Premium or Ultra SSD. When running HANA scale-out on Azure VMs without ANF, there are three solutions for HA:

• Rely on the Azure VM auto-restart feature to restart the failed node. This may be adequate for less critical systems, and many BW on HANA or BW/4HANA systems may fall into that category.
• Use HSR in the same way as you would for HANA scale-up as described in HANA scale-out HA with HANA system replication.
• Use Azure HLI, shown next.

For some workloads, you may be prepared to rely on the VM auto-restart option and not have a standby node. In this case, you can run HANA scale-out with each database node having its own dedicated storage. To run HANA scale-out in this way, you must set the parameter basepath_shared = no in the global.ini. In this case, the storage layout is shown in Figure 2-37:

Figure 2-37: HANA scale-out storage on Azure VMs (source: Microsoft.com)

As shown in the preceding figure, a scale-out deployment still requires shared storage for hosting the /hana/shared directory. As there is no latency requirement for /hana/shared, you can build a custom NFS cluster or use ANF to store the shared filesystem.

The size of the shared filesystem (/hana/shared) is determined by how many nodes are to be used. Assuming the usage of m128s (2 TB) virtual machine, the size of the shared filesystem should be:

• One master node and up to four worker nodes; the /hana/shared volume would need to be 2 TB in size.
• One master node and five to eight worker nodes; the size of /hana/shared should be 4 TB.
• One master node and 9 to 12 worker nodes, a size of 6 TB for /hana/shared would be required.
• One master node and using between 12 and 15 worker nodes; you are required to provide a /hana/shared volume that is 8 TB in size.

Figure 2-38: HANA scale-out with NFS cluster for /hana/shared

A highly available NFS cluster requires a cluster quorum to perform failover. As described previously, currently available options are the Azure Fencing Agent and SBD. Red Hat Enterprise Linux does not support SBD, but has implemented the latest Azure Fencing Agent feature that supports fast failover by forcing the VM shutdown that reduces the time required for failover. For SUSE Enterprise Linux, the SBD device remains the best solution for a cluster quorum, but the Azure fencing Agent fast shutdown option should be supported soon.

SAP recommend that you separate the client communication from the intranode communication. This can be achieved by creating two subnets and assigning two network interface cards to each virtual machine that is part of the scale-out database. The two network interfaces are then connected to separate subnets and using NSGs it is possible to control the traffic.

This is shown in Figure 2-39:

Figure 2-39: Example of one node in a scale-out HANA database (Source: Microsoft.com)

Network traffic between the VMs that form the nodes of the HANA scale-out database, and network traffic between the HANA nodes and the highly available NFS cluster, must not under any circumstances be routed through a Network Virtual Appliance³⁶ (NVA) or similar virtual appliances.

It is important to note that SAP HANA scale-out does not run as a failover cluster and therefore does not require load balancers and fencing devices – these are only necessary for the NFS cluster.

HANA disaster recovery

For disaster recovery, the data must be replicated to another Azure region to ensure no data is lost in case of a region-level failure. For SAP HANA, the recommended solution is to have a server provisioned in the secondary region that will be a target for HANA system replication, to ensure the shortest recovery time. Due to the nature of disaster recovery, the failover should be manual, as it is normal to want to determine whether the primary site will be recovered quickly, or an actual failover to DR is required.

Because of this, clustering software is not normally used, and there is no requirement for a fencing solution such as the Azure Fencing Agent or SBD devices.

Following a failover to the secondary Azure region, the IP address of all servers will change. In order for end users to access the systems, it is standard practice to update the DNS records for these systems to resolve the hostname to a different IP:

Figure 2-40: Disaster recover for SAP HANA scale-up

When it comes to DR for SAP HANA scale-out systems then you will still use HSR async, but the diagram looks slightly more complex:

Figure 2-41: Disaster recovery for SAP HANA scale-out

In many cases, you will probably only worry about DR for production, but you may also want to provide some form of DR for non-production as well. Non-production workloads generally have a lower RTO and RPO requirement and therefore it is possible to use other options to protect them. One option is to use ASR with native backup to build a secondary system in case of failure. As ASR does not guarantee that the database data and log files will be synchronized, the replicated database may be inconsistent. As a solution, a database backup including transaction logs should be copied to Azure Blob storage and geo-replicated. During a failover, the backups can be retrieved and restored to a target system. ASR will orchestrate the VM provisioning and configuration, while the database backup from the cloud storage ensures the database consistency:

Figure 2-42: DR for non-production using ASR and backup/restore

Having considered the options for SAP HANA disaster recovery, we will now look at the options for SAP HANA backup. Even with local synchronous replication for HA and remote asynchronous replication for DR, it is still essential to implement a separate backup solution, as any logical database corruption is quickly replicated to all copies of the database.

HANA database backup

There are a number of ways to back up a HANA database in Azure, using either Azure native tools or third-party backup products.

HANA backup to a filesystem

The easiest backup solution to implement is to back up the database to a filesystem. This requires additional disks to be attached to the VM to store the database dump. The time required to execute the backup is directly influenced by the disk performance.

Therefore, you can combine multiple disks to create a single logical volume with high throughput. The following table provides a sample configuration depending on the virtual machine size:

VM Size	Memory	Backup Disk configuration	Total storage	Total throughput
M128s	2 TB	2x P40	4 TiB	500 Mbps
M208ms_v2	5.7 TB	3x P40	6 TiB	750 Mbps
M416ms_v2	11.4 TB	4x P50	16 TiB	1 000 Mbps

Table 2-9: Backup disks for performance

Once the backup has been written to the filesystem, it can later be replicated to Azure Blob storage for long-term retention using the AzCopy tool. When the copy is finished, the backup can be removed from the local disk or retained if required for fast restore. To decrease the price of storing the backups in Azure Blob, you can use different storage access tiers:

• Hot: For recent filesystem backup
• Cool: Optimized for backup from previous days (short term backup)
• Archive: Storing the backup for audit purposes

Each access tier is characterized by different pricing policies. The hot storage is the most expensive for keeping backup files; however, the price of accessing the data is low. The cool storage is cheaper, but reading the data is more expensive. Finally, the archive tier offers the cheapest to store the backup files, but accessing them is the most expensive and should be only used for files that are used infrequently (or never). Your auditors may require you to keep an annual full system backup for multiple years, and the archive tier may be a good option as it minimizes the cost, and it is unlikely you will ever access the backups.

The cool and archive storage tiers may have a lower availability in comparison to the hot storage.

The Blob storage also offers keeping copies of the storage:

• Locally redundant storage: Provides at least 99.999999999% (11 nines) durability of objects over a given year. It's a default replication for objects stored in the Azure Storage. Three copies of your data are kept in a single Azure region.
• Zone-redundant storage: Data is replicated synchronously to three storage clusters in a single region. Each cluster is physically separated and located in its own availability zone. ZRS offers durability for storage objects of at least 99.9999999999% (12 nines) over a given year.
• Geo-redundant storage: Designed to provide at least 99.99999999999999% (16 nines) durability of objects over a given year. The data is replicated to another Azure region, which is hundreds of miles from the primary region. Azure stores in total six copies of the data – three in the primary region and another three in the secondary region. Data replication between regions is asynchronous, which means it's first saved to LRS storage and then distributed.

Azure Backup for SAP HANA

Azure Backup is a native backup capability that is built into Azure. Microsoft has developed a SAP HANA backint compatible interface for Azure Backup that allows HANA to backup directly to Azure Backup. This capability is currently in private preview and has some limitations:

• Only selected operating systems are supported
• The backup works only for a single instance SAP HANA (no scale-out deployments)
• Multiple SAP HANA instances on the same VM are not supported
• Full and differential backups are supported, but not incremental

The Azure Backup for SAP HANA connects to the database and streams the data directly to the Azure Backup Vault. There is no need to specify a Storage Account as this is managed internally. You can configure the backup frequency and retention policies.

A further limitation of Azure Backup for SAP HANA is that it does not support HANA multi-streaming data backups, which was introduced with HANA 1.0 SP11. This limits the backup throughput and makes it more appropriate for smaller HANA databases.

HANA backup using filesystem snapshot

The quickest way to backup a HANA database is to perform a filesystem snapshot. It is the best solution for a fast backup of larger database, and also supports fast recovery. However, a snapshot backup cannot guarantee the data consistency, and should only be used in conjunction with a regular full streaming backup that will check the data integrity. Azure Storage does not, by default, guarantee application-level consistency when creating a snapshot. Linux VMs do not support the VSS technology, and therefore additional preparation is required to create an application-consistent snapshot. This is especially important when the SAP HANA files are distributed to multiple disks using LVM.

The overall process is as follows:

1. Execute the SAP HANA snapshot preparation
2. Freeze the filesystem
3. Create a disk snapshot in Azure
4. Unfreeze the filesystem
5. Confirm the SAP HANA snapshot

The snapshot backup can be simplified if the VM can be shut down, and this can be used with non-production systems, particularly where you want to use the snapshot to clone a system. As the HANA database does not perform any disk operations when shutdown, and all data held in memory will have been flushed to disk, then the snapshot will always be consistent and will not require any preparations such as the filesystem freeze.

For more details about the SAP HANA snapshot feature, please refer to the SAP Note: 2039883 - FAQ: SAP HANA database and storage snapshots.

HANA Backup using third-party tools

There are a large number of third-party backup tools that integrate with SAP HANA³⁷, are supported in the Azure environment, and can automate the routine backup tasks. In general, these tools provide centralized control of all your backups, potentially across on-premises as well as in Azure, and offer benefits such as compression, deduplication, and automatic data tiering.

At the time of writing, the company that has done the most work to prove its solutions for backing up SAP HANA on Azure is Commvault, who offer a SAP-certified backup solution. The backup is streamed directly to the centralized MediaAgent and stored in Azure Blob storage. The deduplication feature identifies and eliminates redundant data in the backup, thereby reducing not only the volume of data stored in the cloud, but also the bandwidth required for data transfer. A new feature that is due to be released shortly is integration of Commvault Intellisnap with Azure snapshots, which will allow Commvault to also manage the filesystem snapshot process.

The Commvault solution is not limited to HANA Backup. It may also be used to backup the complete SAP landscape, including application servers and all the SAP-supported DBMS types. It can also backup other non-SAP workloads, and therefore can act as the primary backup tool to protect your entire Azure environment. Table 2-10 summarizes the main advantages and disadvantages of each backup solution.

Backup to filesystem	Azure Backup for SAP HANA	Backup using filesystem snapshot	Third-party tools
+ Easy to implement + Native SAP HANA solution	+ Native Azure solution + Allows you to specify backup and retention policies + Avoids custom scripts	+ The backup and recovery is very fast	+ Solutions certified by SAP + Data deduplication decreases the cost of storage + Allows you to specify backup and retention policies
- Requires storage on the virtual machine - Scripting is required to automate backup and to upload data to Blob storage	- Currently in preview - Doesn't support scale-out systems - Only works for a single HANA instance on a VM	- Complex to execute - Requires scripting to automate	- Requires a separate license - Requires additional Azure infrastructure like VMs for management and data movers

Table 2-10: SAP HANA backup option in Azure

Depending on the architecture of your landscape and its complexity, you can choose the right backup solution. The database backup to filesystem is the easiest to implement, but as it requires additional storage attached to the VM, it increases the total cost. Third-party tools offer the widest capabilities; however, they require an additional license and additional Azure resources.

Monitoring and performance optimization

The SAP HANA database is sized according to its memory requirements. The Azure-certified VMs follow the SAP guidance in terms of core to memory ratios. The HANA workload does not produce consistent and stable CPU utilization, instead it is usual to see the CPU usage as quite low much of the time, with spikes reaching to 80-90% utilization.

An important factor for HANA workloads is the disk write speed, especially on the log volumes. Microsoft Azure provides guidance around the storage layout, which may be further optimized by the customer if required. To fulfill the SAP storage latency requirements when using M series and Mv2 series VMs with Premium SSD, the Write Accelerator should always be enabled on the log volumes. Microsoft is gradually certifying other VM series that do not support the Write Accelerator functionality with Ultra SSD. Check the SAP-certified and Supported Hardware Directory for the latest certifications³⁸.

In this section, we have reviewed the options for deploying SAP HANA in Azure, using both virtual machines and HANA large instances. We have looked at the options for HA and DR, and finally considered options for backup and monitoring. Microsoft made a commitment in 2016 to make Azure the most scalable and performant platform for running SAP, and since then they have been continuously releasing and enhancing features to extend what can be supported in Azure. In general terms. Azure can now offer similar or better capabilities for SAP HANA as those available on-premises.