Azure Virtual Machines Oracle database deployment for SAP workload

This document covers several different areas to consider when deploying Oracle Database for SAP workload in Azure IaaS. Before you read this document, we recommend you read Considerations for Azure Virtual Machines DBMS deployment for SAP workload. We also recommend that you read other guides in the SAP workload on Azure documentation.

You can find information about Oracle versions and corresponding OS versions that are supported for running SAP on Oracle on Azure in SAP Note 2039619.

General information about running SAP Business Suite on Oracle can be found at SAP on Oracle. Oracle supports to run Oracle databases on Microsoft Azure. For more information about general support for Windows Hyper-V and Azure, check the Oracle and Microsoft Azure FAQ.

The following SAP notes are relevant for an Oracle Installation

Specifics for Oracle Database on Oracle Linux

Oracle supports to run their database instances on Microsoft Azure with Oracle Linux as the guest OS. For more information about general support for Windows Hyper-V and Azure, see the Azure and Oracle FAQ.

The specific scenario of SAP applications using Oracle Databases is supported as well. Details are discussed in the next part of the document.

General Recommendations for running SAP on Oracle on Azure

Installing or migrating existing SAP on Oracle systems to Azure, the following deployment pattern should be followed:

  1. Use the most recent Oracle Linux version available (Oracle Linux 8.6 or higher).
  2. Use the most recent Oracle Database version available with the latest SAP Bundle Patch (SBP) (Oracle 19 Patch 15 or higher) 2799920 - Patches for 19c: Database.
  3. Use Automatic Storage Management (ASM) for small, medium, and large sized databases on block storage.
  4. Azure Premium Storage SSD should be used. Don't use Standard or other storage types.
  5. ASM removes the requirement for Mirror Log. Follow the guidance from Oracle in Note 888626 - Redo log layout for high-end systems.
  6. Use ASMLib and don't use udev.
  7. Azure NetApp Files deployments should use Oracle dNFS which is Oracle’s own high performance Direct NFS (Network File System) driver solution.
  8. Large Oracle databases benefit greatly from large System Global Area (SGA) sizes. Large customers should deploy on Azure M-series with 4 TB or more RAM size
    • Set Linux Huge Pages to 75% of Physical RAM size
    • Set System Global Area (SGA) to 90% of Huge Page size
    • Set the Oracle parameter USE_LARGE_PAGES = ONLY - The value ONLY is preferred over the value TRUE as the value ONLY is supposed to deliver more consistent and predictable performance. The value TRUE may allocate both large 2MB and standard 4K pages. The value ONLY is going to always force large 2MB pages. If the number of available huge pages isn't sufficient or not correctly configured, the database instance is going to fail to start with error code: ora-27102 : out of memory Linux_x86_64 Error 12 : can't allocate memory. If there's insufficient contiguous memory, Oracle Linux may need to be restarted and/or the Operating System Huge Page parameters reconfigured.
  9. Oracle Home should be located outside of the "root" volume or disk. Use a separate disk or ANF volume. The disk holding the Oracle Home should be 64 Gigabytes in size or larger.
  10. The size of the boot disk for large high performance Oracle database servers is important. As a minimum a P10 disk should be used for M-series or E-series. Don't use small disks such as P4 or P6. A small disk can cause performance issues.
  11. Accelerated Networking must be enabled on all Virtual Machines. Upgrade to the latest Oracle Linux release if there are any problems enabling Accelerated Networking.
  12. Check for updates in this documentation and SAP note 2039619 - SAP Applications on Microsoft Azure using the Oracle Database: Supported Products and Versions - SAP ONE Support Launchpad.

For information about which Oracle versions and corresponding OS versions are supported for running SAP on Oracle on Azure Virtual Machines, see SAP Note 2039619.

General information about running SAP Business Suite on Oracle can be found in the SAP on Oracle community page. SAP on Oracle on Azure is only supported on Oracle Linux (and not Suse or Red Hat) for application and database servers. ASCS/ERS servers can use RHEL/SUSE because Oracle client isn't installed or used on these VMs. Application Servers (PAS/AAS) shouldn't be installed on these VMs. Refer to SAP Note 3074643 - OLNX: FAQ: if Pacemaker for Oracle Linux is supported in SAP Environment. Oracle Real Application Cluster (RAC) isn't supported on Azure because RAC would require Multicast networking.

Storage configuration

There are two recommended storage deployment patterns for SAP on Oracle on Azure:

  1. Oracle Automatic Storage Management (ASM)
  2. Azure NetApp Files (ANF) with Oracle dNFS (Direct NFS)

Customers currently running Oracle databases on EXT4 or XFS file systems with Logical Volume Manager (LVM) are encouraged to move to ASM. There are considerable performance, administration, and reliability advantages to running on ASM compared to LVM. ASM reduces complexity, improves supportability, and makes administration tasks simpler. This documentation contains links for Oracle Database Administrators (DBAs) to learn how to install and manage ASM.

Azure provides multiple storage solutions.

The table below details the support status

Storage type Oracle support Sector Size Oracle Linux 8.x or higher Windows Server 2019
Block Storage Type
Premium SSD Supported 512e ASM Recommended. LVM Supported No support for ASM on Windows
Premium SSD v21 Supported 4K Native or 512e2 ASM Recommended. LVM Supported No support for ASM on Windows. Change Log File disks from 4K Native to 512e
Standard SSD Not supported
Standard HDD Not supported
Ultra disk Supported 4K Native ASM Recommended. LVM Supported No support for ASM on Windows. Change Log File disks from 4K Native to 512e
Network Storage Types
Azure NetApp Service (ANF) Supported - Oracle dNFS Required Not supported
Azure Files NFS Not supported
Azure files SMB Not supported
  1. Azure Premium SSD v2 doesn't have predefined storage sizes. There's no need to allocate multiple disks within an ASM Disk Group or LVM VG. It's recommended to allocate a single Premium SSD v2 disk with the required size, throughput, and IOPS per ASM Disk Group
  2. 512e is supported on Premium SSD v2 for Windows systems. 512e configurations are't recommended for Linux customers. Migrate to 4K Native using procedure in MOS 512/512e sector size to 4K Native Review (Doc ID 1133713.1)

Other considerations that apply list like:

  1. No support for DIRECTIO with 4K Native sector size. Recommended settings for FILESYSTEMIO_OPTIONS for LVM configurations:
    • LVM - If disks with 512/512e geometry are used, FILESYSTEMIO_OPTIONS = SETALL
    • LVM - If disks with 4K Native geometry are used, FILESYSTEMIO_OPTIONS = ASYNC
  2. Oracle 19c and higher fully supports 4K Native sector size with both ASM and LVM
  3. Oracle 19c and higher on Linux – when moving from 512e storage to 4K Native storage Log sector sizes must be changed
  4. To migrate from 512/512e sector size to 4K Native Review (Doc ID 1133713.1) – see section "Offline Migration to 4KB Sector Disks"
  5. SAPInst writes to the pfile during installation. If the $ORACLE_HOME/dbs is on a 4K disk, set filesystemio_options=asynch and see the Section "Datafile Support of 4kB Sector Disks" in MOS Supporting 4K Sector Disks (Doc ID 1133713.1)
  6. No support for ASM on Windows platforms
  7. No support for 4K Native sector size for Log volume on Windows platforms. SSDv2 and Ultra Disk must be changed to 512e via the "Edit Disk" pencil icon in the Azure portal
  8. 4K Native sector size is supported only on Data volumes for Windows platforms. 4K isn't supported for Log volumes on Windows
  9. We recommend reviewing these MOS articles:
    • Oracle Linux: File System's Buffer Cache versus Direct I/O (Doc ID 462072.1)
    • Supporting 4K Sector Disks (Doc ID 1133713.1)
    • Using 4k Redo Logs on Flash, 4k-Disk and SSD-based Storage (Doc ID 1681266.1)
    • Things To Consider For Setting filesystemio_options And disk_asynch_io (Doc ID 1987437.1)

We recommend using Oracle ASM on Linux with ASMLib. Performance, administration, support, and configuration are optimized with deployment pattern. Oracle ASM and Oracle dNFS are going to set the correct parameters or bypass parameters (such as FILESYSTEMIO_OPTIONS) and therefore deliver better performance and reliability.

Oracle Automatic Storage Management (ASM)

Checklist for Oracle Automatic Storage Management:

  1. All SAP on Oracle on Azure systems are running ASM including Development, Quality Assurance, and Production. Small, Medium, and Large databases
  2. ASMLib is used and not UDEV. UDEV is required for multiple SANs, a scenario that doesn't exist on Azure
  3. ASM should be configured for External Redundancy. Azure Premium SSD storage provides triple redundancy. Azure Premium SSD matches the reliability and integrity of any other storage solution. For optional safety, customers can consider Normal Redundancy for the Log Disk Group
  4. Mirroring Redo Log files is optional for ASM 888626 - Redo log layout for high-end systems
  5. ASM Disk Groups configured as per Variant 1, 2 or 3 below
  6. ASM Allocation Unit size = 4MB (default). Very Large Databases (VLDB) OLAP systems such as SAP BW may benefit from larger ASM Allocation Unit size. Change only after confirming with Oracle support
  7. ASM Sector Size and Logical Sector Size = default (UDEV isn't recommended but requires 4k)
  8. If the COMPATIBLE.ASM disk group attribute is set to 11.2 or greater for a disk group, you can create, copy, or move an Oracle ASM SPFILE into ACFS file system. Review the Oracle documentation on moving pfile into ACFS. SAPInst isn't creating the pfile in ACFS by default
  9. Appropriate ASM Variant is used. Production systems should use Variant 2 or 3

Oracle Automatic Storage Management Disk Groups

Part II of the official Oracle Guide describes the installation and the management of ASM:

The following ASM limits exist for Oracle Database 12c or later:

511 disk groups, 10,000 ASM disks in a Disk Group, 65,530 ASM disks in a storage system, 1 million files for each Disk Group. More info here: Performance and Scalability Considerations for Disk Groups (oracle.com)

Review the ASM documentation in the relevant SAP Installation Guide for Oracle available from https://help.sap.com/viewer/nwguidefinder

Variant 1 – small to medium data volumes up to 3 TB, restore time not critical

Customer has small or medium sized databases where backup and/or restore + Recovery of all databases can be accomplished using RMAN in a timely fashion. Example: When a complete Oracle ASM disk group, with data files, from one or more databases is broken and all data files from all databases need to be restored to a newly created Oracle ASM disk group using RMAN.

Oracle ASM disk group recommendation:

ASM Disk Group Name Stores Azure Storage
+DATA All data files 3-6 x P 30 (1 TiB)
Control file (first copy) To increase database size, add extra P30 disks
Online redo logs (first copy)
+ARCH Control file (second copy) 2 x P20 (512 GiB)
Archived redo logs
+RECO Control file (third copy) 2 x P20 (512 GiB)
RMAN backups (optional)
recovery area (optional)

Variant 2 – medium to large data volumes between 3 TB and 12 TB, restore time important

Customer has medium to large sized databases where backup and/or restore, or recovery of all databases can't be accomplished in a timely fashion.

Usually customers are using RMAN, Azure Backup for Oracle and/or disk snapshot techniques in combination.

Major differences to Variant 1 are:

  1. Separate Oracle ASM Disk Group for each database
  2. <DBNAME>+“_” is used as a prefix for the name of the DATA disk group
  3. The number of the DATA disk group is appended if the database spans over more than one DATA disk group
  4. No online redo logs are located in the "data" disk groups. Instead an extra disk group is used for the first member of each online redo log group.
ASM Disk Group Name Stores Azure Storage
+<DBNAME>_DATA[#] All data files 3-12 x P 30 (1 TiB)
All temp files To increase database size, add extra P30 disks
Control file (first copy)
+OLOG Online redo logs (first copy) 3 x P20 (512 GiB)
+ARCH Control file (second copy) 3 x P20 (512 GB)
Archived redo logs
+RECO Control file (third copy) 3 x P20 (512 GiB)
RMAN backups (optional)
Fast recovery area (optional)

Variant 3 – huge data and data change volumes more than 5 TB, restore time crucial

Customer has a huge database where backup and/or restore, or recovery of a single database can't be accomplished in a timely fashion.

Usually customers are using RMAN, Azure Backup for Oracle and/or disk snap techniques in combination. In this variant, each relevant database file type is separated to different Oracle ASM disk groups.

ASM Disk Group Name Stores Azure Storage
+<DBNAME>_DATA[#] All data files 5-30 or more x P30 (1 TiB) or P40 (2 TiB)
All temp files To increase database size, add extra P30 disks
Control file (first copy)
+OLOG Online redo logs (first copy) 3-8 x P20 (512 GiB) or P30 (1 TiB)
For more safety "Normal Redundancy" can be selected for this ASM Disk Group
+ARCH Control file (second copy) 3-8 x P20 (512 GiB) or P30 (1 TiB)
Archived redo logs
+RECO Control file (third copy) 3 x P30 (1 TiB), P40 (2 TiB) or P50 (4 TiB)
RMAN backups (optional)
Fast recovery area (optional)

Σημείωση

Azure Host Disk Cache for the DATA ASM Disk Group can be set to either Read Only or None. Consider that with some of the new M(b)v3 VM types, the usage of read cached Premium SSD v1 storage could result in lower read and write IOPS rates and throughput than you would get if you don't use read cache. All other ASM Disk Groups should be set to None. On BW or SCM a separate ASM Disk Group for TEMP can be considered for large or busy systems.

Adding Space to ASM + Azure Disks

Oracle ASM Disk Groups can either be extended by adding extra disks or by extending current disks. We recommend adding extra disks rather than extending existing disks. Review these MOS articles and links MOS Notes 1684112.1 and 2176737.1

ASM adds a disk to the disk group: asmca -silent -addDisk -diskGroupName DATA -disk '/dev/sdd1'

ASM automatically rebalances the data. To check rebalancing run this command.

ps -ef | grep rbal

oraasm 4288 1 0 Jul28 ? 00:04:36 asm_rbal_oradb1

Documentation is available with:

Monitoring SAP on Oracle ASM Systems on Azure

Run an Oracle AWR (Automatic Workload Repository) report as the first step when troubleshooting a performance problem. Disk performance metrics are detailed in the AWR report.

Disk performance can be monitored from inside Oracle Enterprise Manager and via external tools. Documentation, which might help is available here:

OS level monitoring tools can't monitor ASM disks as there's no recognizable file system. Freespace monitoring must be done from within Oracle.

Training Resources on Oracle Automatic Storage Management (ASM)

Oracle DBAs that aren't familiar with Oracle ASM follow the training materials and resources here:

Azure NetApp Files (ANF) with Oracle dNFS (Direct NFS)

The combination of Azure VMs and ANF is a robust and proven combination implemented by many customers on an exceptionally large scale.

Databases of 100+ TB are already running productive on this combination. To start, we wrote a detailed blog on how to set up this combination:

More general information

Mirror Log is required on dNFS ANF Production systems.

Even though the ANF is highly redundant, Oracle still requires a mirrored redo-logfile volume. The recommendation is to create two separate volumes and configure origlogA together with mirrlogB and origlogB together with mirrlogA. In this case, you make use of a distributed load balancing of the redo-logfiles.

The mount option "nconnect" isn't recommended when the dNFS client is configured. dNFS manages the IO channel and makes use of multiple sessions, so this option is obsolete and can cause manifold issues. The dNFS client is going to ignore the mount options and is going to handle the IO directly.

Both NFS versions (v3 and v4.1) with ANF are supported for the Oracle binaries, data- and log-files.

We highly recommend using the Oracle dNFS client for all Oracle volumes.

Recommended mount options are:

NFS Version Mount Options
NFSv3 rw,vers=3,rsize=262144,wsize=262144,hard,timeo=600,noatime
NFSv4.1 rw,vers=4.1,rsize=262144,wsize=262144,hard,timeo=600,noatime

ANF Backup

With ANF, some key features are available like consistent snapshot-based backups, low latency, and remarkably high performance. From version 6 of our AzAcSnap tool Azure Application Consistent Snapshot tool for ANF, Oracle databases can be configured for consistent database snapshots.

Those snapshots remain on the actual data volume and must be copied away using ANF CRR (Cross Region Replication) Cross-region replication of ANF or other backup tools.

SAP on Oracle on Azure with LVM

ASM is the default recommendation from Oracle for all SAP systems of any size on Azure. Performance, reliability, and support are better for customers using ASM. Oracle provides documentation and training for DBAs to transition to ASM. In cases where the Oracle DBA team doesn't follow the recommendation from Oracle, Microsoft, and SAP to use ASM the following LVM configuration should be used.

Note that: when creating LVM the "-i" option must be used to evenly distribute data across the number of disks in the LVM group.

Mirror Log is required when running LVM.

Minimum configuration Linux:

Component Disk Host Cache Striping1
/oracle/<SID>/origlogaA & mirrlogB Premium None Not needed
/oracle/<SID>/origlogaB & mirrlogA Premium None Not needed
/oracle/<SID>/sapdata1...n Premium None Recommended
/oracle/<SID>/oraarch2 Premium None Not needed
Oracle Home, saptrace, ... Premium None None
  1. Striping: LVM stripe using RAID0
  2. oraarch: LVM is optional

The disk selection for hosting Oracle's online redo logs is driven by IOPS requirements. It's possible to store all sapdata1...n (tablespaces) on a single mounted disk as long as the volume, IOPS, and throughput satisfy the requirements.

Performance configuration Linux:

Component Disk Host Cache Striping1
/oracle/<SID>/origlogaA Premium None Can be used
/oracle/<SID>/origlogaB Premium None Can be used
/oracle/<SID>/mirrlogAB Premium None Can be used
/oracle/<SID>/mirrlogBA Premium None Can be used
/oracle/<SID>/sapdata1...n Premium None Recommended
/oracle/<SID>/oraarch2 Premium None Not needed
Oracle Home, saptrace, ... Premium None None
  1. Striping: LVM stripe using RAID0
  2. oraarch: LVM is optional

Azure Infra: Virtual machine Throughput Limits & Azure Disk Storage Options

Current recommendations for Oracle Storage

  1. Azure Premium Storage – Most customers are deploying on ASM with Premium Storage
  2. Azure NetApp Files - VLDB customers, often with single Oracle databases larger than 50TB are typically using ANF and using Storage Snapshot capabilities of Azure NetApp Files for Backup and Restore
  3. Managed Disk Bursting - Managed disk bursting - Azure Virtual Machines | Microsoft Docs
  4. Azure Write Accelerator - used for the case that the Oracle redo log is based on Premium SSD v1 disks
  5. Online disk extension is fully supported for Premium Storage v1 and works with ASM

Log write times can be improved on Azure M-Series VMs by enabling Write Accelerator. Enable Azure Write Accelerator for the Azure Premium Storage disks used by the ASM Disk Group for online redo log files. For more information, see Write Accelerator.

Using Write Accelerator is optional but can be enabled if the AWR report indicates higher than expected log write times.

Azure Virtual Machine Throughput Limits

Each Azure Virtual machine (VM) type has limits for CPU, Disk, Network, and RAM. These limits are documented in the links below

The following recommendations should be followed when selecting a VM type:

  1. Ensure the Disk Throughput and IOPS is sufficient for the workload and at least equal to the aggregate throughput of the disks
  2. Consider enabling paid bursting especially for Redo Log disk(s)
  3. For ANF, the Network throughput is important as all storage traffic is counted as "Network" rather than Disk throughput
  4. Review this blog for Network tuning for M-series Optimizing Network Throughput on Azure M-series VMs HCMT (microsoft.com)
  5. Review this link that describes how to use an AWR report to select the correct Azure VM
  6. Azure Intel Ev5 Edv5 and Edsv5-series - Azure Virtual Machines |Microsoft Docs
  7. Azure AMD Eadsv5 Easv5 and Eadsv5-series - Azure Virtual Machines |Microsoft Docs
  8. Azure M-series/Msv2-series M-series - Azure Virtual Machines |Microsoft Docs and Msv2/Mdsv2 Medium Memory Series - Azure Virtual Machines | Microsoft Docs
  9. Azure Mv2 Mv2-series - Azure Virtual Machines | Microsoft Docs

Backup/restore

For backup/restore functionality, the SAP BR*Tools for Oracle are supported in the same way as they are on bare metal and Hyper-V. Oracle Recovery Manager (RMAN) is also supported for backups to disk and restores from disk.

For more information about how you can use Azure Backup and Recovery services for Oracle databases, see:

High availability

Oracle Data Guard is supported for high availability and disaster recovery purposes. To achieve automatic failover in Data Guard, you need to use Fast-Start Failover (FSFA). The Observer functionality (FSFA) triggers the failover. If you don't use FSFA, you can only use a manual failover configuration. For more information, see Implement Oracle Data Guard on an Azure Linux virtual machine.

Disaster Recovery aspects for Oracle databases in Azure are presented in the article Disaster recovery for an Oracle Database 12c database in an Azure environment.

Huge Pages & Large Oracle SGA Configurations

VLDB SAP on Oracle on Azure deployments apply SGA sizes in excess of 3TB. Modern versions of Oracle handle large SGA sizes well and significantly reduce IO. Review the AWR report and increase the SGA size to reduce read IO. 

As general guidance Linux Huge Pages should be configured to approximately 75% of the VM RAM size. The SGA size can be set to 90% of the Huge Page size. An approximate example would be a M192ms VM with 4 TB of RAM would have Huge Pages set proximately 3 TB.  The SGA can be set to a value a little less such as 2.95 TB.

Large SAP customers running on High Memory Azure VMs greatly benefit from HugePages as described in this article

NUMA systems vm.min_free_kbytes should be set to 524288 * <# of NUMA nodes>. See Oracle Linux : Recommended Value of vm.min_free_kbytes Kernel Tuning Parameter (Doc ID 2501269.1...

 

Oracle Linux provides a useful GUI management utility:

Oracle Linux has a new package management tool – DNF

Oracle Linux 8: Package Management made easy with free videos | Oracle Linux Blog

Oracle® Linux 8 Managing Software on Oracle Linux - Chapter 1 Yum DNF

Memory and NUMA configurations can be tested and benchmarked with a useful tool - Oracle Real Application Testing (RAT)

Oracle Real Application Testing: What Is It and How Do You Use It? (aemcorp.com)

Information on UDEV Log Corruption issue Oracle Redolog corruption on Azure | Oracle in the field (wordpress.com)

Oracle ASM in Azure corruption - follow up (dbaharrison.blogspot.com)

Data corruption on Hyper-V or Azure when running Oracle ASM - Red Hat Customer Portal

Set up Oracle ASM on an Azure Linux virtual machine - Azure Virtual Machines | Microsoft Docs

Oracle Configuration guidelines for SAP installations in Azure VMs on Windows

SAP on Oracle on Azure also supports Windows. The recommendations for Windows deployments are summarized below:

  1. The following Windows releases are recommended: Windows Server 2022 (only from Oracle Database 19.13.0 on) Windows Server 2019 (only from Oracle Database 19.5.0 on)
  2. There's no support for ASM on Windows. Windows Storage Spaces should be used to aggregate disks for optimal performance
  3. Install the Oracle Home on a dedicated independent disk (don't install Oracle Home on the C: Drive)
  4. All disks must be formatted NTFS
  5. Follow the Windows Tuning guide from Oracle and enable large pages, lock pages in memory and other Windows specific settings

At the time, of writing ASM for Windows customers on Azure isn't supported. The SAP Software Provisioning Manager (SWPM) for Windows doesn't support ASM currently.

Storage Configurations for SAP on Oracle on Windows

Minimum configuration Windows:

Component Disk Host Cache Striping1
E:\oracle\<SID>\origlogaA & mirrlogB Premium None Not needed
F:\oracle\<SID>\origlogaB & mirrlogA Premium None Not needed
G:\oracle\<SID>\sapdata1...n Premium None Recommended
H:\oracle\<SID>\oraarch2 Premium None Not needed
I:\Oracle Home, saptrace, ... Premium None None
  1. Striping: Windows Storage Spaces
  2. oraarch: Windows Storage Spaces is optional

The disk selection for hosting Oracle's online redo logs is driven by IOPS requirements. It's possible to store all sapdata1...n (tablespaces) on a single mounted disk as long as the volume, IOPS, and throughput satisfy the requirements.

Performance configuration Windows:

Component Disk Host Cache Striping1
E:\oracle\<SID>\origlogaA Premium None Can be used
F:\oracle\<SID>\origlogaB Premium None Can be used
G:\oracle\<SID>\mirrlogAB Premium None Can be used
H:\oracle\<SID>\mirrlogBA Premium None Can be used
I:\oracle\<SID>\sapdata1...n Premium None Recommended
J:\oracle\<SID>\oraarch2 Premium None Not needed
K:\Oracle Home, saptrace, ... Premium None None
  1. Striping: Windows Storage Spaces
  2. oraarch: Windows Storage Spaces is optional

Next steps

Read the article