Deploy SAP dialog instances with SAP ASCS/SCS high-availability VMs on RHEL

This article describes how to install and configure Primary Application Server (PAS) and Additional Application Server (AAS) dialog instances on the same ABAP SAP Central Services (ASCS)/SAP Central Services (SCS) high-availability cluster running on Red Hat Enterprise Linux (RHEL).

References

Overview

This article describes the cost optimization scenario where you deploy PAS and AAS dialog instances with SAP ASCS/SCS and Enqueue Replication Server (ERS) instances in a high-availability setup. To minimize the number of VMs for a single SAP system, you want to install PAS and AAS on the same host where SAP ASCS/SCS and SAP ERS are running. With SAP ASCS/SCS being configured in a high-availability cluster setup, you want PAS and AAS also to be managed by cluster. The configuration is basically an addition to an already configured SAP ASCS/SCS cluster setup. In this setup, PAS and AAS are installed on a virtual host name, and its instance directory is managed by the cluster.

For this setup, PAS and AAS require a highly available instance directory (/usr/sap/<SID>/D<nr>). You can place the instance directory file system on the same high-available storage that you used for ASCS and ERS instance configuration. The presented architecture showcases NFS on Azure Files or Azure NetApp Files for a highly available instance directory for the setup.

The example shown in this article to describe deployment uses the following system information:

Instance name Instance number Virtual host name Virtual IP (Probe port)
ABAP SAP Central Services (ASCS) 00 sapascs 10.90.90.10 (62000)
Enqueue Replication Server (ERS) 01 sapers 10.90.90.9 (62001)
Primary Application Server (PAS) 02 sappas 10.90.90.30 (62002)
Additional Application Server (AAS) 03 sapers 10.90.90.31 (62003)
SAP system identifier NW1 --- ---

Note

Install more SAP application instances on separate VMs if you want to scale out.

Diagram that shows the architecture of dialog instance installation with an SAP ASCS/SCS cluster.

Important considerations for the cost-optimization solution

  • Only two dialog instances, PAS and one AAS, can be deployed with an SAP ASCS/SCS cluster setup.
  • If you want to scale out your SAP system with more application servers (like sapa03 and sapa04), you can install them in separate VMs. With PAS and AAS being installed on virtual host names, you can install more application servers by using either a physical or virtual host name in separate VMs. To learn more about how to assign a virtual host name to a VM, see the blog Use SAP Virtual Host Names with Linux in Azure.
  • With a PAS and AAS deployment with an SAP ASCS/SCS cluster setup, the instance numbers of ASCS, ERS, PAS, and AAS must be different.
  • Consider sizing your VM SKUs appropriately based on the sizing guidelines. You must factor in the cluster behavior where multiple SAP instances (ASCS, ERS, PAS, and AAS) might run on a single VM when another VM in the cluster is unavailable.
  • The dialog instances (PAS and AAS) running with an SAP ASCS/SCS cluster setup must be installed by using a virtual host name.
  • You also must use the same storage solution of the SAP ASCS/SCS cluster setup to deploy PAS and AAS instances. For example, if you configured an SAP ASCS/SCS cluster by using NFS on Azure Files, the same storage solution must be used to deploy PAS and AAS.
  • The instance directory /usr/sap/<SID>/D<nr> of PAS and AAS must be mounted on an NFS file system and are managed as a resource by the cluster.

    Note

    For SAP J2EE systems, it's not supported to place /usr/sap/<SID>/J<nr> on NFS on Azure Files.

  • To install more application servers on separate VMs, you can either use NFS shares or a local managed disk for an instance directory file system. If you're installing more application servers for an SAP J2EE system, /usr/sap/<SID>/J<nr> on NFS on Azure Files isn't supported.
  • In a traditional SAP ASCS/SCS high-availability configuration, application server instances running on separate VMs aren't affected when there's any effect on SAP ASCS and ERS cluster nodes. But with the cost-optimization configuration, either the PAS or AAS instance restarts when there's an effect on one of the nodes in the cluster.
  • See NFS on Azure Files considerations and Azure NetApp Files considerations because the same considerations apply to this setup.

Prerequisites

The configuration described in this article is an addition to your already configured SAP ASCS/SCS cluster setup. In this configuration, PAS and AAS are installed on a virtual host name, and its instance directory is managed by the cluster. Based on your storage, use the steps described in the following articles to configure the SAPInstance resource for the SAP ASCS and SAP ERS instance in the cluster.

After you install the ASCS, ERS, and Database instance by using Software Provisioning Manager (SWPM), follow the next steps to install the PAS and AAS instances.

Configure Azure Load Balancer for PAS and AAS

This article assumes that you already configured the load balancer for the SAP ASCS/SCS cluster setup as described in Configure Azure Load Balancer. In the same Azure Load Balancer instance, follow these steps to create more front-end IPs and load-balancing rules for PAS and AAS.

  1. Open the internal load balancer that was created for the SAP ASCS/SCS cluster setup.
  2. Frontend IP Configuration: Create two front-end IPs, one for PAS and another for AAS (for example, 10.90.90.30 and 10.90.90.31).
  3. Backend Pool: This pool remains the same because we're deploying PAS and AAS on the same back-end pool.
  4. Inbound rules: Create two load-balancing rules, one for PAS and another for AAS. Follow the same steps for both load-balancing rules.
  5. Frontend IP address: Select the front-end IP.
    1. Backend pool: Select the back-end pool.
    2. High availability ports: Select this option.
    3. Protocol: Select TCP.
    4. Health Probe: Create a health probe with the following details (applies for both PAS and AAS):
      1. Protocol: Select TCP.
      2. Port: For example, 620<Instance-no.> for PAS and 620<Instance-no.> for AAS.
      3. Interval: Enter 5.
      4. Probe Threshold: Enter 2.
    5. Idle timeout (minutes): Enter 30.
    6. Enable Floating IP: Select this option.

The health probe configuration property numberOfProbes, otherwise known as Unhealthy threshold in the Azure portal, isn't respected. To control the number of successful or failed consecutive probes, set the property probeThreshold to 2. It's currently not possible to set this property by using the Azure portal. Use either the Azure CLI or the PowerShell command.

When VMs without public IP addresses are placed in the back-end pool of an internal (no public IP address) Standard Azure Load Balancer instance, there's no outbound internet connectivity unless more configuration is performed to allow routing to public endpoints. For steps on how to achieve outbound connectivity, see Public endpoint connectivity for virtual machines using Azure Standard Load Balancer in SAP high-availability scenarios.

Important

Don't enable TCP timestamps on Azure VMs placed behind Azure Load Balancer. Enabling TCP timestamps causes the health probes to fail. Set the parameter net.ipv4.tcp_timestamps to 0. For more information, see Load Balancer health probes.

Prepare servers for PAS and AAS installation

When steps in this document are marked with the following prefixes, they mean:

  • [A]: Applicable to all nodes.
  • [1]: Only applicable to node 1.
  • [2]: Only applicable to node 2.
  1. [A] Set up host name resolution.

    You can either use a DNS server or modify /etc/hosts on all nodes. This example shows how to use the /etc/hosts file. Replace the IP address and the host name in the following commands:

    sudo vi /etc/hosts
    
    # IP address of cluster node 1
    10.90.90.7    sap-cl1
    # IP address of cluster node 2
    10.90.90.8     sap-cl2
    # IP address of the load balancer frontend configuration for SAP Netweaver ASCS
    10.90.90.10   sapascs
    # IP address of the load balancer frontend configuration for SAP Netweaver ERS
    10.90.90.9    sapers
    # IP address of the load balancer frontend configuration for SAP Netweaver PAS
    10.90.90.30   sappas
    # IP address of the load balancer frontend configuration for SAP Netweaver AAS
    10.90.90.31   sapaas
    
  2. [1] Create the SAP directories on the NFS share. Mount the NFS share sapnw1 temporarily on one of the VMs, and create the SAP directories to be used as nested mount points.

    1. If you're using NFS on Azure Files:

      # mount temporarily the volume
      sudo mkdir -p /saptmp
      sudo mount -t nfs sapnfs.file.core.windows.net:/sapnfsafs/sapnw1 /saptmp -o noresvport,vers=4,minorversion=1,sec=sys
      
      # create the SAP directories
      sudo cd /saptmp
      sudo mkdir -p usrsapNW1D02
      sudo mkdir -p usrsapNW1D03
      
      # unmount the volume and delete the temporary directory
      cd ..
      sudo umount /saptmp
      sudo rmdir /saptmp
      
    2. If you're using Azure NetApp Files:

      # mount temporarily the volume
      sudo mkdir -p /saptmp
      
      # If using NFSv3
      sudo mount -t nfs -o rw,hard,rsize=65536,wsize=65536,vers=3,tcp 10.90.91.5:/sapnw1 /saptmp
      # If using NFSv4.1
      sudo mount -t nfs -o rw,hard,rsize=65536,wsize=65536,vers=4.1,sec=sys,tcp 10.90.91.5:/sapnw1 /saptmp
      
      # create the SAP directories
      sudo cd /saptmp
      sudo mkdir -p usrsapNW1D02
      sudo mkdir -p usrsapNW1D03
      
      # unmount the volume and delete the temporary directory
      sudo cd ..
      sudo umount /saptmp
      sudo rmdir /saptmp
      
  3. [A] Create the shared directories.

    sudo mkdir -p /usr/sap/NW1/D02
    sudo mkdir -p /usr/sap/NW1/D03
    
    sudo chattr +i /usr/sap/NW1/D02
    sudo chattr +i /usr/sap/NW1/D03
    
  4. [A] Configure swap space. When you install a dialog instance with central services, you must configure more swap space.

    sudo vi /etc/waagent.conf
    
    # Check if property ResourceDisk.Format is already set to y and if not, set it
    ResourceDisk.Format=y
    
    # Set the property ResourceDisk.EnableSwap to y
    # Create and use swapfile on resource disk.
    ResourceDisk.EnableSwap=y
    
    # Set the size of the SWAP file with property ResourceDisk.SwapSizeMB
    # The free space of resource disk varies by virtual machine size. Make sure that you do not set a value that is too big. You can check the SWAP space with command swapon
    # Size of the swapfile.
    #ResourceDisk.SwapSizeMB=2000
    ResourceDisk.SwapSizeMB=10480
    

    Restart the agent to activate the change.

    sudo service waagent restart
    
  5. [A] Add firewall rules for PAS and AAS.

    # Probe and gateway port for PAS and AAS
    sudo firewall-cmd --zone=public --add-port={62002,62003,3302,3303}/tcp --permanent
    sudo firewall-cmd --zone=public --add-port={62002,62003,3303,3303}/tcp
    

Install an SAP Netweaver PAS instance

  1. [1] Check the status of the cluster. Before you configure a PAS resource for installation, make sure the ASCS and ERS resources are configured and started.

    sudo pcs status
    
    # Online: [ sap-cl1 sap-cl2 ]
    #
    # Full list of resources:
    #
    # rsc_st_azure    (stonith:fence_azure_arm):      Started sap-cl1
    #  Resource Group: g-NW1_ASCS
    #      fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started sap-cl1
    #      nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started sap-cl1
    #      vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started sap-cl1
    #      rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started sap-cl1
    #  Resource Group: g-NW1_AERS
    #      fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started sap-cl2
    #      nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started sap-cl2
    #      vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started sap-cl2
    #      rsc_sap_NW1_ERS01  (ocf::heartbeat:SAPInstance):   Started sap-cl2
    
  2. [1] Create file system, virtual IP, and health probe resources for the PAS instance.

    sudo pcs node standby sap-cl2
    sudo pcs resource create vip_NW1_PAS IPaddr2 ip=10.90.90.30 --group g-NW1_PAS
    sudo pcs resource create nc_NW1_PAS azure-lb port=62002 --group g-NW1_PAS
    
    # If using NFS on Azure files
    sudo pcs resource create fs_NW1_PAS Filesystem device='sapnfs.file.core.windows.net:/sapnfsafs/sapnw1/usrsapNW1D02' \
      directory='/usr/sap/NW1/D02' fstype='nfs' force_unmount=safe options='noresvport,vers=4,minorversion=1,sec=sys' \
      op start interval=0 timeout=60 \
      op stop interval=0 timeout=120 \
      op monitor interval=200 timeout=40 \
      --group g-NW1_PAS
    
    # If using NFsv3 on Azure NetApp Files
    sudo pcs resource create fs_NW1_PAS Filesystem device='10.90.91.5:/sapnw1/usrsapNW1D02' \
      directory='/usr/sap/NW1/D02' fstype='nfs' force_unmount=safe \
      op start interval=0 timeout=60 \
      op stop interval=0 timeout=120 \ 
      op monitor interval=200 timeout=40 \
      --group g-NW1_PAS
    
    # If using NFSv4.1 on Azure NetApp Files
    sudo pcs resource create fs_NW1_PAS Filesystem device='10.90.91.5:/sapnw1/usrsapNW1D02' \
      directory='/usr/sap/NW1/D02' fstype='nfs' force_unmount=safe options='sec=sys,vers=4.1' \
      op start interval=0 timeout=60 \
      op stop interval=0 timeout=120 \
      op monitor interval=200 timeout=105 \
      --group g-NW1_PAS
    

    Make sure that the cluster status is okay and that all resources are started. It isn't important on which node the resources are running.

    sudo pcs status
    
    # Node List:
    #   Node sap-cl2: standby
    #   Online: [ sap-cl1 ]
    #
    # Full list of resources:
    #
    # rsc_st_azure    (stonith:fence_azure_arm):      Started sap-cl1
    #  Resource Group: g-NW1_ASCS
    #      fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started sap-cl1
    #      nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started sap-cl1
    #      vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started sap-cl1
    #      rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started sap-cl1
    #  Resource Group: g-NW1_AERS
    #      fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started sap-cl1
    #      nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started sap-cl1
    #      vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started sap-cl1
    #      rsc_sap_NW1_ERS01  (ocf::heartbeat:SAPInstance):   Started sap-cl1
    #  Resource Group: g-NW1_PAS:
    #      vip_NW1_PAS       (ocf::heartbeat:IPaddr2):        Started sap-cl1
    #      nc_NW1_PAS        (ocf::heartbeat:azure-lb):       Started sap-cl1
    #      fs_NW1_PAS        (ocf::heartbeat:Filesystem):     Started sap-cl1
    
  3. [1] Change the ownership of the /usr/sap/SID/D02 folder after the file system is mounted.

    sudo chown nw1adm:sapsys /usr/sap/NW1/D02
    
  4. [1] Install the SAP Netweaver PAS.

    Install the SAP NetWeaver PAS as a root on the first node by using a virtual host name that maps to the IP address of the load balancer front-end configuration for the PAS. For example, use sappas, 10.90.90.30, and the instance number that you used for the probe of the load balancer, for example 02.

    You can use the sapinst parameter SAPINST_REMOTE_ACCESS_USER to allow a nonroot user to connect to sapinst.

    # Allow access to SWPM. This rule is not permanent. If you reboot the machine, you have to run the command again.
    sudo firewall-cmd --zone=public  --add-port=4237/tcp
    
    sudo <swpm>/sapinst SAPINST_REMOTE_ACCESS_USER=sapadmin SAPINST_USE_HOSTNAME=<pas_virtual_hostname>
    
  5. Update the /usr/sap/sapservices file.

    To prevent the start of the instances by the sapinit startup script, all instances managed by Pacemaker must be commented out from the /usr/sap/sapservices file.

    sudo vi /usr/sap/sapservices
    
    # On the node where PAS is installed, comment out the following lines. 
    # LD_LIBRARY_PATH=/usr/sap/NW1/D02/exe:$LD_LIBRARY_PATH;export LD_LIBRARY_PATH;/usr/sap/NW1/D02/exe/sapstartsrv pf=/usr/sap/NW1/SYS/profile/NW1_D02_sappas -D -u nw1adm
    
  6. [1] Create the PAS cluster resource.

    # If using NFS on Azure Files or NFSv3 on Azure NetApp Files
    pcs resource create rsc_sap_NW1_PAS02 SAPInstance InstanceName="NW1_D02_sappas" \
     START_PROFILE=/sapmnt/NW1/profile/NW1_D02_sappas \
     op monitor interval=20 timeout=60 \
     --group g-NW1_PAS
    
    # If using NFSv4.1 on Azure NetApp Files
    pcs resource create rsc_sap_NW1_PAS02 SAPInstance InstanceName="NW1_D02_sappas" \
     START_PROFILE=/sapmnt/NW1/profile/NW1_D02_sappas \
     op monitor interval=20 timeout=105 \
     --group g-NW1_PAS
    

    Check the status of the cluster.

    sudo pcs status
    
    # Node List:
    #   Node sap-cl2: standby
    #   Online: [ sap-cl1 ]
    #
    # Full list of resources:
    #
    # rsc_st_azure    (stonith:fence_azure_arm):      Started sap-cl1
    #  Resource Group: g-NW1_ASCS
    #      fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started sap-cl1
    #      nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started sap-cl1
    #      vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started sap-cl1
    #      rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started sap-cl1
    #  Resource Group: g-NW1_AERS
    #      fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started sap-cl1
    #      nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started sap-cl1
    #      vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started sap-cl1
    #      rsc_sap_NW1_ERS01  (ocf::heartbeat:SAPInstance):   Started sap-cl1
    #  Resource Group: g-NW1_PAS:
    #      vip_NW1_PAS       (ocf::heartbeat:IPaddr2):        Started sap-cl1
    #      nc_NW1_PAS        (ocf::heartbeat:azure-lb):       Started sap-cl1
    #      fs_NW1_PAS        (ocf::heartbeat:Filesystem):     Started sap-cl1
    #      rsc_sap_NW1_PAS02 (ocf::heartbeat:SAPInstance):    Started sap-cl1
    
  7. Configure a constraint to start the PAS resource group only after the ASCS instance is started.

    sudo pcs constraint order g-NW1_ASCS then g-NW1_PAS kind=Optional symmetrical=false
    

Install an SAP Netweaver AAS instance

  1. [2] Check the status of the cluster. Before you configure an AAS resource for installation, make sure the ASCS, ERS, and PAS resources are started.

    sudo pcs status
    
    # Node List:
    #   Node sap-cl2: standby
    #   Online: [ sap-cl1 ]
    #
    # Full list of resources:
    #
    # rsc_st_azure    (stonith:fence_azure_arm):      Started sap-cl1
    #  Resource Group: g-NW1_ASCS
    #      fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started sap-cl1
    #      nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started sap-cl1
    #      vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started sap-cl1
    #      rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started sap-cl1
    #  Resource Group: g-NW1_AERS
    #      fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started sap-cl1
    #      nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started sap-cl1
    #      vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started sap-cl1
    #      rsc_sap_NW1_ERS01  (ocf::heartbeat:SAPInstance):   Started sap-cl1
    #  Resource Group: g-NW1_PAS:
    #      vip_NW1_PAS       (ocf::heartbeat:IPaddr2):        Started sap-cl1
    #      nc_NW1_PAS        (ocf::heartbeat:azure-lb):       Started sap-cl1
    #      fs_NW1_PAS        (ocf::heartbeat:Filesystem):     Started sap-cl1
    #      rsc_sap_NW1_PAS02 (ocf::heartbeat:SAPInstance):    Started sap-cl1
    
  2. [2] Create file system, virtual IP, and health probe resources for the AAS instance.

    sudo pcs node unstandby sap-cl2
    # Disable PAS resource as it will fail on sap-cl2 due to missing environment variables like hdbuserstore. 
    sudo pcs resource disable g-NW1_PAS
    sudo pcs node standby sap-cl1
    # Execute below command to cleanup resource, if required
    pcs resource cleanup rsc_sap_NW1_ERS01
    
    sudo pcs resource create vip_NW1_AAS IPaddr2 ip=10.90.90.31 --group g-NW1_AAS
    sudo pcs resource create nc_NW1_AAS azure-lb port=62003 --group g-NW1_AAS
    
    # If using NFS on Azure files
    sudo pcs resource create fs_NW1_AAS Filesystem device='sapnfs.file.core.windows.net:/sapnfsafs/sapnw1/usrsapNW1D03' \
      directory='/usr/sap/NW1/D03' fstype='nfs' force_unmount=safe options='noresvport,vers=4,minorversion=1,sec=sys' \
      op start interval=0 timeout=60 \
      op stop interval=0 timeout=120 \
      op monitor interval=200 timeout=40 \
      --group g-NW1_AAS
    
    # If using NFsv3 on Azure NetApp Files
    sudo pcs resource create fs_NW1_AAS Filesystem device='10.90.91.5:/sapnw1/usrsapNW1D03' \
      directory='/usr/sap/NW1/D03' fstype='nfs' force_unmount=safe \
      op start interval=0 timeout=60 \
      op stop interval=0 timeout=120 \ 
      op monitor interval=200 timeout=40 \
      --group g-NW1_AAS
    
    # If using NFSv4.1 on Azure NetApp Files
    sudo pcs resource create fs_NW1_AAS Filesystem device='10.90.91.5:/sapnw1/usrsapNW1D03' \
      directory='/usr/sap/NW1/D03' fstype='nfs' force_unmount=safe options='sec=sys,vers=4.1' \
      op start interval=0 timeout=60 \
      op stop interval=0 timeout=120 \
      op monitor interval=200 timeout=105 \
      --group g-NW1_AAS
    

    Make sure that the cluster status is okay and that all resources are started. It isn't important on which node the resources are running. Because the g-NW1_PAS resource group is stopped, all the PAS resources are stopped in the (disabled) state.

    sudo pcs status
    
    # Node List:
    #   Node sap-cl1: standby
    #   Online: [ sap-cl2 ]
    #
    # Full list of resources:
    #
    # rsc_st_azure    (stonith:fence_azure_arm):      Started sap-cl2
    #  Resource Group: g-NW1_ASCS
    #      fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started sap-cl2
    #      nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started sap-cl2
    #      vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started sap-cl2
    #      rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started sap-cl2
    #  Resource Group: g-NW1_AERS
    #      fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started sap-cl2
    #      nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started sap-cl2
    #      vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started sap-cl2
    #      rsc_sap_NW1_ERS01  (ocf::heartbeat:SAPInstance):   Started sap-cl2
    #  Resource Group: g-NW1_PAS:
    #      vip_NW1_PAS       (ocf::heartbeat:IPaddr2):        Stopped (disabled)
    #      nc_NW1_PAS        (ocf::heartbeat:azure-lb):       Stopped (disabled)
    #      fs_NW1_PAS        (ocf::heartbeat:Filesystem):     Stopped (disabled)
    #      rsc_sap_NW1_PAS02 (ocf::heartbeat:SAPInstance):    Stopped (disabled)
    #  Resource Group: g-NW1_AAS:
    #      vip_NW1_AAS       (ocf::heartbeat:IPaddr2):        Started sap-cl2
    #      nc_NW1_AAS        (ocf::heartbeat:azure-lb):       Started sap-cl2
    #      fs_NW1_AAS        (ocf::heartbeat:Filesystem):     Started sap-cl2
    
  3. [2] Change the ownership of the /usr/sap/SID/D03 folder after the file system is mounted.

    sudo chown nw1adm:sapsys /usr/sap/NW1/D03
    
  4. [2] Install an SAP Netweaver AAS.

    Install an SAP NetWeaver AAS as the root on the second node by using a virtual host name that maps to the IP address of the load balancer front-end configuration for the PAS. For example, use sapaas, 10.90.90.31, and the instance number that you used for the probe of the load balancer, for example, 03.

    You can use the sapinst parameter SAPINST_REMOTE_ACCESS_USER to allow a nonroot user to connect to sapinst.

    # Allow access to SWPM. This rule is not permanent. If you reboot the machine, you have to run the command again.
    sudo firewall-cmd --zone=public  --add-port=4237/tcp
    
    sudo <swpm>/sapinst SAPINST_REMOTE_ACCESS_USER=sapadmin SAPINST_USE_HOSTNAME=<aas_virtual_hostname>
    
  5. Update the /usr/sap/sapservices file.

    To prevent the start of the instances by the sapinit startup script, all instances managed by Pacemaker must be commented out from the /usr/sap/sapservices file.

    sudo vi /usr/sap/sapservices
    
    # On the node where AAS is installed, comment out the following lines. 
    #LD_LIBRARY_PATH=/usr/sap/NW1/D03/exe:$LD_LIBRARY_PATH;export LD_LIBRARY_PATH;/usr/sap/NW1/D03/exe/sapstartsrv pf=/usr/sap/NW1/SYS/profile/NW1_D03_sapaas -D -u nw1adm
    
  6. [2] Create an AAS cluster resource.

    # If using NFS on Azure Files or NFSv3 on Azure NetApp Files
    pcs resource create rsc_sap_NW1_AAS03 SAPInstance InstanceName="NW1_D03_sapaas" \
     START_PROFILE=/sapmnt/NW1/profile/NW1_D03_sapaas \
     op monitor interval=120 timeout=60 \
     --group g-NW1_AAS
    
    # If using NFSv4.1 on Azure NetApp Files
    pcs resource create rsc_sap_NW1_AAS03 SAPInstance InstanceName="NW1_D03_sapaas" \
     START_PROFILE=/sapmnt/NW1/profile/NW1_D03_sapaas \
     op monitor interval=120 timeout=105 \
     --group g-NW1_AAS
    

    Check the status of the cluster.

    sudo pcs status
    
    # Node List:
    #   Node sap-cl1: standby
    #   Online: [ sap-cl2 ]
    #
    # Full list of resources:
    #
    # rsc_st_azure    (stonith:fence_azure_arm):      Started sap-cl2
    #  Resource Group: g-NW1_ASCS
    #      fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started sap-cl2
    #      nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started sap-cl2
    #      vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started sap-cl2
    #      rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started sap-cl2
    #  Resource Group: g-NW1_AERS
    #      fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started sap-cl2
    #      nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started sap-cl2
    #      vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started sap-cl2
    #      rsc_sap_NW1_ERS01  (ocf::heartbeat:SAPInstance):   Started sap-cl2
    #  Resource Group: g-NW1_PAS:
    #      vip_NW1_PAS       (ocf::heartbeat:IPaddr2):        Stopped (disabled)
    #      nc_NW1_PAS        (ocf::heartbeat:azure-lb):       Stopped (disabled)
    #      fs_NW1_PAS        (ocf::heartbeat:Filesystem):     Stopped (disabled)
    #      rsc_sap_NW1_PAS02 (ocf::heartbeat:SAPInstance):    Stopped (disabled)
    #  Resource Group: g-NW1_AAS:
    #      vip_NW1_AAS       (ocf::heartbeat:IPaddr2):        Started sap-cl2
    #      nc_NW1_AAS        (ocf::heartbeat:azure-lb):       Started sap-cl2
    #      fs_NW1_AAS        (ocf::heartbeat:Filesystem):     Started sap-cl2
    #      rsc_sap_NW1_AAS03 (ocf::heartbeat:SAPInstance):    Started sap-cl2
    
  7. Configure a constraint to start the AAS resource group only after the ASCS instance is started.

    sudo pcs constraint order g-NW1_ASCS then g-NW1_AAS kind=Optional symmetrical=false
    

Post configuration for PAS and AAS instances

  1. [1] For PAS and AAS to run on any cluster node (sap-cl1 or sap-cl2), the content in $HOME/.hdb of <sid>adm from both cluster nodes needs to be copied.

    # Check current content of /home/nw1adm/.hdb on sap-cl1
    sap-cl1:nw1adm > ls -ltr $HOME/.hdb
    drwx------. 2 nw1adm sapsys 66 Aug  8 19:11 sappas
    drwx------. 2 nw1adm sapsys 84 Aug  8 19:12 sap-cl1
    # Check current content of /home/nw1adm/.hdb on sap-cl2
    sap-cl2:nw1adm > ls -ltr $HOME/.hdb
    total 0
    drwx------. 2 nw1adm sapsys 64 Aug  8 20:25 sap-cl2
    drwx------. 2 nw1adm sapsys 66 Aug  8 20:26 sapaas
    
    # As PAS and AAS is installed using virtual hostname, you need to copy virtual hostname directory in /home/nw1adm/.hdb
    # Copy sappas directory from sap-cl1 to sap-cl2
    sap-cl1:nw1adm > scp -r sappas nw1adm@sap-cl2:/home/nw1adm/.hdb
    # Copy sapaas directory from sap-cl2 to sap-cl1. Execute the command from the same sap-cl1 host. 
    sap-cl1:nw1adm > scp -r nw1adm@sap-cl2:/home/nw1adm/.hdb/sapaas . 
    
  2. [1] To ensure the PAS and AAS instances don't run on the same nodes whenever both nodes are running, add a negative colocation constraint with the following command:

    sudo pcs constraint colocation add g-NW1_AAS with g-NW1_PAS score=-1000
    sudo pcs node unstandby sap-cl1
    sudo pcs resource enable g-NW1_PAS
    

    The score of -1000 ensures that if only one node is available, both the instances continue to run on the other node. If you want to keep the AAS instance down in such a situation, you can use score=-INFINITY to enforce this condition.

  3. Check the status of the cluster.

    sudo pcs status
    
    # Node List:
    #   Online: [ sap-cl1 sap-cl2 ]
    #
    # Full list of resources:
    #
    # rsc_st_azure    (stonith:fence_azure_arm):      Started sap-cl2
    #  Resource Group: g-NW1_ASCS
    #      fs_NW1_ASCS        (ocf::heartbeat:Filesystem):    Started sap-cl2
    #      nc_NW1_ASCS        (ocf::heartbeat:azure-lb):      Started sap-cl2
    #      vip_NW1_ASCS       (ocf::heartbeat:IPaddr2):       Started sap-cl2
    #      rsc_sap_NW1_ASCS00 (ocf::heartbeat:SAPInstance):   Started sap-cl2
    #  Resource Group: g-NW1_AERS
    #      fs_NW1_AERS        (ocf::heartbeat:Filesystem):    Started sap-cl1
    #      nc_NW1_AERS        (ocf::heartbeat:azure-lb):      Started sap-cl1
    #      vip_NW1_AERS       (ocf::heartbeat:IPaddr2):       Started sap-cl1
    #      rsc_sap_NW1_ERS01  (ocf::heartbeat:SAPInstance):   Started sap-cl1
    #  Resource Group: g-NW1_PAS:
    #      vip_NW1_PAS       (ocf::heartbeat:IPaddr2):        Started sap-cl1
    #      nc_NW1_PAS        (ocf::heartbeat:azure-lb):       Started sap-cl1
    #      fs_NW1_PAS        (ocf::heartbeat:Filesystem):     Started sap-cl1
    #      rsc_sap_NW1_PAS02 (ocf::heartbeat:SAPInstance):    Started sap-cl1
    #  Resource Group: g-NW1_AAS:
    #      vip_NW1_AAS       (ocf::heartbeat:IPaddr2):        Started sap-cl2
    #      nc_NW1_AAS        (ocf::heartbeat:azure-lb):       Started sap-cl2
    #      fs_NW1_AAS        (ocf::heartbeat:Filesystem):     Started sap-cl2
    #      rsc_sap_NW1_AAS03 (ocf::heartbeat:SAPInstance):    Started sap-cl2
    

Test the cluster setup

Thoroughly test your Pacemaker cluster by running the typical failover tests.