SWAP space in Linux VM’s on Windows Azure – Part 1

With the recent GA of IaaS on Windows Azure, many new scenarios are opening up. One such scenario was a recent project where I needed to deploy a multi-node Hadoop Cluster on Linux based machines in Azure. To get started, I used a CentOS 6.3 image from the Azure gallery to provision a medium sized VM and proceeded to deploy a single node core Hadoop. This seemed to work fine except that as I started testing slightly heavier workloads, I noticed that the VM would often freeze up or become unresponsive.

It is not difficult to guess that this had something to do with the limited resources of a medium sized VM – after all it has only 2 CPU cores and 3.5 GB of memory. But, while I would have expected the workloads to run slow, I was not expecting the whole VM to become unresponsive or start dropping connections. After discussing this issue with my friends and colleagues, we determined that the VM did not have Swap (i.e. what is called a page file on Windows) configured at all. Thus, its virtual memory system could not swap to disk when the memory pressure increased.

You can check how the system is doing memory wise by running the “free” command from the Linux shell prompt, and in particular, you can use “cat /proc/swaps” to see the state of the swap space – how much is configured and how much is in use. See the screenshot, below.

If the swap space is not configured at all, as is the default case with Linux VM’s provisioned in Azure IaaS, the “cat /proc/swaps” will return nothing, and likewise the “free” command will not show any activity in swap.

An interesting question is why doesn’t the VM provisioning using a Linux library image (i.e. from the Azure gallery) automatically configure swap space. Perhaps the thinking is that the user should decide on the size and location of the swap and do it post provisioning. However, this information is not documented anywhere, and it is quite possible that one continues to use the VM without the swap ever getting configured till processes begin to crash or the VM freezes up.

That said, once the problem has been diagnosed, the solution is a simple set steps to configure a file based swap on the resource disk; a medium sized IaaS VM in Azure comes with 135 GB of resource disk mounted as “/mnt/resource”. Given below is a walkthrough of the steps for configuring a file based swap space on the VM.

• Use the “fallocate” command to allocate a swap file of suitable size, say, 5GB on the resource disk. The syntax is: “fallocate -l 5g /mnt/resource/swap5g” where “swap5g” is the name of the file
• Change the permissions on the file using “chmod” command so that only the root user has read/write permissions on the swap file. The syntax is: “chmod 600 /mnt/resource/swap5g”
• Use the “mkswap” command to set up the file as swap area. The syntax is: “mkswap /mnt/resource/swap5g”
• Enable the use of the swap file using “swapon” command. The syntax is: “swapon /mnt/resource/swap5g”
• The swap is ready for use now, and the “cat /proc/swaps” command should confirm it now. Add an entry to the “/etc/fstab” file so that even if the VM recycles in Azure, the swap settings are retained. The syntax is: echo “/mnt/resource/swap5g none swap sw 0 0” >> /etc/fstab

Here is a transcript of the above commands executed in my VM.

Acknowledgement: Thanks to my colleague Amit Srivastava for help with troubleshooting and resolving the swap issue.

Comments

• Anonymous
  February 16, 2014
  Thanks for writing this. I was running R in an extra small CentOS image and install.package() was silently failing. I eventually narrowed down the reason with strace -e trace=process, and when I saw I had no swap space, google brought me to this article.

• Anonymous
  February 27, 2014
  That command to remount it does not seem to work. I see that it adds it correctly  to the file but the structure in the file seems to be very different. I tested by rebooting the VM and it did not remount it. Any suggestions?

• Anonymous
  March 03, 2014
  I never got that command to work for the echo it would never recreate the swap or start it on boot so I added this. Which creates it every time the machine starts. sudo -i echo "fallocate -l 8g /mnt/resource/swapfile" >> /etc/rc3.d/S99local echo "chmod 600 /mnt/resource/swapfile" >> /etc/rc3.d/S99local echo "mkswap /mnt/resource/swapfile" >> /etc/rc3.d/S99local echo "swapon /mnt/resource/swapfile" >> /etc/rc3.d/S99local

• Anonymous
  May 23, 2014
  Be advised that the procedure described is completely wrong for Linux VMs created in Azure. The /mnt/resource is mounted by the WALinuxAgent and it is not clear when that is done, so even the suggestion to add the commands to /etc/rc3.d/S99local doesn't work because it creates the file before the /mnt/resource is mounted and in my case I got a "/" mount point with no space left. The correct way is to make the changes in /etc/waagent.conf as described in part 2 of this article (just change "part-1" to "part-2" in the URL). Be sure to delete any files created by the fallocate command before rebooting, the WALinuxAgent will recreate the file /mnt/resource/swapfile in every boot.

• Anonymous
  November 05, 2014
  Sorted my azure vm swap issues out, thank you!

• Anonymous
  November 19, 2014
  This is a create solution! Thank you

• Anonymous
  June 15, 2015
  Thanks for this - a great resource.