Rotating ADE KEK, adds two new secrets (Wrapped BEKs) for the same VM

Vasantha Raman A 21 Reputation points
2022-03-08T14:28:21.167+00:00

Hello All,

We enabled ADE for OS+DATA disk for two VMs (RHEL8 - with no data disk attached) with same KEK, using
az vm encryption enable -n vmname -g rsgrp --key-encryption-key kek --disk-encryption-keyvault keyvault-name --volume-type ALL --encrypt-format-all

We tried rotating the KEK for the two VMs by calling the same command above again, now with a newer version of the same KEK. On doing so, we observed that after some time delay, two more secrets of type Wrapped BEKs in addition to existing ones are added for each VM (initially two Wrapped BEK secrets were present and after rotation of KEK it became six). We were expecting just one new secret. Unwrapped values of both the secrets were same. Now, even though the vm encryption status shows only one of those as the ADE secret (obtained by az vm encryption show -n vmname) . If we try to disable/delete the other newly obtained secret, the VM fails to start after a stop, citing "Failed to restart the virtual machine 'vmname'. Error: Error encountered when retrieving secret from the Key Vault with URL: https://kevaultname.vault.azure.net/secrets/secretname/secretversion. Make sure that the secret exists and Key Vault is enabled for volume encryption", even though the secret mentioned is very much present in the vault and is enabled.

Now, Questions are:

  1. Is this a bug or the system is designed this way or maybe, the way we handle key rotation is wrong?
  2. If it is designed this way, will we be charged for the secrets stored in the vault?
Azure Disk Encryption
Azure Disk Encryption
An Azure service for virtual machines (VMs) that helps address organizational security and compliance requirements by encrypting the VM boot and data disks with keys and policies that are controlled in Azure Key Vault.
174 questions
{count} votes

Accepted answer
  1. Sumarigo-MSFT 46,126 Reputation points Microsoft Employee
    2022-03-17T07:29:50.533+00:00

    @Vasantha Raman A Key rotation you should expect to see new secrets generated not only per VM but per volume, in some cases you will see new versions of the secret instead of new secrets, but at the end they are treated on a very similar way, the encryption settings on each drive will be updated to point to the new secrets.

    If you so desire to clean up the KeyVault of unused secrets the recommendation is to check all the encryption settings of all the encrypted VMs and correlate with the current content of each specific KeyVault
    so you can be sure that you are not deleting a secret or a particular version that is being used, then we strongly encourage you to take a backup of the secret just in case that a restore is needed,
    after a secret deletion is a good idea to test out if the VM that was previously associated with it can safely be rebooted,
    if not then the secret should be restored and then another case with us can be open to check the reason why the secret on the encryption settings was not updated.

    We’re currently looking into the key rotation process and trying to identify any possible causes for your VMs to not start while deleting the secret that was not the one that is on the disk encryption settings for each individual disk.

    About the costs, I checked and the information is publicly available here: https://azure.microsoft.com/en-us/pricing/details/key-vault/, the costs associated with secrets is based on secret operations, you can check the cost there by region and currency, if you need more details about that you can reach out to the sales team and they should be able to clarify.

    Please let us know if you have any further queries. I’m happy to assist you further.

    ----------

    Please do not forget to 183996-screenshot-2021-12-10-121802.pngand “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

    0 comments No comments

1 additional answer

Sort by: Most helpful
  1. Sumarigo-MSFT 46,126 Reputation points Microsoft Employee
    2022-03-10T07:18:25.247+00:00

    @Vasantha Raman A Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

    When you perform the Key rotation, a new secret will be added, and what it seems to be happening on a preliminary inspection is that you are getting one secret per volume type after the key rotation, the new secrets are associated with each individual disk and should not be deleted, that will prevent Azure to actually start the VM.

    You can check which secret and key are associated with each individual disk by executing the commands posted here: https://video2.skills-academy.com/en-us/azure/virtual-machines/linux/how-to-verify-encryption-status#single-pass-1 the newly created secret and Key should always exists and be valid, enabled.

    Currently the Key rotation process doesn’t delete the Keys nor secrets on the Key vault and that is to prevent breaking other resources that may be using that Key or Secret, You should be able to delete a secret that is not associated with any VM and that will not brake any VM or prevent it from booting.

    Are you trying to delete the newly created secret and that would be reason for VMs are breaking.

    Please let us know if you have any further queries. I’m happy to assist you further.

    ----------

    Please do not forget to 181802-screenshot-2021-12-10-121802.png and 181803-image.png” wherever the information provided helps you, this can be beneficial to other community members.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.