Key Vault logging (preview) now available

[Updated: 5/4/2016 - Fixed "What is logged?"]

Hello folks,

Today we are announcing public preview of Azure Key Vault logging. Logging is an often requested feature and several customers have told us that it is very critical for them to have logging available so they can use key vault effectively. We are excited to have this very important feature available as a preview. Over next few weeks we want to log feedback (pun intended) from you to make sure key vault logging meets your needs. So give it a try and let us know.

Below Amit provides detailed instructions on how to enable logging and then access the logs.

As usual, we value your input so please take a moment to follow us, join our advisory board, send us private feedback,and/or visit our forum .   

---

Hello Amit here! So you have been using Key Vault, but you also would like to monitor how/when key vault is accessed, and by who. So let's see how to do that.

When you create a key vault, logging is disabled by default. To turn logging on you need to run a Azure PowerShell command and provide a storage account where the logs will be stored. Once logging is turned on for a key vault, all operations for that key vault will be logged in the storage account you provide. A new container named 'insights-logs-auditevent' will be automatically created under the provided storage account. Same storage account can be used for collecting logs for multiple key vaults. So let's see how to turn logging on for a key vault and then access those logs from the storage account. Later I'll also explain which operations are logged and the log format in detail.

So let's get started.

To enable logging for a key vault, you need to download and install Azure PowerShell 1.0.1 or later. Download it here.

Note: With the Azure PowerShell version 1.0.0 and later, you'll notice a few changes to the Azure Key Vault cmdlets. We're planning to write a separate blog soon to explain all these changes.

Step 1:

First create a key vault if you don't have one already. You can also use an existing key vault. Simply run '$kv = Get-AzureRmKeyVault -VaultName <your vault name>' instead of the commands below and then skip to Step 2.

Step 2:

We also need a storage account to store the logs. You can use existing storage account too. However, remember that you need a storage account created using Azure Resource Manager. A 'classic' storage account created using Azure Service Management interface will not work.

Note that in the above command I've created the storage account under the same resource group as the key vault. This is not required but it makes life easy by putting all the related resources under same resource group.

For logging to work, the storage account and the key vault must be under same subscription.

Step 3:

Now enable logging for our key vault. We need to provide the Resource ID for the resource (key vault in this case) for which we need logs and the resource Id for the storage account. The -Enabled flag set to $true enables logging, and when set to $false disables logging. For Key Vault, we currently have only one category of logs, 'AuditEvent'.

That's it! You're done! Logging is turned on for our key vault 'demo3333' and now all the operations performed on the key vault will be logged to the designated storage account.

So what is logged?

All authenticated REST API requests are logged. This includes failed requests due to access permissions, system errors or bad requests.

Type of operations that are logged:

1. All the key vault operations, like Key Vault create, Key Vault set access policy, Key Vault delete and updating other key vault attributes like tags etc. are logged.
2. All Keys and Secrets operations like Create/Modify/Delete keys/secrets, sign/verify/encrypt/decrypt/wrap/unwrap keys, get secrets, list keys/secrets, list versions of keys/secrets etc. are logged.
3. Unauthenticated requests that result in a 401 response (such as requests without a bearer token, malformed, expired, invalid token) are logged.

[Edited - fixed information about logging of unauthenticated requests]

Accessing logs

Now that we have seen how the logs are enabled, let's turn our attention to how to access these logs. Logs are stored in a container named 'insights-logs-auditevent' in the storage account you provide. Here's how you can list all the blobs in this container.

As you can see, the blobs follow a naming convention that can be summarized as follows:
           resourceId=<ARM resource ID>/y=<year>/m=<month>/d=<day of month>/h=<hour>/m=<minute>/filename.json

Note: All the date/time values used for blob names are UTC.

As mentioned earlier, same storage account can be used to collect logs for multiple resources. Thus the full resourceId in the blob name will be very handy to only access or download blobs that you need. First let us see how to download these blobs then we'll see how to use wild-cards to selectively download blobs that meet specific criteria.

First we create a directory where we would like to download our blobs.

Now get a list of all blobs and pipe it through the 'Get-AzureStorageBlobContent' to download the blobs into our destination directory. The '/' delimiter in the blob names will be used to create a full directory structure under the destination directory to download and store the blobs as files.

In the above example, you'll notice that logs for two different key vaults are listed: demo3333 and demo7815. If you want to download logs for only 'demo7815' key vault, use a wild-card pattern for parameter -Blob.

On similar lines, if you want to download logs for all the resources in a resource group, use -Blob '*/RESOURCEGROUPS/<resource group name>/*' pattern.

To download all the logs for the month of November 2015, use -Blob '*/year=2015/m=11/*' and so on.

Other cmdlets for Diagnostic Settings

To disable logging for a resource, use following command:

To query status of diagnostic settings for a resource, use following command:

Log format

Individual blobs are stored as text formatted as a JSON blob. Here's an example of a log entry resulting from a 'Get-AzureRmKeyVault -VaultName demo3333' cmdlet.

Let's look at the fields and what they mean.

Frequently Asked Questions with Answers

Q: How soon will the logs be available after a key vault operation has taken place?
A: SLA is 10 minutes. However we are targeting close to 6 minutes for most cases.

Q: How long will the logs be retained in the storage account?
A: Once the logs are written to your storage account, it's up to you to retain or delete them. In future we will add a feature that allows a user to set retention period so that logs older than 'X' days will be automatically pruned. Stay tuned.

Q: How do I control who can access the logs?
A: Logs are stored in a standard Azure Storage account. You can use standard Azure access control methods to limit access to this storage account.

Comments are disabled, head over to the Azure Key Vault forum to discuss about this blog.