Retrieving Transaction Data For Azure Storage Account using SDK

Graham, James 0 Reputation points
2024-08-30T01:18:41.34+00:00

I am trying to get a rough estimate for moving Azure ADLS storage accounts from LRS to GRS & ZRS.

Firstly, I need to identify write and read operations for our storage accounts.

Am I correct in saying that write and read operations (Defined on the azure blob storage pricing page) are the same thing as "transactions" (Defined on a storage account's Metrics page?

Assuming the above answer is yes, I have been using the python SDK ( azure.mgmt.monitor ) to retrieve transaction data for a storage account, but the data that I am getting back does not match what I am seeing in the Azure monitor UI.

In the azure monitor UI I generated a report for 6/29/2024, and counted the transactions for Blob, File, Queue & Table Storage, which came out to 93.

User's image

If I run the below query using the azure monitor sdk then I get two rows returned for 6/29/2024, both with the exact same timestamp but two different "total" amounts:

In the below query the startDate is June 29 2024 00:00, and the endDate is June 29 2024 11:59User's image

User's image

I'm not sure how to interpret the results, though I do notice that if I subtract the smaller total from the larger one I get 95, which is very close the manual transaction count that I did in the Azure monitor UI.

Please verify how I can get the daily transaction count for a storage account using the Azure sdk.

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,217 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Nehruji R 8,066 Reputation points Microsoft Vendor
    2024-08-30T17:28:08.74+00:00

    Hello Graham, James,

    Greetings! Welcome to Microsoft Q&A Platform.

    For Moving Azure Data Lake Storage (ADLS) accounts from Locally Redundant Storage (LRS) to Geo-Redundant Storage (GRS) or Zone-Redundant Storage (ZRS) involves several steps and considerations.

    For Migration

    Storage Account Size: The total size of your storage account will impact the migration time.

    Number of Files: Large numbers of files can increase the complexity of the migration.

    Read/Write Operations: High throughput or large-scale operations may require careful planning.

    Verify that moving to GRS or ZRS complies with your data governance policies. Back up important data if needed. GRS and ZRS have different pricing compared to LRS. Check the cost implications.

    Azure CLI: Use Azure CLI to update the replication setting

    az storage account update --name <storage-account-name> --resource-group <resource-group-name> --sku <new-sku>

    For Small Accounts (up to a few TB):

    • GRS: Several hours to a day. The asynchronous nature of GRS means replication time can vary.
    • ZRS: Typically, quicker, usually within a few hours, since it involves replication within the same region.
    • Cost Impact: Be aware of cost changes due to different replication types and the data transfer involved.

    To get the daily transaction count for a storage account using the Azure SDK, you can use the Azure Monitor service, which collects and analyses metrics and logs. The Azure SDK can interact with Azure Monitor to retrieve this information

    Azure Monitor Metrics: Azure Monitor metrics are required to get transaction counts

    Set Up Your Azure Credentials: Ensure you have the necessary credentials to access Azure resources. This might involve setting up environment variables or using a configuration file.

    Install Azure SDK: Install the Azure SDK package for your language if you haven’t already. For example, in Python, you would install the azure-monitor-query package.

    Below is an example using Python to get the daily transaction count for an Azure Storage Account: from azure. Identity import Default Azure Credential

    from azure. monitor. query import Metrics Query Client.

    # Initialize the client
    credential = DefaultAzureCredential()
    client = MetricsQueryClient(credential)
     
    # Set parameters
    subscription_id = 'your-subscription-id'
    resource_group = 'your-resource-group'
    storage_account_name = 'your-storage-account'
    resource_id = f'/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.Storage/storageAccounts/{storage_account_name}'
     
    # Query the metric
    response = client.query(
      resource_id=resource_id,
      metric_names=['TransactionCount'],
      timespan='P1D',  # Last 1 day
      interval='PT1H'  # Hourly granularity
    )
    # Print results
    for metric in response.metrics:
      print(f"Metric: {metric.name}")
       for time_series_element in metric.timeseries:
       for data in time_series_element.data:
              print(f"Timestamp: {data.timestamp}, Value: {data.total}")
    
    

    Hope this answer helps! please let us know if you have any further queries. I’m happy to assist you further.

    Please "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.