Data Management Instance Scaling in Azure Data Explorer and VM Reservation Implications

Wes Verhagen 25 Reputation points
2024-07-17T13:11:18.4566667+00:00

Hello community,

I have a question regarding the scaling of Data Management instances in Azure Data Explorer (ADX), focusing on VM types and cost implications.

Recently, our ADX setup unexpectedly utilized D2 v2 VMs for data management instead of the anticipated D1 v2 VMs, resulting in higher costs. Similarly, I've noticed variations in VM usage (D1, D2, D3) within our cost management reports over different months.

My query is twofold:

  1. What factors influence the scaling of Data Management instances within the same engine SKU in ADX? We intended to use D1 v2 consistently but have observed changes to D2 v2 and D3 v2.
  2. Given this variability, how can we accurately predict and manage our budget? Is it possible to purchase VM reservations for specific types (e.g., D1 v2) if instances may change dynamically?

Any insights or best practices on these topics would be greatly appreciated.

Thank you!

Best regards,
Wes Verhagen

Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
519 questions
Accepted answer
  1. Bhargava-MSFT 30,391 Reputation points Microsoft Employee
    2024-07-17T20:26:27.3866667+00:00

    Hello Wes Verhagen,

    Welcome to the Microsoft Q&A forum.

    It seems like you have configured the ADX cluster with "Optimized autoscale" https://video2.skills-academy.com/en-us/azure/data-explorer/manage-cluster-horizontal-scaling?WT.mc_id=Portal-Microsoft_Azure_Kusto#optimized-autoscale-recommended-option

    Optimized autoscale:

    Optimized Autoscale is a built-in feature that helps clusters perform their best when demand changes. You can choose to scale your cluster manually to a specific instance count, or via a custom Optimized Autoscale policy that scales based on metric(s) thresholds. Optimized Autoscale enables your cluster to be performant and cost effective by adding and removing instances based on demand

    • If the cluster is underutilized, it is scaled in to lower cost without affecting the required performance.
    • If the cluster is overutilized, it is scaled out to maintain optimal performance

    In your case, the changes in VM types (from D1 v2 to D2 v2 and D3 v2) are likely due to the Optimized Autoscale feature responding to varying demands on your cluster.

    To have more control over the scaling and potentially reduce costs, you might want to look into the custom autoscale feature. Custom autoscale allows you to set specific policies and thresholds for scaling based on your needs.

    https://video2.skills-academy.com/en-us/azure/data-explorer/manage-cluster-horizontal-scaling?WT.mc_id=Portal-Microsoft_Azure_Kusto#custom-autoscale

    I hope this helps. Please let me know if you have any further questions.

    2 people found this answer helpful.
    0 comments

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.