Cold storage query pricing

Hakkarainen Antti 6 Reputation points
2020-10-06T12:31:13.35+00:00

I am evaluating Time Series Insights for my 3rd party UI application that would use the TSI Query API to get data for analytics.

My fear is that eventually the cost of cold storage queries will grow too large, because my users frequently need to query for data that is older than the maximum retention period (31 days) of warm storage.

Right now I am ingesting approximately one million events every day and their storage size in TSI seems to be about 200 Bytes per event. This totals in 73GB per year.

If I understand TSI pricing correctly, cold scanning this amount of data costs 73*0,009€ = 0,657€. What does "scanning cold storage" actually mean? Does it mean that if I query with search parameters that span for whole year and all the time series, the cost for this one query will be 0,657€, even if the query returns a lot smaller aggregare result set?

Obviously a user would never query all the time series like that, but I might have 1000 users who query a bit smaller sets several times every day, and I fear that the total cost grows too large.

Any ideas how to better manage the costs in this kind of scenario, where I have to support lots of queries to old events?

Azure Time Series Insights
Azure Time Series Insights
An Azure internet of things (IoT) analytics platform to monitor, analyze, and visualize industrial IoT analytics data at scale.
78 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. António Sérgio Azevedo 7,666 Reputation points Microsoft Employee
    2020-10-09T15:36:25.033+00:00

    Thank you for your clarification in comments @Hakkarainen Antti .

    These are the pricing details for Time Series Insights Gen2: https://azure.microsoft.com/en-us/pricing/details/time-series-insights/

    31194-image.png

    Assuming you configure all monthly data for warm and cold data analysis and the aggregation of cold queries scanned only 6GB of data:

    1. Your ingress data is 6GB/month = €24.824 per month
    2. Your Warm Storage is 6 GB*€2.530/GB = €15.18 per month
    3. Your Cold Storage is 6 GB*€0.026/GB = €0.156 (Uses Blob pricing) per month
    4. Warm Queries = No charge
    5. Cold Queries = 6 GB*€0.009/GB = €0.054 per month

    Total = €40.214 per month

    To your main question

    The most important thing I would like to know is what does "cold storage scanning" mean in pricing info.

    It means that your cold queries (how much data you needed to parse) will dictate the pricing in the end of the month (not the results of the query). Please leverage the following doc Plan and manage costs for Azure Blob storage - there you can learn how to optimize the storage account so that you pay only for what you need. Use cost management features to set budgets and monitor costs, review forecasted costs, and monitor spending trends to identify areas where you might want to act. You can also create budgets to manage costs and create alerts that automatically notify stakeholders of spending anomalies and overspending risks.

    Let me know if this information is helpful and let you rest assured that the total cost will not grow too large without your consent?

    Thanks!

    1 person found this answer helpful.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.