Sentinel _BilledSize and estimate_data_size differences

Sándor Tőkési 181

hey folks

Could somebody tell me the relationship between the _BilledSize field in a log and the result of the estimate_data_size(*) KQL command?

I do understand that the _BilledSize field contains the info of the size of the data I have to pay for in Sentinel.
The estimate_data_size() command seemingly calculates the size of the whole log including the free fields as well (like the _BilledSize field, Type field, etc, etc). Based on this I assumed that the _BilledSize field is going to be equal or close to the estimate_data_size() minus the size of the free fields.

But when I check the difference in my logs I can see different results. For example in the AuditLog table in some cases the value in the _BilledSize field is more 20% bigger than the result of the estimate_data_size() minus the free fields (or 18%+ bigger if I don't remove the free fields). But in case of other events in the same table the _BilledSize field contains a value 10% smaller than the estimate_data_size() - the free fields.

Does anybody has an explanation of how the _BilledSize field is calculated, and how the estimate_data_size(*) function works compared to this? I'm also curious why the difference between the two is so big and so (seemingly) unpredictable.

I can see these differences in default tables without any ingestion_time filter (DCR) in place, or any modification in the schema.

Marilee Turscak-MSFT 36,411 Reputation points Microsoft Employee

2024-07-09T23:14:55.7933333+00:00

@Sándor Tőkési ,

The _BilledSize column specifies the size in bytes of data that will be billed to your Azure account if _IsBillable is true. The estimate_data_size() returns the size in bytes of the selected columns of a tabular expression in Azure Data Explorer.

The documentation notes that the "The billable data volume calculation is generally substantially smaller than the size of the entire incoming JSON-packaged event. On average, across all event types, the billed size is around 25 percent less than the incoming data size. It can be up to 50 percent for small events."

So it sounds more likely that the columns you're selecting are not equal and you're looking at different sets of data, but I'm adding the Azure Data Explorer and Azure Monitor tags to this thread to help clarify.
Sándor Tőkési 181 Reputation points

2024-07-11T09:16:11.9366667+00:00

thanks @Marilee Turscak-MSFT
I use the estimate_data_size() command with the asterisk attribute, that should calculate the whole event. (I believe the asterisk was used as markdown command in my original request). But just to be clear I use the command with the asterisk attribute meaning I calculate the bytes of the whole event.

I do understand the the log will be smaller than the network 'packet' (raw log) due to parsing. But this does not explain why the _BilledSize field is significantly different than the result of the output of the estimate_data_size(*).

The _BilledSize specifies the bytes I have to pay for.
And the estimate_data_size(*) minus the size of the free fields should be exactly the same, bytes I have to pay for. So, the size of the whole event in bytes minus the size of free fields should be the bytes I have to pay for - which is the _BilledSize field.

But again, sometimes the _BilledSize field is significantly smaller than this and sometimes it is significantly bigger for different events in the same table. (even for default tables, without any ingestion time modification).

Share via

Sentinel _BilledSize and estimate_data_size differences