I have recently added a data connector for AKS to my Sentinel workspace and it has caused a major hike in the amount of logs ingested in the workspace (which eventually increases the costs as well) I want to know: How can I check which tables are ingesting logs from AKS? How can I check what type of data is coming in from AKS Can I somehow configure the type and size of logs coming in from AKS?

Hello, this is to answer Q1, which will help us answer Q2 & Q3, you can use a query such as this, or one of the workbooks like "Workspace Usage". Adjust query of you need more than 30days of data. Usage | make-series Gbytes=sum(Quantity)/1000 default=0 on TimeGenerated from ago(30d) to now() step 1d by DataType | render columnchart Then from the chart you should see on and after the date you enabled AKS which Table grew in size (the bar size will be larger). Example output for AzureDiagnostics , my data is pretty stable, but your looking for a a bar to be growing or a brand new Table.

Sorry for the delayed, reply, that connector https://github.com/Azure/Azure-Sentinel/tree/master/Solutions/Azure%20kubernetes%20Service shows that the data is located in AzureDiagnostics and the category that the two hunting queries use is kube-audit e.g. AzureDiagnostics | where Category == "kube-audit" So if you adjust my query for that: AzureDiagnostics | where Category startswith "kube" | make-series Gbytes=sum(_BilledSize)/(1024*1024*1024) default=0 on TimeGenerated from ago(30d) to now() step 1d by Category, ResourceProvider | render columnchart You will be able to see that Category and others (amend line 2 if you dont, to be "kube-audit"), or remove the line if you want to see all data categories. You can then further refine by pod or container id? AzureDiagnostics | where Category == "kube-audit" | summarize Gbytes=sum(_BilledSize)/(1024*1024*1024) by containerID_s //, pod_s

How can I analyze the logs coming from AKS and reduce them?

Najam ul Saqib 280

I have recently added a data connector for AKS to my Sentinel workspace and it has caused a major hike in the amount of logs ingested in the workspace (which eventually increases the costs as well)

I want to know:

How can I check which tables are ingesting logs from AKS?
How can I check what type of data is coming in from AKS
Can I somehow configure the type and size of logs coming in from AKS?

Akshay kumar Mandha 385 Reputation points Microsoft Vendor

2024-09-04T17:10:48.6766667+00:00

Hi Najam ul Saqib,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here

I was Just checking in to see, If you got a chance to see the answer posted by Clive Watson in resolving the issue

If it was helpful, please click "Upvote and Accept Answer" on his post to let us know.
We're here to help, so if you have any further questions, don't hesitate to ask.

Thank You..!
Akshay kumar Mandha 385 Reputation points Microsoft Vendor

2024-09-05T17:22:16.3666667+00:00

Hi Najam ul Saqib,

I wanted to follow up If you got a chance to see the answer posted by Clive Watson in resolving the issue. If it was helpful, please click "Upvote and Accept Answer" on his post to let us know.

Thank you..!

2 answers

Clive Watson 6,351 Reputation points MVP

2024-09-04T09:05:57.49+00:00
Hello, this is to answer Q1, which will help us answer Q2 & Q3, you can use a query such as this, or one of the workbooks like "Workspace Usage". Adjust query of you need more than 30days of data.

Usage | make-series Gbytes=sum(Quantity)/1000 default=0 on TimeGenerated from ago(30d) to now() step 1d by DataType | render columnchart

Then from the chart you should see on and after the date you enabled AKS which Table grew in size (the bar size will be larger). Example output for AzureDiagnostics, my data is pretty stable, but your looking for a a bar to be growing or a brand new Table.
Please sign in to rate this answer.
Najam ul Saqib 280 Reputation points

2024-09-09T08:02:13.22+00:00

@Clive Watson Thank you, I can just see AzureDiagnostics logs in my case and no other logs are there. I am still unsure on how can I reduce the logs coming in from AKS i.e. Q3 above.

Akshay kumar Mandha 385 Reputation points Microsoft Vendor

2024-09-09T15:12:47.4033333+00:00

Hi Najam ul Saqib,
Thank you for getting back to us.

Based upon your query, you can do this by going to your AKS cluster in the Azure portal, clicking on "Diagnostic settings" under the "Monitoring" section, and then adding a new diagnostic setting. From there, you can choose which log categories you want to collect, such as ContainerLogs or PodLogs, and where you want to send them, like a Log Analytics workspace.

You can also use the Azure CLI to create a diagnostic setting. For example, you can use a command like az monitor diagnostic-settings create to create a new setting that collects specific log categories and sends them to a Log Analytics workspace.

By configuring diagnostic settings, you can reduce the amount of logs coming in from AKS and only collect the data that's relevant to you

If it was helpful, please click "Upvote and Accept Answer" on his post to let us know.!

Akshay kumar Mandha 385 Reputation points Microsoft Vendor

2024-09-10T17:33:41.5166667+00:00

Hi Najam ul Saqib,
I wanted to follow up If you got a chance to see my comment for resolving the issue. If it was helpful, please click "Upvote and Accept Answer" on his post to let us know.

Najam ul Saqib 280 Reputation points

2024-09-12T01:50:15.97+00:00

@Akshay kumar Mandha but in my case I am using an AKS data connector from Sentinel and not some manual configuration to send logs to the workspace.

Also, from what I understand from your reply, I can see that there are different types of logs in AKS i.e. ContainerLogs, PodLogs, etc but when I see in the workspace chart as mentioned above, I only see AzureDiagnostic logs and no further distinction is made. Why?

Akshay kumar Mandha 385 Reputation points Microsoft Vendor

2024-09-12T11:05:41.8833333+00:00

Hi Najam ul Saqib,
Thank you for getting back to us We are looking into it and will get back to you soon.

Akshay kumar Mandha 385 Reputation points Microsoft Vendor

2024-09-12T14:28:37.8733333+00:00

Hi Najam ul Saqib,
Thank you for your patience while we are reviewing the issue. For AKS data connector from Sentinel is a pre-configured connector that streams AKS diagnostics logs into Microsoft Sentinel. To see other log types like ContainerLogs, PodLogs, etc., you might need to configure the log collection settings in the AKS data connector to include those log types. Additionally, you should also check your workspace settings to ensure that the other log types are not being filtered out or hidden. For more more please do refer this documentation
If it was helpful, please click "Upvote" on my comment let us know, if have any query.!

Akshay kumar Mandha 385 Reputation points Microsoft Vendor

2024-09-13T17:01:01.2566667+00:00

Hi Najam ul Saqib,
Just wanted to check if you had chance to review my comment as well as Clive Watson also provided the some solution regarding the query and logs If it is found helpful, Could you please click an "Upvote" and accept the answer posted by Clive Watson
Thank you..!
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Clive Watson 6,351 Reputation points MVP

2024-09-12T15:59:30.46+00:00
Sorry for the delayed, reply, that connector https://github.com/Azure/Azure-Sentinel/tree/master/Solutions/Azure%20kubernetes%20Service shows that the data is located in AzureDiagnostics and the category that the two hunting queries use is kube-audit

e.g.

AzureDiagnostics

| where Category == "kube-audit"

So if you adjust my query for that:

AzureDiagnostics | where Category startswith "kube" | make-series Gbytes=sum(_BilledSize)/(1024*1024*1024) default=0 on TimeGenerated from ago(30d) to now() step 1d by Category, ResourceProvider | render columnchart

You will be able to see that Category and others (amend line 2 if you dont, to be "kube-audit"), or remove the line if you want to see all data categories.

You can then further refine by pod or container id?

AzureDiagnostics | where Category == "kube-audit" | summarize Gbytes=sum(_BilledSize)/(1024*1024*1024) by containerID_s //, pod_s
Please sign in to rate this answer.
Najam ul Saqib 280 Reputation points

2024-09-15T14:23:11.96+00:00

@Clive Watson this is helpful, and last concern, if too many logs are coming in from AKS can they be reduced somehow?
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How can I analyze the logs coming from AKS and reduce them?

2 answers

Your answer