Ingesting custom logs from VM to custom table

Justin Sylvestre 0 Reputation points
2024-06-27T19:20:12.4733333+00:00

Hello,

I'm having issues ingesting a custom log file from my IIS-enabled VM to a log analytics workspace. The steps I've taken to resolve this issue include:

-Verified that the extension was successfully installed and provisioned

-Verified that the extension (Azure Monitor Agent) was running by querying a heartbeat

-Verified that the DCR exists and is associated with the virtual machine (My DCR has an .XML file in the VM)

-Verified that the agent was able to download the associated DCR(s) from AMCS service (DCR's are located in config chunks)

My goal is to collect W3C logs from a C# site (deployed with nlog). The app already exports the logs to a logfile, and I've created a custom table in my log analytics workspace with similar columns. My DCR points to the custom logs folder and has an endpoint setup. Despite this, my custom table cannot ingest any data.

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
2,960 questions
Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
7,461 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Sina Salam 6,501 Reputation points
    2024-06-27T22:57:32.0566667+00:00

    Hello Justin Sylvestre,

    Welcome to the Microsoft Q&A and thank you for posting your questions here.

    Problem

    I understand that you are having challenges to collect W3C logs from a C# site (deployed with nlog) and you would need to ensure your custom table can ingest data.

    Solution

    You have done a lot. Well done. To troubleshoot and successfully collect W3C logs from your C# site and ensure your custom table in the log analytics workspace ingests data, review this these steps carefully:

    1. Though, you mentioned DCR is well connected and configure. Kindly review this sample code snippet (JSON for DCR) to ensure your DCR is configured to collect custom logs from the specified path. 
    {
     "kind": "Linux",
     "properties": {
       "dataSources": {
         "customLogs": [
           {
             "name": "W3CLogs",
             "directory": "C:\\logs\\",
             "filter": "w3c_logfile.log",
             "format": "W3C"
           }
         ]
       },
       "destinations": {
         "logAnalytics": {
           "workspaceId": "<your_workspace_id>"
         }
       }
     }
   }

    Make sure the log path (C:\logs\w3c_logfile.log) matches the path you configured in your Data Collection Rule (DCR).

    1. Since DCR was downloaded and applied by the agent. Use the following Kusto Query Language (KQL) query to check if data is being ingested into your custom table in Log Analytics:
    CustomTable_CL
| where TimeGenerated > ago(1h)
| sort by TimeGenerated desc

    Replace CustomTable_CL with the name of your custom table.

    https://video2.skills-academy.com/en-us/azure/data-explorer/kusto/query/

    1. Ensure the Azure Monitor Agent has read permissions for the log directory and files. https://video2.skills-academy.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-overview
    2. Review Azure Monitor Agent logs for any errors related to log collection or DCR application. https://video2.skills-academy.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-troubleshoot
    3. Double-check the DCR configuration for any syntax errors or incorrect paths. https://video2.skills-academy.com/en-us/azure/azure-monitor/agents/data-collection-rule-azure-monitor-agent

    Kindly utilize the links above for more detailed steps and information.

    Accept Answer

    I hope this is helpful! Do not hesitate to let me know if you have any other questions.

    ** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

    Best Regards,

    Sina Salam

    0 comments