Collect logs from text files with the Azure Monitor Agent and ingest to Microsoft Sentinel

Create data collection rule (DCR)

To get started, open either the Custom Logs via AMA data connector in Microsoft Sentinel and create a data collection rule (DCR).

1. For Microsoft Sentinel in the Azure portal, under Configuration, select Data connectors.
   For Microsoft Sentinel in the Defender portal, select Microsoft Sentinel > Configuration > Data connectors.

2. Type custom in the Search box. From the results, select the Custom Logs via AMA connector.

3. Select Open connector page on the details pane.

   

4. In the Configuration area, select +Create data collection rule.

   

5. In the Basic tab:

   • Type a DCR name.
   • Select your subscription.
   • Select the resource group where you want to locate your DCR.

   

6. Select Next: Resources >.

Define VM resources

In the Resources tab, select the machines from which you want to collect the logs. These are either the machines on which your application is installed, or your log forwarder machines. If the machine you're looking for doesn't appear in the list, it might not be an Azure VM with the Azure Connected Machine agent installed.

1. Use the available filters or search box to find the machine you're looking for. Expand a subscription in the list to see its resource groups, and a resource group to see its VMs.

2. Select the machine that you want to collect logs from. The check box appears next to the VM name when you hover over it.

   

   If the machines you selected don't already have the Azure Monitor Agent installed on them, the agent is installed when the DCR is created and deployed.

3. Review your changes and select Next: Collect >.

Configure the DCR for your application

1. In the Collect tab, select your application or device type from the Select device type (optional) drop-down box, or leave it as Custom new table if your application or device isn't listed.

2. If you chose one of the listed applications or devices, the Table name field is automatically populated with the right table name. If you chose Custom new table, enter a table name under Table name. The name must end with the _CL suffix.

3. In the File pattern field, enter the path and file name of the text log files to be collected. To find the default file names and paths for each application or device type, see Specific instructions per application type. You don't have to use the default file names or paths, and you can use wildcards in the file name.

4. In the Transform field, if you chose a custom new table in step 1, enter a Kusto query that applies a transformation of your choice to the data.

   If you chose one of the listed applications or devices in step 1, this field is automatically populated with the proper transformation. DO NOT edit the transformation that appears there. Depending on the chosen type, this value should be one of the following:

   • source (the default—no transformation)
   • source | project-rename Message=RawData (for devices that send logs to a forwarder)

5. Review your selections and select Next: Review + create.

Review and create the rule

After you complete all the tabs, review what you entered and create the data collection rule.

1. In the Review and create tab, select Create.

   

   The connector installs the Azure Monitor Agent on the machines you selected when creating your DCR.

2. Check the notifications in the Azure portal or Microsoft Defender portal to see when the DCR is created and the agent is installed.

3. Select Refresh on the connector page to see the DCR displayed in the list.

Install the Azure Monitor Agent

Follow the appropriate instructions from the Azure Monitor documentation to install the Azure Monitor Agent on the machine hosting your application, or on your log forwarder. Use the instructions for Windows or for Linux, as appropriate.

• Install the AMA using PowerShell
• Install the AMA using the Azure CLI
• Install the AMA using an Azure Resource Manager template

Create Data Collection Rules (DCRs) using the Azure Monitor Logs Ingestion API. For more information, see Data collection rules in Azure Monitor.

Create the data collection rule

Use the following ARM template to create or modify a DCR for collecting text log files:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "resources": [
        {
            "type": "Microsoft.Insights/dataCollectionRules",
            "name": "{DCR_NAME}",
            "location": "{DCR_LOCATION}",
            "apiVersion": "2022-06-01",
            "properties": {
                "streamDeclarations": {
                    "Custom-Text-{TABLE_NAME}": {
                        "columns": [
                            {
                                "name": "TimeGenerated",
                                "type": "datetime"
                            },
                            {
                                "name": "RawData",
                                "type": "string"
                            },
                        ]
                    }
                },
                "dataSources": {
                    "logFiles": [
                        {
                            "streams": [
                                "Custom-Text-{TABLE_NAME}"
                            ],
                            "filePatterns": [
                                "{LOCAL_PATH_FILE_1}","{LOCAL_PATH_FILE_2}"
                            ],
                            "format": "text",
                            "name": "Custom-Text-{TABLE_NAME}"
                        }
                    ]
                },
                "destinations": {
                    "logAnalytics": [
                        {
                            "workspaceResourceId": "{WORKSPACE_RESOURCE_PATH}",
                            "workspaceId": "{WORKSPACE_ID}",
                            "name": "workspace"
                        }
                    ]
                },
                "dataFlows": [
                    {
                        "streams": [
                            "Custom-Text-{TABLE_NAME}"
                        ],
                        "destinations": [
                            "DataCollectionEvent"
                        ],
                        "transformKql": "source",
                        "outputStream": "Custom-{TABLE_NAME}"
                    }
                ]
            }
        }
    ]
}

Replace the {PLACE_HOLDER} values with the following values:

Associate the DCR with the Azure Monitor Agent

If you create the DCR using an ARM template, you still must associate the DCR with the agents that will use it. You can edit the DCR in the Azure portal and select the agents as described in Define VM resources.