RestApiPoller data connector reference for the Codeless Connector Platform

To create a RestApiPoller data connector with the Codeless Connector Platform (CCP), use this reference as a supplement to the Microsoft Sentinel REST API for Data Connectors docs.

Each dataConnector represents a specific connection of a Microsoft Sentinel data connector. One data connector might have multiple connections, which fetch data from different endpoints. The JSON configuration built using this reference document is used to complete the deployment template for the CCP data connector.

For more information, see Create a codeless connector for Microsoft Sentinel.

Data Connectors - Create or update

Reference the Create or Update operation in the REST API docs to find the latest stable or preview API version. The difference between the create and the update operation is the update requires the etag value.

PUT method

https://management.azure.com/subscriptions/{{subscriptionId}}/resourceGroups/{{resourceGroupName}}/providers/Microsoft.OperationalInsights/workspaces/{{workspaceName}}/providers/Microsoft.SecurityInsights/dataConnectors/{{dataConnectorId}}?api-version={{apiVersion}}

URI parameters

For more information about the latest API version, see Data Connectors - Create or Update URI Parameters.

Request body

The request body for a RestApiPoller CCP data connector has the following structure:

{
   "name": "{{dataConnectorId}}",
   "kind": "RestApiPoller",
   "etag": "",
   "properties": {
        "connectorDefinitionName": "",
        "auth": {},
        "request": {},
        "response": {},
        "paging": "",
        "dcrConfig": ""
   }
}

RestApiPoller

RestApiPoller represents an API Poller CCP data connector where you customize paging, authorization and request/response payloads for your data source.

Authentication configuration

The CCP supports the following authentication types:

• Basic
• APIKey
• OAuth2
• Jwt

As a best practice, use parameters in the auth section instead of hard-coding credentials. For more information, see Secure confidential input.

In order to create the deployment template which also uses parameters, you need to escape the parameters in this section with an extra starting [. This allows the parameters to assign a value based on the user interaction with the connector. For more information, see Template expressions escape characters.

To enable the credentials to be entered from the UI, the connectorUIConfig section requires instructions with the desired parameters. For more information, see Data connector definitions reference for the Codeless Connector Platform.

Basic auth

Example Basic auth using parameters defined in connectorUIconfig:

"auth": {
    "type": "Basic",
    "UserName": "[[parameters('username')]",
    "Password": "[[parameters('password')]"
}

APIKey

APIKey auth examples:

"auth": {
    "type": "APIKey",
    "ApiKey": "[[parameters('apikey')]",
    "ApiKeyName": "X-MyApp-Auth-Header",
    "ApiKeyIdentifier": "Bearer"
}

This example results in the secret defined from user input sent in the following header: X-MyApp-Auth-Header: Bearer apikey

"auth": { 
    "type": "APIKey",
    "ApiKey": "123123123",
}

This example uses the default values and results in the following header: Authorization: token 123123123

"auth": { 
    "type": "APIKey",
    "ApiKey": "123123123",
    "ApiKeyName": ""
}

Since the ApiKeyName is explicitly set to "", the result is the following header: Authorization: 123123123

OAuth2

The Codeless Connector Platform supports OAuth 2.0 authorization code grant and client credentials. The Authorization Code grant type is used by confidential and public clients to exchange an authorization code for an access token. After the user returns to the client via the redirect URL, the application will get the authorization code from the URL and use it to request an access token.

Auth code flow is for fetching data on behalf of a user's permissions and client credentials is for fetching data with application permissions. The data server grants access to the application. Since there is no user in client credentials flow, no authorization endpoint is needed, only a token endpoint.

Example: OAuth2 authorization_code grant type

"auth": {
    "type": "OAuth2",
    "ClientId": "[[parameters('appId')]",
    "ClientSecret": "[[parameters('appSecret')]",
    "tokenEndpoint": "https://login.microsoftonline.com/{{tenantId}}/oauth2/v2.0/token",
    "authorizationEndpoint": "https://login.microsoftonline.com/{{tenantId}}/oauth2/v2.0/authorize",
    "authorizationEndpointHeaders": {},
    "authorizationEndpointQueryParameters": {
        "prompt": "consent"
    },
    "redirectUri": "https://portal.azure.com/TokenAuthorize/ExtensionName/Microsoft_Azure_Security_Insights",
    "tokenEndpointHeaders": {
        "Accept": "application/json",
        "Content-Type": "application/x-www-form-urlencoded"
    },
    "TokenEndpointQueryParameters": {},
    "scope": "openid offline_access some_scope",
    "grantType": "authorization_code"
}

Example: OAuth2 client_credentials grant type

"auth": {
    "type": "OAuth2",
    "ClientId": "[[parameters('appId')]",
    "ClientSecret": "[[parameters('appSecret')]",
    "tokenEndpoint": "https://login.microsoftonline.com/{{tenantId}}/oauth2/v2.0/token",
    "tokenEndpointHeaders": {
        "Accept": "application/json",
        "Content-Type": "application/x-www-form-urlencoded"
    },
    "TokenEndpointQueryParameters": {},
    "scope": "openid offline_access some_scope",
    "grantType": "client_credentials"
}

Jwt

Example: JSON web token (JWT)

"auth": {
    "type": "JwtToken",
    "userName": {
        "key":"username",
        "value":"[[parameters('UserName')]"
    },
    "password": {
        "key":"password",
        "value":"[[parameters('Password')]"
    },
    "TokenEndpoint": {"https://token_endpoint.contoso.com"},
    "IsJsonRequest": true
}

Request configuration

The request section defines how the CCP data connector sends requests to your data source, like the API endpoint and how often to poll that endpoint.

When the API requires complex parameters, use the queryParameters or queryParametersTemplate which include some built-in variables.

StartTimeAttributeName example

Consider this example:

• StartTimeAttributeName = from
• EndTimeAttributeName = until
• ApiEndpoint = https://www.example.com

The query sent to the remote server is: https://www.example.com?from={QueryTimeFormat}&until={QueryTimeFormat + QueryWindowInMin}

QueryTimeIntervalAttributeName example

Consider this example:

• QueryTimeIntervalAttributeName = interval
• QueryTimeIntervalPrepend = time:
• QueryTimeIntervalDelimiter = ..
• ApiEndpoint = https://www.example.com

The query sent to the remote server is: https://www.example.com?interval=time:{QueryTimeFormat}..{QueryTimeFormat + QueryWindowInMin}

Request examples using Microsoft Graph as data source API

This example queries messages with a filter query parameter. For more information, see Microsoft Graph API query parameters.

"request": {
  "apiEndpoint": "https://graph.microsoft.com/v1.0/me/messages",
  "httpMethod": "Get",
  "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ",
  "queryWindowInMin": 10,
  "retryCount": 3,
  "rateLimitQPS": 20,
  "headers": {
    "Accept": "application/json",
    "User-Agent": "Example-app-agent"
  },
  "QueryTimeIntervalAttributeName": "filter",
  "QueryTimeIntervalPrepend": "receivedDateTime gt ",
  "QueryTimeIntervalDelimiter": " and receivedDateTime lt "
}

The previous example sends a GET request to https://graph.microsoft.com/v1.0/me/messages?filter=receivedDateTime gt {time of request} and receivedDateTime lt 2019-09-01T17:00:00.0000000. The timestamp updates for each queryWindowInMin time.

The same results are achieved with this example:

"request": {
  "apiEndpoint": "https://graph.microsoft.com/v1.0/me/messages",
  "httpMethod": "Get",
  "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ",
  "queryWindowInMin": 10,
  "retryCount": 3,
  "rateLimitQPS": 20,
  "headers": {
    "Accept": "application/json",
  },
  "queryParameters": {
    "filter": "receivedDateTime gt {_QueryWindowStartTime} and receivedDateTime lt {_QueryWindowEndTime}"
  }
}

Another option is when the data source expects 2 query parameters, one for start time and one for end time.

Example:

"request": {
  "apiEndpoint": "https://graph.microsoft.com/v1.0/me/calendarView",
  "httpMethod": "Get",
  "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ",
  "queryWindowInMin": 10,
  "retryCount": 3,
  "rateLimitQPS": 20,
  "headers": {
    "Accept": "application/json",
  },
  "StartTimeAttributeName": "startDateTime",
  "EndTimeAttributeName": "endDateTime",
}

This sends a GET request to https://graph.microsoft.com/me/calendarView?startDateTime=2019-09-01T09:00:00.0000000&endDateTime=2019-09-01T17:00:00.0000000

For complex queries, use QueryParametersTemplate. This next example sends a POST request with parameters in the body.

Example:

request: {
  "apiEndpoint": "https://graph.microsoft.com/v1.0/me/messages",
  "httpMethod": "POST",
  "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ",
  "queryWindowInMin": 10,
  "retryCount": 3,
  "rateLimitQPS": 20,
  "headers": {
    "Accept": "application/json",
  },
  "isPostPayloadJson": true,
  "queryParametersTemplate": "{\"query":"TableName | where createdTimestamp between (datetime({_QueryWindowStartTime}) .. datetime({_QueryWindowEndTime}))\"}"
}

Response configuration

Define the response handling of your data connector with the following parameters:

Response configuration examples

A server response with JSON format is expected, with the requested data in the property value. The response property status indicates to ingest the data only if the value is success.

"response": {
  "EventsJsonPaths ": ["$.value"],
  "format": "json",
  "SuccessStatusJsonPath": "$.status",
  "SuccessStatusValue": "success",
  "IsGzipCompressed: true
 }

The expected response in this example prepares for a CSV with no header.

"response": {
  "EventsJsonPaths ": ["$"],
  "format": "csv",
  "HasCsvHeader": false
 }

Paging configuration

When the data source can't send the entire response payload all at once, the CCP data connector needs to know how to receive portions of the data in response pages. The paging types to choose from are:

Configure LinkHeader or PersistentLinkHeader

The most common paging type is when a server data source API provides URLs to the next and previous pages of data. For more information on the Link Header specification, see RFC 5988.

LinkHeader paging means the API response includes either:

• the Link HTTP response header
• or a JSON path to retrieve the link from the response body.

PersistentLinkHeader paging has the same properties as LinkHeader, except the link header persists in backend storage. This option enables paging links across query windows. For example, some APIs don't support query start times or end times. Instead, they support a server side cursor. Persistent page types can be used to remember the server side cursor. For more information, see What is a cursor?.

Here are some examples:

Paging: {
  "pagingType": "LinkHeader",
  "linkHeaderTokenJsonPath" : "$.metadata.links.next"
}

Paging: {
 "pagingType" : "PersistentLinkHeader", 
 "pageSizeParameterName" : "limit", 
 "pageSize" : 500 
}

Configure NextPageUrl

NextPageUrl paging means the API response includes a complex link in the response body similar to LinkHeader, but the URL is included in the response body instead of the header.

Example:

Paging: {
 "pagingType" : "NextPageUrl", 
  "nextPageTokenJsonPath" : "$.data.repository.pageInfo.endCursor", 
  "hasNextFlagJsonPath" : "$.data.repository.pageInfo.hasNextPage", 
  "nextPageUrl" : "https://api.github.com/graphql", 
  "nextPageUrlQueryParametersTemplate" : "{'query':'query{repository(owner:\"xyz\")}" 
}

Configure NextPageToken or PersistentToken

NextPageToken pagination uses a token (a hash or a cursor) that represents the state of the current page. The token is included in the API response, and the client appends it to the next request to fetch the next page. This method is often used when the server needs to maintain the exact state between requests.

PersistentToken pagination uses a token that persists server side. The server remembers the last token retrieved by the client and provides the next token in subsequent requests. The client continues where it left off even if it makes new requests later.

Examples:

Paging: {
 "pagingType" : "NextPageToken", 
 "nextPageRequestHeader" : "ETag", 
 "nextPageTokenResponseHeader" : "ETag" 
}

Paging: {
 "pagingType" : "PersistentToken", 
    "nextPageParaName" : "gta", 
    "nextPageTokenJsonPath" : "$.alerts[-1:]._id" 
}

Configure Offset

Offset pagination specifies the number of pages to skip and a limit on the number of events to retrieve per page in the request. Clients fetch a specific range of items from the data set.

Example:

Paging: {
   
       "pagingType": "Offset", 
        "offsetParaName": "offset" 
 }

DCR configuration

Example CCP data connector

Here's an example of all the components of the CCP data connector JSON together.

{
   "kind": "RestApiPoller",
   "properties": {
      "connectorDefinitionName": "ConnectorDefinitionExample",
      "dcrConfig": {
           "streamName": "Custom-ExampleConnectorInput",
           "dataCollectionEndpoint": "https://example-dce-sbsr.location.ingest.monitor.azure.com",
           "dataCollectionRuleImmutableId": "dcr-32_character_hexadecimal_id"
            },
      "dataType": "ExampleLogs",
      "auth": {
         "type": "Basic",
         "password": "[[parameters('username')]",
         "userName": "[[parameters('password')]"
      },
      "request": {
         "apiEndpoint": "https://rest.contoso.com/example",
         "rateLimitQPS": 10,
         "queryWindowInMin": 5,
         "httpMethod": "GET",
         "queryTimeFormat": "UnixTimestamp",
         "startTimeAttributeName": "t0",
         "endTimeAttributeName": "t1",
         "retryCount": 3,
         "timeoutInSeconds": 60,
         "headers": {
            "Accept": "application/json",
            "User-Agent": "Example-app-agent"
         } 
      },
      "paging": {
         "pagingType": "LinkHeader"
      },
      "response": {
         "eventsJsonPaths": ["$"]
      }
   }
}