Microsoft Sentinel Threat Indicators API - nextLink returns same page

Question

Hello,

I have an issue where the nextLink is always returning the first page of the Threat Indicators in Sentinel.

I'm using the following API-Uri to retrieve all Threat Indicators in a Sentinel Workspace

https://management.azure.com/subscriptions/$subscriptionId/resourceGroups/$resourceGroup/providers/Microsoft.OperationalInsights/workspaces/$workspace/providers/Microsoft.SecurityInsights/threatIntelligence/main/queryIndicators?api-version=2023-11-01

My request body only sets the pageSize to 100.

When sending another request to the "nextLink", I get the same page back which my initial API request returned. This is how I get the nextLink:

($rawOutput.Content | ConvertFrom-Json).nextLink

The normal do-while loop I usually do for paging then results in an endless loop which fills my indicatorVariable with duplicates of the same first 100 indicators. The nextLink is also the same every time:

https://management.azure.com/subscriptions/$subscriptionId/resourceGroups/$resourceGroup/providers/Microsoft.OperationalInsights/workspaces/$workspace/providers/Microsoft.SecurityInsights/threatIntelligence/main/queryIndicators?api-version=2023-11-01&$skipToken=[{"compositeToken":{"token":"+RID:~wkAwAMso-86EhB4AAAAAAA==#RT:1#TRC:2#RTD:eXTCpa71Yy1sqZIRoXXABTMxMzUuMTQuMzFVMTI7MTc7MjIvNjo5NTI5NFsA#ISV:2#IEO:65567#QCF:8#FPC:AgF6enoEAIEENgA=","range":{"min":"","max":"FF"}},"orderByItems":[{"item":"2024-03-20T01:06:11.5984183Z"}],"rid":"wkAwAMso-86EhB4AAAAAAA==","skipCount":0,"filter":"true"}]

At this point I think theres is something wrong with the API. Can somebody please confirm that?

Kind regards,
Benedict

Answer

Hi @Benedict Schmieder , one thing to check is whether the $skipToken parameter is being properly passed in the nextLink URL. The $skipToken parameter is used to indicate the starting point for the next page of results. If this parameter is not being passed correctly, it could explain why you are getting the same page of results every time.

Another thing to check is whether there are any filters or sorting options being applied to the API request that could be affecting the results. If there are filters or sorting options being applied, it's possible that they are causing the API to return the same results every time.

If you check these and still can't find a solution, let me know and we can look into your environment.

Please let me know if you have any questions and I can help you further.

If this answer helps you please mark "Accept Answer" so other users can reference it.

Thank you,

James

Answer

Hi James,
since you've asked if my question was solved, I'm not sure if you saw my comments on your post. Let me answer here again:

I provided the complete nextLink in my initial post. There is also the skipToken in form of a composite token in there. I'm no expert (hence my post here) but I think the skipCount in this token shouldn't be 0, right?

As written previously, I did not apply sorting of any kind, however, applying sorting makes no difference whatsoever.

Kind regards,
Benedict

Answer

The solution is to extract the skiptoken from the nextlink and put it in the filter of the next request. Not really documented or default behavior

# extract skipToken	from nextlink
$skipToken = $nextLink -replace '.*skipToken=([^&]*).*', '$1'
$Filter = @{
    "skipToken" = $skipToken
    "pageSize"  = 500
} | ConvertTo-Json
$Result = Invoke-AzRestMethod -Uri $QueryUri -Method "POST" -Payload $Filter

Share via

Microsoft Sentinel Threat Indicators API - nextLink returns same page

3 answers

Your answer