cross tenant alerting

Robert Macdougall 20

Hi,following on from this notification

https://azure.microsoft.com/en-us/updates/sending-a-log-search-alert-with-cross-tenant-target-resource-will-no-longer-be-supported/

the part 'As of March 15, 2024, this behavior will change and sending a log search alert with a cross tenant target resource (except for the lighthouse case) will no longer be supported.'

I use azure lighthouse to manage a number of customer subscriptions, up until the above date I used an Azure monitor log search alert to query all log anayltic workspaces in customer delegated subscriptions.

and as I use lighthouse I (wrongly) assumed that 'except for the lighthouse case' would mean that lighthouse behaviour would not change, until all the alerts stopped working.

the alert query rule is saved in the managing tenant, when it has a condition to alert on , it returns an error 'Alert is blocked due to unauthorized target resource (target resource tenant is different from rule tenant).'

that i thought was sort of the point of lighthouse, create a rule in the managing tenant so that you do not need monitor rules in every customer for the same thing. ie. rather than have x number of low disk space alerts , you just need 1 in the managing tenant that will create alerts when the conditions are met.

the KQL works from log analytics so it doesnt appear to be an issue reading the data from the customer workspace.

this was used a basis for the intial alert setup

https://video2.skills-academy.com/en-us/azure/lighthouse/how-to/monitor-at-scale#query-data-across-customer-workspaces

this document however hasnt been updated after the cross tenant bulletin at the top of this question so i am not sure exactly how a cross tenant log search alert will work now with lighthouse if we cannot have the alert rule saved in the managing tenant.

I havent found any documention that explains how cross tenant alerting will now work after the march 15th changes.

anyone else had a similar situation that can maybe shine a light on this ?

many thanks

AnuragSingh-MSFT 21,236 Reputation points

2024-05-07T08:25:56.8066667+00:00

@Robert Macdougall, thank you for posting this question on Microsoft Q&A.

The Azure update you referred are related to forwarding targets (notification, action group etc.) of alerts after they are created. However, the issue you are facing seems to be related to the alert creation process itself. The error points to the expected behavior that "alert rule should be in the same tenant as the target of alert/resource being monitored". I will have to check if this behavior is true, even for lighthouse scenarios.

I tested a sample scenario where I created and saved the log-based alert rule in "managed by" subscription targeting a LA workspace in "managed" subscription. I don't get any error as you have mentioned - "Alert rule should be in the same tenant as the target of alert/resource being monitored" but I don't see any alert being created as well in "the managed" by subscription.

Can you please let me know where you see the error - "alert rule should be in the same tenant as the target of alert/resource being monitored"?
Robert Macdougall 20 Reputation points

2024-05-07T11:34:59.4666667+00:00

many thanks for the update.

where i see the error is

in the Alert rule, i find that when it should fire it creates an event in 'Diagnose and solve issues' which says 'An alert fired from your alert rule was blocked due to an unauthorized target resource.'

when i then click into viewing resource health it shows the extra detail

'Alert is blocked due to unauthorized target resource (target resource tenant is different from rule tenant).'

i have attached some images of these events

for detail on the alert config

the rule is saved in the managing tenant, it is scoped to a LA workspace in the same tenant.

that workspace has some functions that create the union of workspaces from the managed subscriptions and this function is used in the alert KQL to query the data
AnuragSingh-MSFT 21,236 Reputation points

2024-05-14T08:17:38.52+00:00

@Robert Macdougall, thank you for the update.

I have reached out to the Alerting team and will get back to this thread soon. Based on my observation, the tenant difference is creating an issue here, as the alert target should be in the same tenant where the alert rule is being created.
AnuragSingh-MSFT 21,236 Reputation points

2024-05-21T12:22:35.3566667+00:00

@Robert Macdougall, apologies for the delayed response.

I heard back from the team that it will require additional information from your alerting environment setup (backend logs etc.). Due to the nature of further investigation required, I would request you to please open a Azure support ticket, so that it can be investigated 1:1. I am sorry for the inconvenience.
Robert Macdougall 20 Reputation points

2024-05-21T13:45:20.5766667+00:00

mant thanks for your help so far, I have raised the necessary ticket.

I will update here with the findings so that others may find the info if they need it.
AnuragSingh-MSFT 21,236 Reputation points

2024-05-21T15:14:03.38+00:00

@Robert Macdougall, thank you very much.
CRISPIN AKOMEA-AGYIN 0 Reputation points

2024-05-30T14:37:09.14+00:00

@Robert Macdougall, were you able to find the cause of the error and a solution. We have a similar setup as you have described using a single log search alert running a cross-workspace query on workspaces in 2 tenants.

Thank you
Robert Macdougall 20 Reputation points

2024-05-30T16:14:00.97+00:00

@CRISPIN AKOMEA-AGYIN

Hi,

the case is currently being investigated by the Lighthouse team, we are scheduled to have a call at the start of next week to hopefully progress it further.

I will update when I have more info

Share via

cross tenant alerting