Azure Signal R - Not acting as backplane as expected - cannot send message to specific user if not connected to same server

So we implemented Signal R on ASP.NET Core and used single server at first due to limitations with scaling

using default scenario not serverless

We discussed on implementing backplane or using Azure Signal R service - ended up going with Azure

The issue we are trying to solve for is when we need to send a message to a specific user through a controller

• We have a microservice architecture
• App A has Signal R hubs (user connects to) with multiple instances (5)
• App B needs to send message to user connected to a hub on APP A (calls controller on App A with data to send)
• If user is not connected to same server as the server App B connected to through load balancer when calling controller, message does not go through and gives error
• If they connect to same server then it goes through as expected

I was under the impression Azure Signal R doubled as backplane to solve this issue https://video2.skills-academy.com/en-us/aspnet/core/signalr/scale?view=aspnetcore-8.0

We set up service like this

We can successfully send a message to a user in a hub like this when connected to the same server (100% of the time in local)

[Authorize]
public class MyHub: Hub<IRateLockClient> {
    public async Task SendUpdateToUser(string username, List<RateLockUpdateModel> messages)
    {
        await Clients.User(username).ReceiveMessage(messages);
    }
}

It was my expectation that regardless of which app server the user is connected to (server 1) any server (ex server 2) could send a message to this user via Clients.User(username)

In current setup it seems this isn't the case - and connected users are still managed in app server instead of Azure signal R service

Is there any way around this? So that any app server can send to any specific user?

I've also tried using groups, but works the same - groups in my setup are app server dependent - so even if users are in same group - they cannot connect if on different servers

I'm hoping I am overlooking something here and can in fact target specific users or groups from any app server, otherwise I would just manage client connection in DB or backplane manually and not use Azure Signal R which I thought was supposed to help manage this scenario