What components and how would I configure them to handle the following use case?

So my concern is with potential limitations and the need for complicated custom development.

I do business on behalf of many different clients.

It is imperative that they only see the data they are authorized to see/retrieve.

Assume I have 5-10 different event streams that a client could "subscribe" to. I have 1000 clients. I want the clients to maintain their subscriptions and I want them to pull the data for their use.

I assume partitioning is not to be used to manage this. Partitions should be used for load management.

Consumer groups could be used, but if I have a pairing of event Stream/client ID it would be one to one or I use some other implementation of consumer group?

It appears to me:

A custom function to handle the subscription.

A custom solution to manage the offsets.

A custom function to handle the filtering.

Input on how to structure the consumer groups.

I see you mention multiple event hubs.

Can you please send me your ideas on these items?

@McCann, J (John)
I understand your concerns about potential limitations and the need for complicated custom development.

Custom Function to Handle Subscriptions:

• You're correct that a custom function is needed to handle client subscriptions. This function can be implemented as an Azure Function or a Web API that allows clients to subscribe to specific event streams.
• The function can store the subscription information in a database or a cache, such as Azure Cosmos DB or Redis, for efficient lookup and management.

Custom Solution to Manage Offsets:

• You're right again that a custom solution is needed to manage offsets for each client. This can be achieved by storing the offset information in a database or a cache, along with the subscription information.
• When a client pulls data, the custom solution can retrieve the latest offset for that client and event stream, and then fetch the relevant events from the Event Hub.

Custom Function to Handle Filtering:

• A custom function is indeed needed to filter events based on the client ID. This function can be implemented as a part of the event processing pipeline, using Azure Functions or Azure Stream Analytics.
• The function can use the client ID from the auth token to filter events and only return authorized data to the client.

Structuring Consumer Groups:

• You're correct that consumer groups can be used to manage the event stream consumption for each client. However, instead of using a one-to-one pairing of event stream/client ID, you can use a more scalable approach.
• Create a single consumer group for each event stream, and then use the client ID to filter events within that consumer group. This way, you can manage multiple clients subscribing to the same event stream without creating a separate consumer group for each client.

Event Hubs:

• You can create a single Event Hub with multiple partitions, each partition corresponding to a specific event stream (e.g., API Message Request, API Message Response, etc.).
• This approach allows you to manage multiple event streams within a single Event Hub, reducing the complexity of managing multiple Event Hubs.

Hope this helps. Please let us know if you have any further questions.