Hello @DEEPAK KUMPALA ,
Your planned architecture is possible and while a few things need more clarity the keyword seems to be "without data loss".
To build your IoT Service in .Net you should consider the Azure IoT Device SDK what provides built-in functionality for resiliency like re-connection and retry, explained in the Manage connectivity and reliable messaging by using Azure IoT Hub device SDKs article. Although these features are useful, if your IoT Service was shut down while offline you would lose all non-processed messages. Therefore, you would need some way to persist the messages and mark them as processed when your device client receives the acknowledge from IoT Hub. As you identified already, one option would be Mosquitto MQTT broker what provides a "persistence" option to write all connection, subscription and message data to disk, see the mosquitto man page and consider settings for persistent_client_expiration
, max_queued_bytes
and max_queued_messages
options.
When designing your solution, you should have clarity about few things
- What is "a large number of messages"?
- What is the min/avg/max message frequency and size?
- What is the accepted/expected delivery time from source to destination when talking about near real-time (NRT)?
- Do you need all messages, or do they get useless after a certain time?
- Is it suitable to upload buffered messages on a different than the hot path, e.g., directly writing it to storage?
- What are the overall uptime goals and how to ensure resiliency for each solution component, great starting point.
If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.