Azure IoT Edge as a Transparent Gateway - Offline Scenario Yields Hundreds of Duplicate D2C Messages

Amir Zuker (CV) 75 Reputation points
2023-03-07T06:42:45.0066667+00:00

Hi,

We're using Azure IoT Edge as a transparent gateway: Downstream Devices -mqtt-> IoT Edge -mqtt-> IoT Hub.
We have pipelines set up that persist data from d2c messages: IoT Hub -msg-route-> EventHub --> Azure Function --> CosmosDb.

Normally, if we don't have the edge or we do and everything is connected and online - we do see some duplicates from time to time, but it's few and sporadic, and that's acceptable to us.
However, when we use the Edge and have an offline scenario, meaning the Edge can't connect to the Hub for some time and then reconnects - we see hundreds of duplicates in CosmosDb.

We understand we use at-least-once delivery guarantee (QoS1) and that is desired, and few duplicates are acceptable to us, but hundreds for every offline occurrence is something we want to avoid.
We know we can implement different kinds of de-duplications post Hub flows but that too is sub-optimal, and we would like to make it so it doesn't happen from the device end in the first place (not at this rate anyway).

We debugged the communication at the device and may have found the problem. It seems that when offline, Azure IoT Edge sends a PUBACK back after one or two minutes!
This causes the device to re-send messages, and we suspect that is why we end up with that many duplicates in the other end.

We could ask developers at the device-end to modify timeouts to account for the Azure IoT Edge behavior but we wish to avoid that, taking several minutes to PUBACK a message isn't considered a typical MQTT broker behavior.

That brings me to the actual questions:

  • Can we configure or control the PUBACK-related behavior in Azure IoT Edge and reduce the time it takes it to send the PUBACK back?
  • Are there any other suggestions that we should consider in terms of dealing with duplicate messages?

Thanks,,
Amir Zuker

Azure IoT
Azure IoT
A category of Azure services for internet of things devices.
399 questions
Azure IoT Edge
Azure IoT Edge
An Azure service that is used to deploy cloud workloads to run on internet of things (IoT) edge devices via standard containers.
573 questions
0 comments
Accepted answer
  1. AshokPeddakotla-MSFT 33,756 Reputation points
    2023-03-08T08:34:51.3633333+00:00

    Amir Zuker (CV) Thanks for reaching out! It's a great question.

    Can we configure or control the PUBACK-related behavior in Azure IoT Edge and reduce the time it takes it to send the PUBACK back?

    According to the information you provided, it seems that when the Edge is offline and then reconnects, it takes a long time to send the PUBACK back, which causes the device to resend messages and results in duplicates in the other end.

    You can try to reduce the time it takes for IoT Edge to send the PUBACK back by increasing the keep-alive interval or the retry interval in the device configuration.

    Are there any other suggestions that we should consider in terms of dealing with duplicate messages? We know we can implement different kinds of de-duplications post Hub flows but that too is sub-optimal, and we would like to make it so it doesn't happen from the device end in the first place (not at this rate anyway).

    For dealing with duplicate messages, you can implement different kinds of de-duplication post-Hub flows, such as using a unique identifier for each message and checking for duplicates before storing the message in CosmosDB.

    You may want to consider develop a mechanism on the device side to detect when the Azure IoT Edge is offline and buffer the messages until the connection is re-established. This can help prevent duplicates from being generated in the first place. For example, you could have the device buffer messages locally in a queue until it receives a PUBACK from the Azure IoT Edge, confirming that the message was successfully delivered. This can help ensure that messages are not sent multiple times when the Azure IoT Edge is offline

    Also, see Enable duplicate message detection for an Azure Service Bus queue or a topic

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.