I double checked everything again and found my error. I saw that I used the edge device certificates instead of the leaf device certificates that are correct in the context for my application. Everything is working fine if I use the leaf device certificates. Sorry.
How to set up X.509 authentication with Azure IoT Device SDK?
I followed these instructions to setup X.509 authentication with Azure IoT C Device SDK:
https://video2.skills-academy.com/de-de/azure/iot-hub/iot-hub-security-x509-get-started
https://github.com/Azure/azure-iot-sdk-c/blob/master/tools/CACertificates/CACertificateOverview.md
and the sample C code from https://github.com/Azure/azure-iot-sdk-c/blob/master/iothub_client/samples/iothub_ll_client_x509_sample/iothub_ll_client_x509_sample.c
I changed the connection string in the sample code and exchanged the device certificate and private key with the ones that were generated during the tutorial.
I created the Root CA according to the tutorial, registered the Root CA in the IoT Hub and proofed the possession. The certificate is marked as valid in the IoT Hub.
Next, I created a new device using X509 CA authentication.
When I run the demo code the following error appears:
Func:mqtt_operation_complete_callback Line:1758 Connection Not Accepted: 0x5: Not Authorized
Did I miss a step during setup?