Thank you for posting a detailed question here.
I tried a similar set-up at my end and was able to RDP into my VM using my Public Load Balancers IP.
You can try validation steps below to see if everything is set-up correctly.
- Can you please validate in the inbound NSG rule, the source IP is your local machine's public IP address? In Azure Load Balancer a response to an inbound flow is always a response from a virtual machine. When the flow arrives on the virtual machine, the original source IP address is also preserved. More information can be found here.
- Can you please validate if the on the VM's overview page the public IP configuration is showing the correct value (the IP listed should be Load Balancer's public IP)? If this value is not correct then there will issue with the backend pool configuration. You can delete the current backend pool association and recreate it following this documentation.
- Lastly there can be other problems which can cause this issue like local machine's firewall or any other NSGs blocking this connectivity. It will help troubleshoot the issue if you can create a fresh VM (and not assign it to the Load Balancers backend pool) and assign it a Public IP address, perform RDP into that VM. If you face issues you can follow this troubleshooting guide to help resolve this issue.
Hope this helps! Please let me know if the issue persists. Thank you!
--
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.