Public Load Balancer, Network Security Group, filter by source IP does not work
tldr;
How do I restrict access to a public load balancer to specific client ip addresses?
Is it possible to see the source ip of traffic to the load balancer in the event that I see malicious traffic?
I have a public load balancer with a Virtual Machine Scale set behind it handling syslog traffic. I am trying to use the network security group to only allow specific ip's to send traffic to the load balancer. However, the only NSG rules that appear to work have a source of "any" or a source of "internet". Any rules with specific ips or ip subnets do not work. If I turn on flow logs, the source IP is always the load balancer. So I cannot even verify where the traffic is coming from.
I have diagnostics turned on for both the load balancer and the NSG but i cannot find the real source IP in any of the diagnostics or logs I can turn on.
If I give a VM a public IP, the NSG rules work as I would expect them to. Its only through the load balancer that I have an issue.
How do I restrict access to specific servers and how do I see the source ip of traffic to the load balancer?
Thanks in advance for any help or suggestions.