Azure Premium Frontdoor linked to internal Load Balancer in front of Azure VM in private subnets not working

Alvin 25

Hello,

Following this link https://video2.skills-academy.com/en-us/azure/frontdoor/standard-premium/how-to-enable-private-link-internal-load-balancer I implemented Azure front door premium with private link services linked to an internal load balancer in front of Azure Virtual machines in a private subnet, this appears not to be working. Any ideas on what I could be doing wrong?

Kindly note that this architecture works fine with a public load balancer and without the private link service.

KapilAnanth-MSFT 39,446 Reputation points Microsoft Employee

2023-09-06T05:13:15.5766667+00:00
@Alvin

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you would like to connect your application running behind ILB to AFD via Private Link Service.

I see you have mentioned this works with Public Load Balancer.

However,

can you confirm if the same set up works with directly accessing the ILB (without AFD)?

You can test this by deploying a dummy VM in the same VNET/Subnet as ILB and try accessing it from this VM using it's Private IP?

Cheers,

Kapil
Alvin 25 Reputation points

2023-09-06T08:15:42.2066667+00:00

@KapilAnanth-MSFT

Thank you for your response.

Pinging the Internal Load Balancer private IP from a Dummy VM inside the ILB subnet returned no packets.
But nslookup for the DNS of the internal load balancer returned the private IP of the internal load balancer
KapilAnanth-MSFT 39,446 Reputation points Microsoft Employee

2023-09-06T08:19:13.7+00:00

@Alvin

That means there is an issue with your application itself.

Since you informed your backend is a web server, can you try accessing the ILB over HTTP and see if the server responds?

Cheers,

Kapil
Alvin 25 Reputation points

2023-09-06T08:57:03.6066667+00:00
@KapilAnanth-MSFT

The backend runs a container and has its exposed port mapped to port 443 on the VM.
Curl command from the web server VM to its internal IP:443 address returned content.

<html> <head><title>400 The plain HTTP request was sent to HTTPS port</title></head> <body> <center><h1>400 Bad Request</h1></center> <center>The plain HTTP request was sent to HTTPS port</center> <hr><center>nginx/1.18.0 (Ubuntu)</center> </body> </html>

</html>

Curl command from the web server VM to the IP of the ILB:443 returned nothing.
Curl command from the Dummy VM to the IP of the ILB:443 returned

<html> <head><title>400 The plain HTTP request was sent to HTTPS port</title></head> <body> <center><h1>400 Bad Request</h1></center> <center>The plain HTTP request was sent to HTTPS port</center> <hr><center>nginx/1.18.0 (Ubuntu)</center> </body> </html>

Curl command from the Dummy VM to the IP of the ILB:443 returned nothing.

Please note that the web server VM is accessed via a Bastion Host.

Apparently HTTP access to the ILB returns no response.
KapilAnanth-MSFT 39,446 Reputation points Microsoft Employee

2023-09-06T09:39:01.9466667+00:00
@Alvin

This indicates there could be a misconfiguration on the ILB.

I see your service is returning HTTP Response 400

Is this expected?

Shouldn't you be getting a 200 Response code?

What Port and Protocol are you using for Health Probes?

If you are using HTTP/HTTPS - and response code other than 200 (for example, 403, 404, or 500) are considered failure

Also, you can change the Probe to use a TCP probe.

Please check the ILB Metrics for "Health probe status"

See if the health probes are actually succeeding.

https://video2.skills-academy.com/en-us/azure/load-balancer/load-balancer-standard-diagnostics#are-the-backend-instances-for-my-load-balancer-responding-to-probes

What port are you using for your Load Balancer Rule.

Port 80 or Port 443?

Cheers,

Kapil
Alvin 25 Reputation points

2023-09-06T09:50:44.6133333+00:00

The ILB Probe

@KapilAnanth-MSFT

The ILB health probes are fine. Backend port and health probe is on TCP port 443.
The 400 is a configuration issue from Nginx, I am happy getting this for now.
KapilAnanth-MSFT 39,446 Reputation points Microsoft Employee

2023-09-06T10:04:50.7166667+00:00
@Alvin

In that case, we should troubleshoot why ILB is not responding.

I see you have enabled "Floating IP"

May I ask if this is expected?

If not, can you please disable this and perform curl to the ILB IP from the dummy VM and let me know the results?

Also, please make sure there are no NSGs blocking in the ILB FrontEnd subnet.

Cheers,

Kapil
Alvin 25 Reputation points

2023-09-06T10:22:38.92+00:00

@KapilAnanth-MSFT

Curl from Dummy VM to ILB returned a response after disabling floating IP.
Curl from web server VM to ILB returns nothing.

Here are the VM and ILB subnets NSGs respectively
KapilAnanth-MSFT 39,446 Reputation points Microsoft Employee

2023-09-06T10:38:13.46+00:00
@Alvin

"Curl from Dummy VM to ILB returned a response after disabling floating IP."

This is good news

Now, ILB issue is resolved then.

Now, let's configure the AFD.

AFAIK, accessing the AFD should work now.

If it does not, please make sure you "Enable health probes" is unchecked in the Origin Group configuration.

And the origin is configured according to the docs : https://video2.skills-academy.com/en-us/azure/frontdoor/standard-premium/how-to-enable-private-link-internal-load-balancer#enable-private-connectivity-to-an-internal-load-balancer

Make sure the Host Name and Origin Host Header are the private IP of the internal load balancer

Cheers,

Kapil
Alvin 25 Reputation points

2023-09-06T11:31:31.8966667+00:00

@KapilAnanth-MSFT

ILB or VM Origin Group

ILB or VM Origin Private Link Service

VM or ILB Origin

I have Azure Front door premium set up with multiple frontends all the way to Origin Groups. For the Origin group of concern, I have Azure Container Apps and the web server VM set up as different origins, using different Private Link Services. I reduced the Weight of the ACA (Azure Container App) Origin to 50, and increased priority to 5, for the VM it is weight = 1000 and priority = 1.

The ACA Origins works fine.
The VM Origin does not, I have done exactly all that is described here
https://video2.skills-academy.com/en-us/azure/frontdoor/standard-premium/how-to-enable-private-link-internal-load-balancer#enable-private-connectivity-to-an-internal-load-balancer
I have additionally disabled health probes, it still doesn't work.

Can you please confirm that there is nothing wrong with the NSGs?
KapilAnanth-MSFT 39,446 Reputation points Microsoft Employee

2023-09-07T13:11:20.3633333+00:00
@Alvin

Wrt, "Can you please confirm that there is nothing wrong with the NSGs?"

Yes

As long as you are locally able to access the ILB from DummyVM, this should work with AFD Private EndPoint as well from NSG perspective.

The error you have shared usually happens whenever there is a change in the AFD.

Can you please confirm if you are still facing it?

Cheers,

Kapil
Alvin 25 Reputation points

2023-09-07T18:27:07.93+00:00

@KapilAnanth-MSFT

Thanks for your response.

Yes I am still facing it.
I believe the problem here is accessing a webserver running in a VM with no public IP address behind a Bastion Host from an internal Load Balancer. I have tried this out with a completely different setup and each time Curl command to the private IP address of the Internal Load Balancer returned nothing.
I believe this is the problem we need to resolve.

Cheers.
Alvin 25 Reputation points

2023-09-11T08:02:58.1533333+00:00

Hello @KapilAnanth-MSFT

Could you please implement the set up below

Set up a webserver in the VM and run Curl command to the ILB private IP address and listening port. Please kindly let me know if this works, the problems am having with boils down to this set up.
KapilAnanth-MSFT 39,446 Reputation points Microsoft Employee

2023-09-11T10:05:28.2+00:00
@Alvin

Wrt,

"I believe the problem here is accessing a webserver running in a VM with no public IP address behind a Bastion Host from an internal Load Balancer"

Bastion Host is only used to provide access to VMs via RDP or SSH

Bastion Host has no significance here, at all.

Also, if the site is behind an ILB

You must access it from a private network only - not from Internet

Also, you did confirm this issue was resolved when you disabled "Floating IP" - isn't that not correct?

Looking at your setup,

Setting up a webserver in the VM and run Curl command to the ILB private IP address and listening port from a VM in the same VNET works perfectly for me as long as Floating IP is disabled in the ILB

May I ask, in your setup, where exactly the source is?

Cheers,

Kapil
Alvin 25 Reputation points

2023-09-11T11:35:34.0733333+00:00

@KapilAnanth-MSFT

Thanks for your response.

If you recall, based on your suggestions, what I tested earlier was to set up a Dummy-VM with public IP within the same Subnet as the ILB, and with floating IP disabled. Curl to the ILB from the Dummy-VM worked perfectly, However curl to the same ILB from the Webserver private VM (The VM in subnet 3 in the diagram above) behind a Bastion Host did not work.

To answer your question about source, if I get correctly what you mean by "source", I would say the source here is the Private VM (No public IP) in subnet 3 behind the Bastion Host. Kindly note that the Dummy-VM is not included in the set-up described in the diagram above.

Kind regards.
KapilAnanth-MSFT 39,446 Reputation points Microsoft Employee

2023-09-11T11:45:36.66+00:00
@Alvin

Please note :

"Bastion Host" or VM behind "Bastion Host" does not add any value.

Bastion Host is only used to provide access to VMs via RDP or SSH, and that is it's only purpose

There is no significance whether you access the VM via Bastion or not, it is all the same.

"However curl to the same ILB from the Webserver private VM (The VM in subnet 3 in the diagram above)"

If this is the backend VM of the ILB, then this is an expected behavior.

Why would you want to access the local server using it's ILB when you can directly access it as "local host"

Outbound flow from a backend VM to a frontend of an internal Load Balancer will fail

Refer : https://video2.skills-academy.com/en-us/azure/load-balancer/components#limitations

Again, an ILB can only be accessed in a private Network.

To make it accessible via Public Internet, we are leveraging AFD + Private Link

In your case, if you were to confirm that accessing the ILB via DummyVM works fine, then I would say ILB set up is correct and is working as expected.

Can you confirm if access to ILB from DummyVM works fine or not?

If yes, what is your next requirement here?
- To make the site accessible via Internet? or something else?

Cheers,

Kapil
Alvin 25 Reputation points

2023-09-11T11:55:04.3166667+00:00

@KapilAnanth-MSFT

Thank you for the clarification about why curl command to the ILB from the backend VM is failing.

I am looking into the possibility of a setting in my Private Link Service Subnet "Private endpoint network policy" being an issue with my setup, I will confirm this and feedback.

Thanks a lot for the help, learned quite a lot here.
KapilAnanth-MSFT 39,446 Reputation points Microsoft Employee

2023-09-12T13:04:01.7766667+00:00

@Alvin

You are most welcome.

Meanwhile, I shall summarize my comments and post it as a new answer.

I would highly appreciate if you could Accept the answer and close this thread

Original posters help the community find answers faster by identifying the correct answer.

Thanks,

Kapil

Accepted answer

KapilAnanth-MSFT 39,446 Reputation points Microsoft Employee

2023-09-12T13:10:56.5533333+00:00
@Alvin

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you would like to connect your application running behind ILB to AFD via Private Link Service.

You informed us directly accessing the ILB did not work from a dummy VM in same VNET

Your backend(ILB) was not responding

I checked the configuration and isolated the issue being "Floating IP" enabled

Disabling this, made the ILB accessible from the dummyVM.

Later, you wanted to access the ILB from one of the backend VMs of the ILB

Outbound flow from a backend VM to a frontend of an internal Load Balancer will fail

Refer : https://video2.skills-academy.com/en-us/azure/load-balancer/components#limitations

You can access the local server using directly "local host"

No need to use ILB for this.

Also, an ILB can only be accessed in a private Network.

To make it accessible via Public Internet, we are leveraging AFD + Private Link.

In your case, if you were to confirm that accessing the ILB via DummyVM works fine, then I would say ILB set up is correct and is working as expected.

Now, we should troubleshoot why this would not work as a backend of AFD.

You informed us you are looking into the Private Link Service Subnet "Private endpoint network policy"

Thanks,

Kapil
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

Azure Premium Frontdoor linked to internal Load Balancer in front of Azure VM in private subnets not working

0 additional answers