RDS Multiple Farms on Azure Gateway Issue

I have an old remote desktop farm with Windows Server 2016 hosted on Azure. All roles are on one server with three hosting servers.

I created a new farm in the same subnet and separated the roles on different 2022 servers. My plan is to migrate to the new farm. The problem is when we run an application using the new farm from the Azure public address, the connections are still handled by the old gateway. However, applications still runs using the new hosting servers. This means if we turn off the old gateway, nothing works anymore. To separate the farms, we used a different port as the Azure application gateway listener that points to the new gateway server, but the connections still go to the old gateway. I could not find the reason. Has anyone faced the same problem before?