Dear All
We encounter an issue with networking / DNS on our Azure HDInsight Spark cluster. The cluster is joined to our AAD (i.e., it's a cluster with ESP enabled).
The cluster gets automatically created with a PS runbook and ARM template file. This is the last line of the runbook to give you an idea:
New-AzureRmResourceGroupDeployment -Name ${clusterName}${deployTime} -ResourceGroupName $ResourceGroupName -TemplateUri $templateUri -TemplateParameterObject $parameters
This process works fine and provisions the cluster into our Vnet. The Vnet has custom DNS setup.
Description of the issue:
First, perform command hostname -f on the primary headnode of the cluster. This returns something like hn0-prdupc.domainn.onmicrosoft.com
Second, run command cat /etc/hosts on the primary headnode:
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
10.250.0.23 hn0-prdupc.domain.onmicrosoft.com hn0-prdupc.axa2i4dkt35e1ksvyarqcgbjjb.ax.internal.cloudapp.net headnodehost hn0-prdupc.domain.onmicrosoft.com. hn0-prdupc headnodehost. # SlaveNodeManager
10.250.0.17 wn0-prdupc.domain.onmicrosoft.com wn0-prdupc wn0-prdupc.domain.onmicrosoft.com. wn0-prdupc.axa2i4dkt35e1ksvyarqcgbjjb.ax.internal.cloudapp.net
...
Third, perform command nslookup hn0-prdupc.domain.onmicrosoft.com
nslookup hn0-prdupc.domain.onmicrosoft.com
Server: 10.90.80.4
Address: 10.90.80.4#53
Name: hn0-prdupc.domain.onmicrosoft.com
Address: 10.250.0.22
For some reason, whatsoever, the answer from the DNS lookup shows a wrong IP address (maybe one from a previous provision step?)
The IP address obtained from the nslookup is different from the IP address in the /etc/hosts file and different from the output of command ifconfig.
I am not an expert, but how is that supposed to work when clusters are created? How / When are DNS entries supposed to be updated in that whole provisioning process? Where / How shall we start to tackle that issue?
Any help / pointers / references to resolve this issue are highly appreciated.
Christoph