Azure policy "Configure Log Analytics extension on Azure Arc enabled Linux servers" fails to install Log Analytics extensions 50 % of the cases

Edvinas Sulzickis 1

Hello, I am trying to use Azure Arc functionality and update virtual machines on OCI cloud provider. To be able to run updates on non-azure machines there are few prerequisites:

azcmagent installed on non-azure machines
log analytics workspace on Azure cloud
automation account for log analytics workspace
log analytics agent extension on non-azure machines

I chose to use Azure Policy to install log analytics agent extension on non-azure machines, but policy fails to do so 50 % of the time. Either installing on completely new non-azure server or when I try to uninstall extension manually and create remediation task for Azure policy

Is it only me who experience that?

2 answers

AnuragSingh-MSFT 21,356 Reputation points

2022-01-07T12:29:48.023+00:00

Hi @Edvinas Sulzickis

Welcome to Microsoft Q&A! Thanks for posting the question.

The following questions should help narrow down the cause of these failures:

1. Are there specific errors that you observe for these set of servers where the installation fails?

2. Do these servers have something in common (restrictions placed, network connectivity, firewall or from security standpoint).

3. On these servers, does the manual installation fail too? ref: Install Log Analytics agent on Linux computers

You may use the Log Analytics Agent Linux Troubleshooting Tool to help investigate and resolve this issue.
Please let me know if you have any questions.

---
Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.
Please sign in to rate this answer.
Edvinas Sulzickis 1 Reputation point

2022-01-10T07:45:56.483+00:00

Hi, thank you for the response. Will try to answer your questions:

1) Yes
2) I have 2 bastion servers Oracle Autonomous Linux and 4 vm's based Oracle Linux 7. There is no logic about why it fails, because sometimes it fails on provisioning extension on bastion machines, sometime on rest of the machines
3) Tried to do a manual installation. Same behaviour. Sometimes I have to provision extension manually 3 times before it successfully installs

AnuragSingh-MSFT 21,356 Reputation points

2022-01-14T11:34:58.203+00:00

@Edvinas Sulzickis , Thank you for the response. Apologies for the delayed response.
I converted the comment (with attached gc-ext-log.txt) below to private because it contained potentially sensitive information. We strongly recommend not sharing PII and security sensitive data on public forum.

I looked at the error logs and it fails while trying to access the local Azure Instance Metadata Service (IMDS) identity endpoint. The endpoint is a REST endpoint accessible only from within the server using a well-known, non-routable IP address. Please refer to this article for more details: Authenticate against Azure resources with Azure Arc-enabled servers

Since the issue resolves after a couple of retries, it looks to be a (slow) network issue or a transient error, where it took a while for the local server to be configured for Azure Arc. In case you continue to see this issue, I would request opening a support ticket with us so that it can be investigated 1:1. In case you need help opening a support ticket, please let me know.

Edvinas Sulzickis 1 Reputation point

2022-01-14T11:48:01.367+00:00

Sure. I would like to investigate it 1:1. The thing is that even if it shows "successfull" state of installing extension in activity logs, on extensions tab I see "Failed" status. And if it is in "Failed" status I am unable to do any updates with automation account

AnuragSingh-MSFT 21,356 Reputation points

2022-01-27T09:16:45.233+00:00

@Edvinas Sulzickis , thank you for your time working over this issue with our support team. Based on the investigation, it was found that the images used in OCI for the VMs were not supported for LogAnalytics agent.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Andrew Blumhardt 9,856 Reputation points Microsoft Employee

2022-01-07T17:37:23.357+00:00

Here are some logs you can check. Verify that the OS version is supported, verify access to public endpoints (firewall ports), and confirm ARC activation. If you have a support agreement, consider opening a support case.

/var/opt/azcmagent/log/himds.log
/var/opt/azcmagent/log/azcmagent.log
/opt/logs/dsc.log
/opt/logs/dsc.telemetry.txt
/var/lib/GuestConfig/ext_mgr_logs
/var/lib/GuestConfig/extension_logs
Please sign in to rate this answer.
Edvinas Sulzickis 1 Reputation point

2022-01-10T08:12:58.217+00:00

/var/lib/GuestConfig/extension_logs file content

[TELEMETRY Pull Client] [ERROR] [21fa681d-a2cd-4a80-8b9e-8f44d6d17893] Failed to update extensions Error : Failed to get the compute meta_data from the url : http://localhost:40342/metadata/instance?api-version=2019-03-11. Status Code '503'. Error Message '{"error":"service_unavailable","error_description":"Service not available. Check Agent log for details.","error_codes":[503],"timestamp":"2022-01-10 08:39:14.095846377 +0100 CET m=+6.284950553","trace_id":"","correlation_id":"643708bc-6248-4888-987c-90922ba47210"}
[TID 19766] [TELEMETRY DISPATCHER] [ERROR] [21fa681d-a2cd-4a80-8b9e-8f44d6d17893] Failed to finish extension workflow. Error : Failed to get the compute meta_data from the url : http://localhost:40342/metadata/instance?api-version=2019-03-11. Status Code '503'. Error Message '{"error":"service_unavailable","error_description":"Service not available. Check Agent log for details.","error_codes":[503],"timestamp":"2022-01-10 08:39:14.095846377 +0100 CET m=+6.284950553","trace_id":"","correlation_id":"643708bc-6248-4888-987c-90922ba47210"}
Failed to get extension info for extension at path: /var/lib/waagent/Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.14.9.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure policy "Configure Log Analytics extension on Azure Arc enabled Linux servers" fails to install Log Analytics extensions 50 % of the cases

2 answers

Your answer