Runbooks remain queued when using custom hybrid worker credentials, work fine as LocalSystem

@Andrew VO , @Christian Hiebler , thank you for bringing this issue to our attention. After a bit of test and research, here are my findings about this issue. I will summarize it below, so that it can help others also looking for answers to similar questions.

Issue:
Based on the behavior mentioned above, when you use a different user (apart from the default Local System) as Run as credential for Hybrid worker, the runbook jobs targeted to run on this hybrid worker remains in Queued state for approximately 15 minutes and then goes in Suspended state. This happens despite having this user added to Administrators group on the Worker Machine and explicit permissions have to be added to some directories to get this to work.

Cause
This happens because of Filtered Token feature of User Access Control (UAC) on Windows Server - When a user with administrative or other powerful privileges or group memberships logs on, Windows creates two access tokens to represent the user account. The unfiltered token has all the user's group memberships and privileges. The filtered token represents the user with the equivalent of standard user rights. By default, this filtered token is used to run the user's programs. The unfiltered token is associated only with elevated programs. To better protect those users who are members of the local Administrators group, we implement UAC restrictions on the network. This mechanism helps prevent loopback attacks. This mechanism also helps prevent local malicious software from running remotely with administrative rights.
For more details, please refer to the following article:
How to disable User Account Control (UAC) on Windows Server

Resolution and workaround
The resolution of this issue would be - to add the following permissions to the user used in run as credential for Hybrid Runbook Worker. These permissions will have to be added explicitly instead of adding the user to Administrators because by default, the Filtered Token would disallow the user from accessing protected resources/resources requiring elevated permissions.

Registry path

HKLM\SYSTEM\CurrentControlSet\Services\EventLog (read)
HKLM\SYSTEM\CurrentControlSet\Services\WinSock2\Parameters (full access)
HKLM\SOFTWARE\Microsoft\Wbem\CIMOM (full access)
HKLM\Software\Policies\Microsoft\SystemCertificates\Root (full access)
HKLM\Software\Microsoft\SystemCertificates (full access)
HKLM\Software\Microsoft\EnterpriseCertificates (full access)
HKLM\software\Microsoft\HybridRunbookWorker (full access)
HKLM\software\Microsoft\HybridRunbookWorkerV2 (full access)
HKEY_CURRENT_USER\SOFTWARE\Policies\Microsoft\SystemCertificates\Disallowed (full access)
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Setup\PnpLockdownFiles (full access)

Folders

C:\ProgramData\AzureConnectedMachineAgent\Tokens (read)
C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\**<version>**\HybridWorkerPackage\HybridWorkerAgent (full access)

Please refer to this link for details - Use a credential asset for a Hybrid Runbook Worker group

The workaround to adding explicit permission as mentioned above to the user used in "run as credential" for Hybrid Runbook Worker will involve the following steps. Please note this would be a not recommended workaround, as it would disable the default UAC security check in place.

1. Add the user to local "Administrators" group
2. Disable the local group policy - User Account Control: Run all administrators in Admin Approval Mode, available in Group Policy editor here: Local Security Policy --> Security Settings --> Local Policies --> Security Options.

3. Reboot the machine

Please let me know if you have any questions.

Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.