SCOM 2019 - Management Server Not monitored state (Newly build 3 servers but 1 is having issues.)

AdamMohamed-3032 56 Reputation points
2024-02-13T17:12:28.8866667+00:00

Hello All,

Our current SCOM 2019 environment in Azure (3 MS) and as per our management request planning to move the SCOM environment from Azure On-Prem. SO I have build 3 New MS servers and 2 MS servers are healthy and the other one is shows Not monitored state. I have tried re-installed SCOM 2019 and updated the DB server name in registry and Config file but still same issue.

In events in seeing the below error and its not receiving the MP's.

Please advise how to fix the issue. Event Error : Log Name:      Operations Manager
Source:        OpsMgr Management Configuration
Date:          2/13/2024 6:52:48 AM
Event ID:      29120
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      MS.tm.domain.com
Description:

OpsMgr Management Configuration Service failed to process configuration request (Xml configuration file or management pack request) due to the following exception   Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessException: Data access operation failed
   at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessOperation.ExecuteSynchronously(Int32 timeoutSeconds, WaitHandle stopWaitHandle)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.CmdbDataProvider.GetManagementPack(Guid managementPackVersionDependentId)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.TracingConfigurationDataProvider.GetManagementPack(Guid managementPackVersionDependentId)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.AgentRequestProcessor.ProcessManagementPackRequest(ManagementPackRequestDescriptor requestDescriptor)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.AgentRequestProcessor.Execute()
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.ThreadManager.ResponseThreadStart(Object state)


System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at System.Data.SqlClient.TdsParserStateObject.TryReadUInt32(UInt32& value)
   at System.Data.SqlClient.TdsParserStateObject.TryReadPlpLength(Boolean returnPlpNullIfNull, UInt64& lengthLeft)
   at System.Data.SqlClient.TdsParserStateObject.TryReadPlpBytes(Byte[]& buff, Int32 offst, Int32 len, Int32& totalBytesRead)
   at System.Data.SqlClient.TdsParser.TryReadSqlValue(SqlBuffer value, SqlMetaDataPriv md, Int32 length, TdsParserStateObject stateObj, SqlCommandColumnEncryptionSetting columnEncryptionOverride, String columnName)
   at System.Data.SqlClient.SqlDataReader.TryReadColumnInternal(Int32 i, Boolean readHeaderOnly)
   at System.Data.SqlClient.SqlDataReader.TryReadColumn(Int32 i, Boolean setTimeout, Boolean allowPartiallyReadColumn)
   at System.Data.SqlClient.SqlDataReader.GetSqlBinary(Int32 i)
   at System.Data.SqlClient.SqlDataReader.TryGetBytesInternal(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length, Int64& remaining)
   at System.Data.SqlClient.SqlDataReader.GetBytesInternal(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length)
   at System.Data.SqlClient.SqlDataReader.GetBytes(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.ManagementPackGetOperation.ReadResourceList(SqlDataReader reader)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.ManagementPackGetOperation.ReadData(SqlDataReader reader)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.ReaderSqlCommandOperation.SqlCommandCompleted(IAsyncResult asyncResult)
ClientConnectionId:0180c432-2b37-4041-a4d3-11b2b9638452
Error Number:-2,State:0,Class:11
Event Xml:
<Event xmlns="[http://schemas.microsoft.com/win/2004/08/events/event">]
<System>
<Provider Name="OpsMgr Management Configuration" />
<EventID Qualifiers="32768">29120</EventID>
<Level>3</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2024-02-13T12:52:48.315725900Z" />
<EventRecordID>31477</EventRecordID>
<Channel>Operations Manager</Channel>
<Computer>MS.tm.doamin.com</Computer>
<Security />
</System>
<EventData>
<Data>OpsMgr Management Configuration Service</Data>
<Data>Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessException: Data access operation failed
   at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessOperation.ExecuteSynchronously(Int32 timeoutSeconds, WaitHandle stopWaitHandle)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.CmdbDataProvider.GetManagementPack(Guid managementPackVersionDependentId)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.TracingConfigurationDataProvider.GetManagementPack(Guid managementPackVersionDependentId)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.AgentRequestProcessor.ProcessManagementPackRequest(ManagementPackRequestDescriptor requestDescriptor)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.AgentRequestProcessor.Execute()
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.ThreadManager.ResponseThreadStart(Object state)


System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at System.Data.SqlClient.TdsParserStateObject.TryReadUInt32(UInt32& value)
   at System.Data.SqlClient.TdsParserStateObject.TryReadPlpLength(Boolean returnPlpNullIfNull, UInt64& lengthLeft)
   at System.Data.SqlClient.TdsParserStateObject.TryReadPlpBytes(Byte[]& buff, Int32 offst, Int32 len, Int32& totalBytesRead)
   at System.Data.SqlClient.TdsParser.TryReadSqlValue(SqlBuffer value, SqlMetaDataPriv md, Int32 length, TdsParserStateObject stateObj, SqlCommandColumnEncryptionSetting columnEncryptionOverride, String columnName)
   at System.Data.SqlClient.SqlDataReader.TryReadColumnInternal(Int32 i, Boolean readHeaderOnly)
   at System.Data.SqlClient.SqlDataReader.TryReadColumn(Int32 i, Boolean setTimeout, Boolean allowPartiallyReadColumn)
   at System.Data.SqlClient.SqlDataReader.GetSqlBinary(Int32 i)
   at System.Data.SqlClient.SqlDataReader.TryGetBytesInternal(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length, Int64& remaining)
   at System.Data.SqlClient.SqlDataReader.GetBytesInternal(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length)
   at System.Data.SqlClient.SqlDataReader.GetBytes(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.ManagementPackGetOperation.ReadResourceList(SqlDataReader reader)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.ManagementPackGetOperation.ReadData(SqlDataReader reader)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.ReaderSqlCommandOperation.SqlCommandCompleted(IAsyncResult asyncResult)
ClientConnectionId:0180c432-2b37-4041-a4d3-11b2b9638452
Error Number:-2,State:0,Class:11</Data>
</EventData>
</Event>

Operations Manager
Operations Manager
A family of System Center products that provide infrastructure monitoring, help ensure the predictable performance and availability of vital applications, and offer comprehensive monitoring for datacenters and cloud, both private and public.
1,441 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. XinGuo-MSFT 15,781 Reputation points
    2024-02-14T07:51:45.7233333+00:00

    Hi,

    System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  ........System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
    
    

    I would say there are issues with the DB. Can you verify the DB connection works fine.

    Perform a UDL test to check the SQL Server Connectivity.

    • Create a new text file (e.g., sqlconn.txt), and then rename the extension from .txt to .udl.
    • Test the connection using the SQL Server OLE DB provider

    Screenshot of OLE DB Data Link Pages - Connection Tab


  2. XinGuo-MSFT 15,781 Reputation points
    2024-02-16T09:11:10.67+00:00

    Hi,

    We have six MS (three on Azure and three on-Prem). The Azure MS are working perfectly, but the on-premises seem to have problems with everyone.

    I suspect it's a network communication issue:

    We need to check the SCOM Management Servers have Operations Manager event log.

    Check the port allow communication between the different Operations Manager features on your network.

    https://video2.skills-academy.com/en-us/system-center/scom/plan-security-config-firewall?view=sc-om-2022