Usage of Repadmin When Troubleshooting Event ID 1311
Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2, Windows Server 2008, Windows Server 2008 R2
By all means, this topic is not about how to troubleshoot events that have Event ID 1311. In this topic, we are attempting to expose the various usage of repadmin while troubleshooting 1311 in Windows 2000 domains based on Microsoft Knowledge Base (KB) article 307593, How to Troubleshoot Event ID 1311 Messages on a Windows 2000 Professional Domain (https://go.microsoft.com/fwlink/?LinkId=121799). Some or all of the repadmin subcommands used here may be used in Windows Server 2003 environments as well.
The KB article RESOLUTION section has the following action plan. This topic examines how to apply the various repadmin subcommands against each action plan. All of the repadmin subcommands listed in this topic have associated examples either in this section or elsewhere in this document.
Resolution steps from the KB article |
Action plan by using repadmin |
Determine if the event ID 1311 messages are site-specific or forest-wide. |
To determine the scope of event ID 1311 messages:
To determine the ISTG’s, use Repadmin /ISTG. |
Determine if site link bridging is turned on and if the network is fully routed. |
To determine this, use repadmin /showattr (Determine if site link bridging is turned on). |
Verify that all of the sites are defined in site links. |
Every site defined in Active Directory must be hosted or reside in a site link. The repadmin /showism command (Verify inter-site cost matrix and orphaned sites) is useful for locating improperly configured sites. |
Detect and remove preferred bridgeheads. |
To search for preferred bridgehead servers use repadmin /showattr (Determine if site link bridging is turned on). |
Resolve Active Directory replication failures in the forest |
When you want to discover and troubleshoot replication failures, the following repadmin subcommands can be useful:
|
Determine if source servers are overloaded. |
A domain controller that is overloaded with a large number of direct replication partners or a replication schedule that is overly aggressive can create a backlog in which some partners never receive changes from a hub domain controller. The following subcommands can be useful in this situation:
|
Determine if site links are disjointed. |
"Disjoint site links" is an Active Directory configuration in which the topology is broken into two or more parts in which some sites do not replicate because site definitions and site link definitions are incorrect. Disjoint site links are the most difficult improper configuration to troubleshoot. The following subcommands may be useful in this situation:
|
Delete connections if the KCC is in "Keep Connection Mode." |
If the Knowledge Consistency Checker (KCC) builds a different path around a site-to-site connection failure, but it retries the failing connection every 15 minutes because it is in "connection keeping mode," delete all broken connections and let the KCC rebuild them. Wait two times the longest replication schedule in the forest. |
Determine if site link bridging is turned on
Site link bridging is enabled in Active Directory if the following conditions are true:
The Bridge all site links check box is selected for the IP protocol and the SMTP protocol in the Active Directory Sites and Services snap-in.
The Options attribute for the IP protocol and the SMTP protocol is NULL or set to zero (0) for the following distinguished name (DN) paths:
CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=root domain of forest
CN=SMTP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=root domain of forest
There are two values that we could set from the graphical user interface (GUI): Ignore Schedules and Bridge all site links. In our example (figure 3.11.1), the IP transport has Bridge all site links enabled and SMTP transport has both values selected.
The following table lists the various values that the options attribute take.
Option value |
Description |
0X0 |
Only Bridge all site links is selected from the above |
0X1 |
Both the values are selected |
0X2 |
None selected |
0X3 |
Only Ignore schedules is selected |
Detect preferred bridgeheads
Preferred bridgeheads are selected when the following condition is true:
bridgeheadTransportList attribute is set to either one of the following values or both values:
CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=<root domain of forest>
CN=SMTP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=<root domain of forest>
By using repadmin /showattr, we are setting the base at configuration partition and applying a filter for server ObjectClass and looking for all of the domain controllers that have this value set to use either IP or SMTP transports.
If the search returns any results, note the name of server in the distinguished name path in which the bridgeheadTransportList attribute is populated.
In the example in figure 3.11.2, ROOTDC01 is selected as a preferred Bridgehead for IP transport in site HUB.
Verify inter-site cost matrix and orphaned sites
Repadmin /showism displays intersite messaging routes calculated by the Intersite Messaging Service and is very useful for locating improperly configured sites. This operation cannot be executed remotely.
As the KCC runs through the progressions of analyzing intersite site links and connections, it must query the Intersite Messaging Service (ISM) service to retrieve data about the network configuration to make intelligent decisions about routing changes.
To display cost and frequency configurations of replication between sites, use the following command:
Syntax
repadmin /showism [<TransportDN>] [/verbose]
Parameter |
Description |
<TransportDN> |
Specifies whether the mail server is using SMTP or remote procedure call (RPC) to send messages. |
/verbose |
Lists detailed information. |
Note
The repadmin.exe /showism cannot be executed against a remote domain controller.
Example: Display inter-site cost matrix
How to interpret the data
Showism was used against the IP transport and hence the output is specific to IP.
If a specific transport is not specified, the output will contain both IP and SMTP details.
The numbers in an entry appear in the following order:
- Cost: Replication interval: Options
There are four key pieces of information:
Text regarding the status of bridgehead servers.
Total cost between two sites. The cost value indicates the preference for a network link for replicating directory information between sites.
Frequency of replication in minutes between the two sites.
Options for each replication link.
In the example in figure 3.11.3, we have five sites and Bridge all site links is enabled, which means that site link transitivity is enabled. Therefore, if we see any "-1:0:0" entries for one or more covered Active Directory sites, we must ensure that the affected sites are listed in a site link. In this example, site Branch4 is not included in any site links and therefore disconnected from rest of the sites. Event 1311 will certainly occur here due to this configuration problem.
Fields of interest |
Definition |
"0:0:0" |
Each site matrix contains one "0:0:0" entry that refers to itself. |
"200:30:1" |
An entry that contains positive numbers for the cost value and replication interval value (for example, "200:30:1" or "100:15:1") indicates that the site connection is good. Specifically in our example for Site BRANCH1 Site(0) CN=BRANCH1,CN=Sites,CN=Configuration,DC=contoso,DC=com 0:0:0, 200:30:1, 200:30:1, -1:0:0, 100:15:1
And so on the rest of the sites… |
"-1:0:0" |
A "-1:0:0" entry indicates that the site connection is not working. This occurs if one or more of the following conditions are true:
|
Note
Repadmin /failcache
Repadmin /failcache displays a list of replication failures that KCC is aware of. Run this command from the console of each ISTG domain controller in the forest to discover replication failures for bridgeheads in the site for that ISTG.
Syntax
repadmin /failcache <DC_LIST>
Parameter |
Description |
<DC_LIST> |
Specifies the host name of a domain controller, or a list of domain controllers separated by a space. For details about <DC_LIST>, see repadmin /listhelp. |
Example: Display replication failures that KCC is aware of
The example in figure 3.11.4 shows sample output from the repadmin /failcache command.
The output from the repadmin /failcache command is divided into two sections explained in the following table.
KCC Link Failures |
Lists errors for existing connection links. The ISTG domain controller imports showreps ("repsfroms") data for every bridgehead server in its site. However, the ISTG domain controller does not list errors. The link failure cache is emptied at the beginning of every KCC run and refilled during the course of the current run. |
KCC Connection Failures |
Lists unsuccessful attempts to build connection objects between domain controllers ("reps from" or "reps to"). When you run the repadmin /failcache command from the ISTG domain controller, it lists entries that are imported from bridgeheads in the site. At the beginning of each KCC run, the KCC examines each entry in the connection failure cache and tries to DsBind to the failing server. If the bind succeeds, the entry is removed. |
In the example in figure 3.11.4.1, the failures are a result of some topology changes from the past and would continue to exist due to the value of the replTopologyStayOfExecution attribute, which determines how long domain controller metadata is retained in Active Directory after a domain controller has been removed.
Example: Output when there are no failures
When there are no failures, the output should appears as it does in figure 3.11.4.2.
Note
The repadmin /failcache command differs from the repadmin /showrepl command in two ways:
- The repadmin /showrepl command shows the naming context that is failing.
- The repadmin /failcache command does not.
Repadmin /KCC
Repadmin /KCC forces the KCC to recalculate replication topology for a specified domain controller. By default, this recalculation occurs every 15 minutes.
Syntax
repadmin /kcc <DC_LIST> [/async]
Parameter |
Description |
<DC_LIST> |
Specifies the host name of a domain controller, or a list of domain controllers separated by a space. For details about <DC_LIST>, see repadmin /listhelp. |
/async |
Specifies that replication will be asynchronous. This means that repadmin starts the replication event, but it does not expect an immediate response from the destination domain controller. Use this parameter to start the KCC and not wait for it to finish. |
Example 1: Running the KCC on the local domain controller
Example 2: Running the KCC against the ISTG of the HUB site
Example 3: Running the KCC against all the global catalog servers in the forest
Example 4: Running the KCC against all the domain controllers in the BRANCH2 site
Repadmin /ISTG
Repadmin /ISTG returns the server name of the ISTG server for a specified site.
Syntax
repadmin /istg <DC_LIST> [/verbose]
Parameters |
Descriptions |
<DC_LIST> |
Specifies the host name of a domain controller, or a list of domain controllers separated by a space. For details about <DC_LIST>, see repadmin /listhelp. |
/verbose |
Lists detailed information. |
Example: Display ISTGs in my environment
In the example in figure 3.11.6, the ISTGs are listed from the perspective of the local domain controller from which the command was run. It is important to note that this information may be different from the perspective of each domain controller, depending on the forest-wide Active Directory convergence time and replication status.
Repadmin /querysites
Repadmin /querysites use routing information to determine cost of a route from a specified site to another specified site or sites.
Syntax
repadmin /querysites <FromSiteRDN> <ToSite1RDN> <ToSite2RDN>...]
Parameter |
Description |
<FromSiteRDN> |
Specifies the relative distinguished name of the site from which the cost is calculated. |
<ToSite1RDN> |
Specifies the relative distinguished name of the site to which the cost is calculated. |
Example 1: Display cost between BRANCH1 and HUB
Example 2: Display cost between BRANCH1 and BRANCH2
Due to site link transitivity, the cost from BRANCH1 to BRANCH2 is aggregated by adding the cost from BRANCH1 to HUB (100) with the cost from HUB to BRANCH2 (100).
Example 3: Display cost between BRANCH1 and Branch2
Note that the relative distinguished name of the site is case sensitive and hence the error.
Note
Repadmin /queue
Repadmin /queue displays tasks that are waiting in the replication queue.
Syntax
repadmin /queue <DC_LIST>
Parameter |
Description |
<DC_LIST> |
Specifies the host name of a domain controller, or a list of domain controllers separated by a space. For details about <DC_LIST>, see repadmin /listhelp. |
Example: Display the queue length against the local domain controller
Under normal circumstances this list should always be empty and the command should be run outside of the replication window when troubleshooting domain controller overload was caused due to replication requests.
Example: Queue contains one item
Repadmin /bridgeheads
Repadmin /bridgeheads lists the bridgehead servers for a specified site.
Syntax
repadmin /bridgeheads [<DC_LIST>] [/verbose]
Parameter |
Description |
<DC_LIST> |
Specifies the host name of a domain controller, or a list of domain controllers separated by a space. For details about <DC_LIST>, see repadmin /listhelp. |
/verbose |
Lists detailed information. |
For clarity:
The following example shows only bridgeheads only for the HUB site.
The following example shows the normal and verbose modes to help compare them.
“The RPC service is unavailable” status is abbreviated as RPC.
“The operation completed successfully” status is abbreviated as status.
Example 1: Repadmin /bridgeheads rootdns
Bridgeheads for site HUB (rootdns.contoso.com):
Source Site Local Bridge Trns Fail. Time # Status
=========== ============ ==== ============== === ======
BRANCH2 BRANCH-HUB-BH IP 2005-02-14 14:18:52 3 RPC.
Configuration research
BRANCH1 BRANCH-HUB-BH IP (never) 0 Success.
Configuration ForestDnsZones DomainDnsZones research
BRANCH3 BRANCH-HUB-BH IP (never) 0 Success.
Configuration DomainDnsZones ForestDnsZones research
Example 2: Repadmin /bridgeheads rootdns /verbose
Bridgeheads for site HUB (rootdns.contoso.com):
Source Site Local Bridge Trns Fail. Time # Status
=========== ============ ==== ============== === ======
BRANCH2 BRANCH-HUB-BH IP 2005-02-14 14:18:52 3 RPC.
Naming Context Attempt Time Success Time #Fail Last Result
============== ============ ============ ===== ===========
Configuration 2005-02-14 14:51:41 2005-02-14 14:18:51 3 RPC.
research 2005-02-14 14:53:15 2005-02-14 14:18:52 2 RPC.
Source Site Local Bridge Trns Fail. Time # Status
=========== ============ ==== ============== === ======
BRANCH1 BRANCH-HUB-BH IP (never) 0 Success
Naming Context Attempt Time Success Time #Fail Last Result
============== ============ ============ ===== ===========
Configuration 2005-02-14 14:51:41 2005-02-14 14:51:41 0 Success.
ForestDnsZones 2005-02-14 14:52:37 2005-02-14 14:52:37 0 Success.
DomainDnsZones 2005-02-14 14:53:15 2005-02-14 14:53:15 0 Success.
research 2005-02-14 14:52:37 2005-02-14 14:52:37 0 Success.
Source Site Local Bridge Trns Fail. Time # Status
=========== ============ ==== ============== === ======
BRANCH3 BRANCH-HUB-BH IP (never) 0 Success.
Naming Context Attempt Time Success Time #Fail Last Result
============== ============ ============ ===== ===========
Configuration 2005-02-14 14:51:42 2005-02-14 14:51:42 0 Success.
DomainDnsZones 2005-02-14 14:53:15 2005-02-14 14:53:15 0 Success.
ForestDnsZones 2005-02-14 14:52:37 2005-02-14 14:52:37 0 Success.
research 2005-02-14 14:53:15 2005-02-14 14:53:15 0 Success.
How to interpret the data
Repadmin /bridgeheads is run remotely against a domain controller in the HUB site and the output is the perspective of the topology for ROOTDNS. In these examples, we are seeing local bridgehead server BRANCH-HUB-BH is having replication problems with the remote bridgehead server in the BRANCH2 site.
Fields of interest |
Explanation |
Source Site |
Source site from where the local bridge head (inbound) is pulling data. Remember replication is always inbound. |
Local Bridge |
Local Bridge head server for the site for which the tool is displaying results. In the example in figure 3.11.9.2, BRANCH-HUB-BH is the bridgehead server of the HUB site. |
Trns |
In the example in figure 3.11.9.2, the transport is IP. |
Fail time |
This is the last successful replication time. |
# |
Number of failures since the last successful replication time. |
Status |
Replication status. |
Naming Context |
Directory partition. Remember Bridgeheads are partition specific. |
Attempt time |
Last replication attempt time with the remote bridgehead. |
Success time |
Last successful replication time with the remote bridgehead. |
#Fail |
Number of attempts since the failure per partition. |
Last result |
Latest replication status. |
Note
Replication is performed for each partition. But sometimes we do not see the Schema partition listed in the previous example as a naming context (partition) and hence there are no bridgeheads listed. This is not a limitation of the tool; it has to do with the how information is stored in the connection object that is queried to determine the bridgehead. If you see the configuration partition in the output, it is implied that schema is also included because the KCC calculates the configuration and schema partitions to have the same replication topology.
Repadmin /showmsg
Repadmin /showmsg displays the error message for a given error number.
Syntax
repadmin /showmsg <Win32Error> | <DSEventID> /NTDSMSG}
Parameter |
Description |
<Win32Error> |
Returns a short description of the given Win32 error code. |
<DSEventID>/NTDSMSG |
Returns the actual event log text for the specified event ID. |
Example: Display the error message for the win32error 1722 and DS event ID 1404
Repadmin /viewlist
By default, this subcommand is used to displays a list of domain controllers. It could also be used to form an Lightweight Directory Access Protocol (LDAP) query to list only objects in the directory.
Syntax
repadmin /viewlist <DC_LIST> <OBJ_LIST>
Parameter |
Description |
<DC_LIST> |
Specifies the host name of a domain controller, or a list of domain controllers separated by a space. For details about <DC_LIST>, see repadmin /listhelp. |
<OBJ_LIST> |
This parameter takes a distinguished name (DN) or a special keyword that expands into a DN. The keywords are:
|
Example 1: Display all the DC’s in the forest
Example 2: Display all the Group Policy objects in the domain directory partition for the domain of the domain controller that repadmin is running against
Note the usage of OBJ_LIST and OBJ_LIST OPTIONS. For details please refer to repadmin /listhelp.
Open sessions with the domain controller
The repadmin /showctx command displays a list of computers that have opened sessions with a specified domain controller.
Syntax
repadmin /showctx <DC_LIST> [/nocache]
Parameter |
Description |
<DC_LIST> |
Specifies the host name of a domain controller, or a list of domain controllers separated by a space. For details about <DC_LIST>, see repadmin /listhelp. |
/nocache |
Specifies that globally unique identifiers (GUIDs) are left in hexadecimal form. By default, GUIDs are translated into strings. |