Different ways to narrow down timeout error while importing users from AD via inSync Connector
Problem description
There could be scenarios when you get a timeout error while you are importing users from Active Directory (AD) via the AD Connector.
The message This seems to be taking a longer time than expected. Please try again after some time or click "Ok" to reload the page is a general message shown on the GUI when any API takes more than five minutes and is expected.
The instructions provided in this article shall help you to identify where the delay is occurring and resulting in the timeout message.
Procedure
Do the steps as follows:
- Check the size of the AD user group from which the users are supposed to get imported.
Note: Druva recommends optimized AD mappings limited to group/sub groups, so that the search returns less number of users.
- Check if the issue persists by creating AD mapping pointing to a Global Catalog (GC) (in case of parent-child domain scenario).
- To find the GC inside a domain, run the following command on any Domain Controller (DC):
C:\>dsquery server -isgc -domain "contoso.local"
Note: Domain name in the above command is contoso.local.
- Alternatively, do a nslookup on any domain-joined machine pointing to the internal DNS server by running the following command:
C:\>nslookup gc._msdcs.contoso.local
- To find the GC inside a domain, run the following command on any Domain Controller (DC):
- Collect network trace on AD Connector while reproducing the issue to check if there is any network latency or packet drop scenario between the inSync Cloud and AD connector or between AD connector and Domain Controller.
- Check if the delay is from the AD itself. To check this, run dsquery on the pointed domain controller for the AD group mapped via AD mapping. Run below PowerShell command:
dsquery group -samid "inSync_users" | dsget group -members -expand | dsget user -samid -upn -email -sid
Note: The AD group name is inSync_users.
- Also, you can check if the AD connector server is able to fetch the details of the users from AD without making a query via AD connector application. For this, you can install OpenLDAP ClientTools on AD connector server and emulate the exact same query as done by the inSync Cloud using OpenLDAP ClientTools. Do the following steps to check this:
-
Enable Field Engineering logging on DC to see what query AD connector makes while importing users from AD. To enable Field Engineering logging Set the value for "15 Field Engineering" to 5.
Note: The default value is Zero. This value can be found in HKLM\System\CurrentControlSet\Services\NTDS\Diagnostics
-
After changing the value, you will start seeing the entries for expensive and inefficient LDAP searches. You get the following event LOG on DC, when the AD connector makes a query when you click on the Import new users button on AD mapping.
Log Name Directory Service Source Microsoft-Windows-ActiveDirectory_DomainService Date 3/6/2020 3:33:20 PM Event ID 1644 Task Category Field Engineering Level Information Keywords Classic User CONTOSO\minzi Computer ADCONN-DC2.contoso.local Description Internal event: A client issued a search operation with the following options. Client 123.45.67.890:12345 Starting node: DC=contoso,DC=local Filter: (objectCategory=CN=Person,CN=Schema,CN=Configuration,DC=contoso,DC=local) Search scope: subtree Attribute selection: cn,mail,userPrincipalName,cn,department,countryCode,userAccountControl,objectGUID Server controls: -
With the help of this event log, you can make the following LDAP search query which needs to be run on an AD connector using Openldap client tool.
C:\OpenLDAP\ClientTools>ldapsearch.exe -h contoso.local -D
CN=minzi,CN=Users,DC=contoso,DC=local" -w "PassWord" -b "OU=inSync Users,DC=contoso,DC=local" -s sub (objectCategory=CN=Person,CN=Schema,CN=Configuration,DC=contoso,DC=local)" -f cn mail userPrincipalName department countryCode userAccountControl objectGUIDParameters used in the commend are as follows:
-h <domain name or IP address/FQDN of the DC you want the answer from> -D <base DN of service account entered in AD connector application -w <password of the service account> -b <base DN for search; base DN of the OU where AD group resides> -s <search scope> Note: The following portion of the command remains the same every time
-s sub
"(objectCategory=CN=Person,CN=Schema,CN=Configuration,DC=contoso,DC=local)" -f cn mail userPrincipalName department countryCode userAccountControl objectGUI -
Run the above command on AD connector. You can expect the following output:
C:\OpenLDAP\ClientTools>ldapsearch.exe -h contoso.local -D "CN=minzi,CN=Users,DC=contoso,DC=local" -w "PassWord" -b "OU=inSync Users,DC=contoso,DC=local" -s sub "(objectCategory=CN=Person,CN=Schema,CN=Configuration,DC=contoso,DC=local)" -f cn mail userPrincipalName department countryCode userAccountControl objectGUID
ldap_connect_to_host: TCP contoso.local:389
ldap_new_socket: 732
ldap_prepare_socket: 732
ldap_connect_to_host: Trying 172.16.53.114:389
ldap_pvt_connect: fd: 732 tm: -1 async: 0
attempting to connect:
connect success
# extended LDIF
#
# LDAPv3
# base <OU=inSync Users,DC=contoso,DC=local> with scope subtree
# filter: (objectCategory=CN=Person,CN=Schema,CN=Configuration,DC=contoso,DC=local)
# requesting: -f cn mail userPrincipalName department countryCode userAccountControl objectGUID
#
# ironman, inSync Users, contoso.local
dn: CN=ironman,OU=inSync Users,DC=contoso,DC=local
cn: ironman
objectGUID:: fHdF82N9eEOxCB1GR/d5bw==
userAccountControl: 66048
countryCode: 0
userPrincipalName: ironman@contoso.local
mail: ironman@gmail.com
# Thor, inSync Users, contoso.local
dn: CN=Thor,OU=inSync Users,DC=contoso,DC=local
cn: Thor
objectGUID:: v6O48fo+mEyEjS+IYMnVoA==
userAccountControl: 66048
countryCode: 0
userPrincipalName: Thor@contoso.local
mail: Thor@gmail.com
# William Doe, inSync Users, contoso.local
dn: CN=William Doe,OU=inSync Users,DC=contoso,DC=local
cn: William Doe
objectGUID:: teM4fZ7yhEypXjnVfzhwlA==
userAccountControl: 66048
countryCode: 0
userPrincipalName: William@contoso.local
mail: William@gmail.com
# Thanos, inSync Users, contoso.local
dn: CN=Thanos,OU=inSync Users,DC=contoso,DC=local
cn: Thanos
objectGUID:: zYlCHMPw6kK3qJkc4x5q9A==
userAccountControl: 66048
countryCode: 0
userPrincipalName: Thanos@contoso.local
mail: thanos@gmail.com
# search result
search: 2
result: 0 Success
# numResponses: 5
# numEntries: 4
Note: For Active AD user the value of userAccountControl: 66048
For Disabled AD users the value of userAccountControl: 66050
-
-
You can also get the exact time it took for the AD connector to fetch the user details from the DC. Run the following PowerShell command using the same AD credentials which are used for AD auth by inSync.
Import-Module ActiveDirectory
Measure-Command -Expression { Get-ADUser -filter * -searchbase "OU=inSyncUsers,DC=contoso,DC=local" }
You get the following output:
PS C:\Users\minzi> Measure-Command -Expression { Get-ADUser -filter * -searchbase "OU=inSyncUsers,DC=contoso,DC=local" }
Days 0 Hours 0 Minutes 0 Seconds 0 Milliseconds 843 Ticks 8431356 TotalDays 9.75851388888889E-06 TotalHours 0.000234204333333333 TotalMinutes 0.01405226 TotalSeconds 0.8431356 TotalMilliseconds 843.1356 By performing these steps, you can narrow down and find where the delay is occurring.