HEAT Discovery - All Agent Tasks Stuck in Pending status

Version 1

    Details

     

    In some cases when agent tasks get hung up in "Pending" status, there may be a problem with the Discovery system's access to SQL.  This will happen when the SQL access is set to Windows Authentication in the System Configuration Wizard.

    (Note: in 2016.1.1+ fresh installations the SCW should set these correctly but earlier versions or environments upgraded from earlier versions may not handle this as smoothly.)

    This will be apparent with the following log entries:

    On the Discovery Web server in AgentTaskWS.frslog:
    <log4net:event logger="FrontRangeSaaS" timestamp="2016-12-07T14:00:16.3963482-05:00" level="ERROR" thread="37" domain="/LM/W3SVC/1/ROOT/AgentTaskWs-2-131255950277833851" username="NT AUTHORITY\NETWORK SERVICE"><log4net:message><![CDATA[GetClientTasks: FaultException: clientId=[...], clientAuthenticationKey=[fc9cea28edac43a1817afb529f10b0cb], xmlArguments=[<Arguments SchemaVersion="0.2"><InventorySettingsVersion><VersionNumber></VersionNumber><OU></OU><InventorySettings></InventorySettings></InventorySettingsVersion><Task Type="GATEWAYSETTINGS"><Argument Name="Version">1</Argument></Task></Arguments>], reason=Failed to open connection to database. Check credential and privilege details., source=mscorlib, errors=Failed to open connection to database. Check credential and privilege details. Failed to open connection to database. Check credential and privilege details.]]></log4net:message>
    ....

    and in the TaskProcessor and / or AssetProcessor Log file(s):
    <log4net:event logger="FrontRangeSaaS" timestamp="2016-12-07T14:00:16.3873526-05:00" level="FATAL" thread="49" domain="/LM/W3SVC/3/ROOT/TaskProcessor-2-131255947277244034" username="NT AUTHORITY\NETWORK SERVICE"><log4net:message>Login failed for user '{domain}\{machineName}$'.
     Login failed for user '{domain}\{machineName}$'.</log4net:message>


    Meanwhile, the client machine's log files will show the following in the JobQueue_GetClientTasksQueue_xxxx.log" file:
    16:41.42.391  E Warning (Module:mgmtagnt.exe, Severity:0x03): TaskManagementTasks.dll  Error received from Saas webservice: http://{DiscoveryWebServerName}/AgentTaskWs/AgentTaskWS.asmx
    Error: "Server error: Failed to open connection to database. Check credential and privilege details."

     


    Resolution

     

    Please check the Application Pool Identities for all the "HEAT" Application Pools have a service account mapped to their identity that has valid access to SQL: