HEAT Discovery - Messages aren’t processing, they either say DispatchError or Timedout

Version 1

    Details

    HEAT Discovery - Messages aren’t processing, they either say DispatchError or Timedout.


    Logs workspace shows errors on the Message Queue Service:
    Service Name: MessageQueueService
    Error Code: AggregateException
    Sub System Id: MessageQueueService


    Exception:
    System.AggregateException: One or more errors occurred. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ServiceModel.ServiceActivationException: The requested service, 'http://{appserver}:8382/AssetProcessor/InventoryService.svc' could not be activated

    Message:
    Error in Method invoke Tenant :{tenantURL} - Handler :DiscoAssetProcessor, error The requested service, 'http://{appserver}:8382/AssetProcessor/InventoryService.svc' could not be activated. See the server's diagnostic trace logs for more information. One or more errors occurred. Exception has been thrown by the target of an invocation. The requested service, 'http://{appserver}:8382/AssetProcessor/InventoryService.svc' could not be activated. See the server's diagnostic trace logs for more information.


    Event viewer shows:
    Event code: 3005 
    Event message: An unhandled exception has occurred. 
    Event time: 3/16/2017 10:57:11 AM 
    Event time (UTC): 3/16/2017 3:57:11 PM 
    Event ID: c5d18b24d15e4708975fc0070d8adfbd 
    Event sequence: 10 
    Event occurrence: 9 
    Event detail code: 0 
     
    Application information: 
        Application domain: /LM/W3SVC/7/ROOT/AssetProcessor-1-131341482838717049 
        Trust level: Full 
        Application Virtual Path: /AssetProcessor 
        Application Path: E:\Program Files\HEAT Software\HEAT\IMServer\IMServices\AssetProcessor\ 
        Machine name: {appserver}
     
    Process information: 
        Process ID: 10944 
        Process name: w3wp.exe 
        Account name: {serviceaccout}
     
    Exception information: 
        Exception type: InsufficientMemoryException 
        Exception message: Memory gates checking failed because the free memory (628613120 bytes) is less than 5% of total memory.  As a result, the service will not be available for incoming requests.  To resolve this, either reduce the load on the machine or adjust the value of minFreeMemoryPercentageToActivateService on the serviceHostingEnvironment config element.
       at System.ServiceModel.Activation.ServiceMemoryGates.Check(Int32 minFreeMemoryPercentage, Boolean throwOnLowMemory, UInt64& availableMemoryBytes)
       at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CheckMemoryCloseIdleServices(EventTraceActivity eventTraceActivity)
       at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
     
    Request information: 
        Request URL: http://{appserver}:8382/AssetProcessor/InventoryService.svc?wsdl 
        Request path: /AssetProcessor/InventoryService.svc 
        User host address: ::1 
        User:  
        Is authenticated: False 
        Authentication Type:  
        Thread account name: {serviceaccount} 


     


    Resolution

    The Asset Processor will check for sufficient RAM available before being activated.  If the RAM resources are not sufficient it will not be able to run.


    A reboot may temporarily solve this issue but a long-term fix is to add more RAM or stand up another server and move components to it.  In some cases we find this prevalent when the SQL server is on the same host as the application server.  Best practice is to have SQL on it's own server in all cases except Demo setups.