How To: Start Troubleshooting Anti-Virus Issues

Version 6

    Purpose

     

    Give some information to get troubleshooting started for Anti-Virus issues.

     

    Overview

     

     

    Information

     

    Logging:

    Log Locations and Names:

    ·         Server Side:

    o   Endpoint Distribution

    §  "C:\Program Files (x86)\HEAT Software\EMSS\Endpoint Distribution Services\logs\edsrolling_DownloadManager.BitDefenderContent.log"

    o   Logging volume controlled by LogMinimumSeverity value in the "C:\Program Files (x86)\HEAT Software\EMSS\Endpoint Distribution Services\LM.Logging.config" file.

     

    ·         Agent Side: C:\ProgramData\HEAT Software\EMSSAgent\logs\AV

    o   Real-Time Monitoring_yyyy_mmm_hh:mm:ss.log 

    o   Recurring-Scan_yyyy_mmm_hh:mm:ss.log

    o   ScanSummary.txt (one only)

    o   QuarantingScanRestore.log (one only, used when new defs come down to see if Quarantined files can be cleaned)

    o   Epui-customer-scan_yyyy_mmm_hh:mm:ss.log

    o   Epui-full-scan_yyyy_mmm_hh:mm:ss.log

    o   Lmhost-main.log

    ·         Each scan will create a file in the AV logs folder: C:\ProgramData\HEAT Software\EMSSAgent\logs\AV Recurring scans will log dependent on the logging level set when creating the recurring scan in the EMSS UI. Custom scans will have detailed logging by default.

     

    Enable/Disabling max logging:

    ·         Agent Side: TRACE-level agent logging will capture AV activities in the lmhost logs.

     

    Definition File Locations:

    ·         Server Side: C:\Program Files (x86)\HEAT Software\EMSS\Content\AntiVirus\Definitions

    ·         Agent Side: C:\ProgramData\HEAT Software\EMSSAgent\data\persist\live\AV

     

    Policy Files:

    • EMSS Console AV     Policy changes result in a new AvPolicy and an update to the Wpolicy to  reflect the new AvPolicy being sent to the endpoint.
    • On the endpoint this results in many     xml files located here:

    C:\ProgramData\HEAT Software\EMSSAgent\data\persist\live\AV:

    §  Basepolicy.xml -> This contains the AV specific content taken from the wpolicy

    §  Avgroups.xml -> this is the groups node from the wpolicy.

    §  Avfeature.xml -> is the result of: 1. Filter of groups, smackdown applied, merged with wpolicy.

    §  AvPolicy.xml -> is the avpolicy!

    Event uploads:

    Agent Side:

    ·         Scan events are stored in C:\ProgramData\HEAT Software\EMSSAgent\data\persist\live\AV\Snf (Store n Forward) folder and uploaded when online. If Agent is offline, they are stored here until they can be uploaded EMSS server and acknowledgement from EMSS server is received. 

    Emergency ..:

    Disabling of AV:

    ·         Can be disabled in EMSS Web Console, but not possible via endpoint. See below for removal of AV instead.

     

    Removal of AV:

    ·         The nice way to remove it (aside from the EMSS Console): Hand craft a wpolicy without the AV module and send it to lmhost using lmctl.

    ·         The forceful way to remove it: Delete the gzflt.sys from c:\windows\system32 and reboot

     

    Manual Update of AV for network-disconnected endpoints:

    ·         Process Doc to be done up, possible with Defs file copy from online-agent to offline-agent drives.

     

     

    Troubleshooting

     

    BD Definitions Download problems:

    Server Side:

    ·         Test that you can download the AV definition updates manually in the web console. TOOLS=>Subscription Updates=> Configure (button)=> AntiVirus (tab)=> Download now button.

    ·         If Replication gives a FALSE for AV definitions, check AV content link in TOOLS=>Subscription Updates=> Configure=> AntiVirus=>Test Link button.

    ·         Or copy the link to a Web browser and have "/av32bit/versions.id" on the end. It should list the current ID value in XML.

    ·         Check …\edsrolling_DownloadManager.BitDefenderContent.log for specified errors. "Process_Exited:" is followed by the outcome of the download attempt.

    Agent Side:

    ·         Lmhost-main.log for Defs transfer information.

    ·         C:\ProgramData\HEAT Software\EMSSAgent\data\persist\live\AV\ScanEngine X  (exact name of X is listed in the live.txt file in same folder).

     

    Performance issues:

    Likely causes of sudden endpoint performance drop due to AV:

    • Smart Cache has been flushed (or cache files deleted) so all files need to be scanned again. ("C:\Program Files\HEAT Software\EMSSAgent\live\smartdb-ntfs.db" and smartmd5cache.dat)

     

    Likely causes of sudden network bandwidth drop due to AV:

    o    Definitions being downloaded again.

    Real Time Monitoring

    • Users notice sluggish endpoint performance (Apps slow to open etc)
    • Check Policy Settings
    • Have default settings been changed (e.g. Extended Sandbox)?  Why?
    • Have recommended exclusions been applied?
      • e.g. SQL excludes if SQL DB present
    • Turn on detailed logging for Real Time Monitoring (DB script (Real Time Monitor logging))
      • Use to identify files being scanned
      • Copy log file for analysis
    • Procmon analysis
      • Engage Development to assist

    Recurring Scan

    ·         User notices performance impacts while AV scan is running

    o   Have default settings been changed (e.g. Scan Archives)?  Why?

    o   Adjust CPU throttling settings

    Low:

     

          <rcs_cpubusymaxusage Value="10" />

          <rcs_cpuidlemaxusage Value="25" />

     

    ======

    Medium:

     

      <rcs_cpubusymaxusage value="30" />

      <rcs_cpuidlemaxusage value="70" />

     

    ======

    High:

     

      <rcs_cpubusymaxusage value="100" />

      <rcs_cpuidlemaxusage value="100" />

     

     

    AV content not coming down to the EMSS Server from GSS

    Non-AirGap Environment

    Turn on Verbose logging on the server
    1. On the EMSS server, set the Server logging level to verbose by going to

    \%Install Path%\Endpoint Distribution Services\LM.logging.config

    Update the following from Warning...

    <appSettings>

        <add key="LogMinimumSeverity" value="Verbose" />

        <add key="LogStackFrame" value="False" />

        <add key="LogMaxFiles" value="10" />

        <add key="LogEnabled" value="True" />

      </appSettings>

     

    Examine the MirrorSDK logging file
    1. The MirrorSDK, when verbose logging is turned on logs to:  edsrolling_DownloadManager.BitDefenderContent
    2. The error code is returned in the following line:

    Process_Exited: BitDefender Download Completed with exit code:

    • Success
    • Error
    • NoContent

    Success is when content has been successfully downloaded.

    NoContent is when an attempt has been made to download content, but that it has not changed since the last download. As a result, no content is returned.

    Error is when an error has occurred. If an error has occurred the logging messages above will indicate the error type. From experience these tend to be a HTTP error code. For these please follow the following steps.

    Verify that the server is pointing to the correct GSS url
    1. In the EMSS UI, go to Tools -> Subscription Updates -> Configure... -> AntiVirus tab.
    2. Verify that the AntiVirus content storage location for AntiVirus     Agent version 8.2 + is

    http://cache.lumension.com/avcontent

    1. Verify that the link works by clicking on ‘Test Link…’
    Verify that you can access the content from your EMSS Server
    1. Take the url in the AntiVirus content storage location for AntiVirus Agent version 8.2 + and append /av32bit/versions.id

    http://cache.lumension.com/avcontent/av32bit/versions.id

    1. Verify that you get an xml file similar to:

    <?xml version="1.0"?>

    <info><all><id value="45609"/><md5 value="7ed73b2dc23df8f99446019196635cb4"/></all><sig>wuoGT/ezUbm2JXIl29s/adxyNS7f3u1H8By+H22bnUsh2k+b/Bh2aBUmiubQguI748IOpQr1rOfCm+9JYeoDNfwxPE1INCvwjJRIzH2NTNCuXxkRtzY6l8IkeC26TYX8rJ1985n5oJrtmQ1DQ9090wlthxurs2EVcHnK/+2+n+9+RCzb/nSvQOxGrNwGvFJ/IwdKgcppgkXyxeZfS/5TbD3vaW7KMPiWnhHGIpRhPfdsxpqNXMoZ6MMYTc74xW5d8uhf9pKqgw/sYSpZAXVpfCJsgyHwBTzc70VdTatO+SxrNwZ0pJp3VjtP8fAfDFu1QU1z+4kRnPEglNPI65zOVw==</sig><time value="1423655565">2015-02-11 13:52:45</time><type val="2"/></info>

    1. Take the url in the AntiVirus content storage location for AntiVirus Agent version 8.2 + and  append /av64bit/versions.id

    http://cache.lumension.com/avcontent/av64bit/versions.id

    1. Verify that you get an xml file.
    Verify that the MirrorSDK config file is set up ok.
    1. On the EMSS Server, go to the Endpoint Distribution Services folder location
    2. /%Install Path%/Endpoint Distribution Services/providers.cnf
    3. Verify that the WWW_DIR and WWW_DIR64 url setting are as follows:

    [WWW_DIR]

    name=x86 32-bit anti-malware database

    url=http://cache.lumension.com/avcontent/av32bit

     

    [WWW_DIR64]

    name=x86-64 64-bit anti-malware database

    url=http://cache.lumension.com/avcontent/av64bit

     

     

    AirGap Environment

     

    Verify that the content is accessible

     

    1. Take the url in the AntiVirus content storage location for  AntiVirus Agent version 8.2 + and append /av32bit/versions.id

    Example: http://localhost:10000/avcontent/av32bit/versions.id

    1. Verify that you get an xml file similar to:

    <?xml version="1.0"?>

    <info><all><id value="45609"/><md5 value="7ed73b2dc23df8f99446019196635cb4"/></all><sig>wuoGT/ezUbm2JXIl29s/adxyNS7f3u1H8By+H22bnUsh2k+b/Bh2aBUmiubQguI748IOpQr1rOfCm+9JYeoDNfwxPE1INCvwjJRIzH2NTNCuXxkRtzY6l8IkeC26TYX8rJ1985n5oJrtmQ1DQ9090wlthxurs2EVcHnK/+2+n+9+RCzb/nSvQOxGrNwGvFJ/IwdKgcppgkXyxeZfS/5TbD3vaW7KMPiWnhHGIpRhPfdsxpqNXMoZ6MMYTc74xW5d8uhf9pKqgw/sYSpZAXVpfCJsgyHwBTzc70VdTatO+SxrNwZ0pJp3VjtP8fAfDFu1QU1z+4kRnPEglNPI65zOVw==</sig><time value="1423655565">2015-02-11 13:52:45</time><type val="2"/></info>

    1. Take the url in the AntiVirus content storage location for     AntiVirus Agent version 8.2 + and  append /av64bit/versions.id

    Example:  http://localhost:10000/avcontent/av64bit/versions.id

    1. Verify that you get an xml file.

    If an error is returned, then IIS has not been configured correctly. Check that the local definition repository has been configured correctly and that the MIME Types have been set up correctly.

     

    Verify that the local definition repository has been configured correctly

     

     

    v1 and v2 must be at the root.

     

    Verify that the MIME Types are correct

    View the MIME Types for each of the virtual directories

    • avcontent (or whatever the customer has called it)
    • v1
    • v2

    Each of these should have the following:

    Extension :    .*

    MIME Type:    application/octet-stream

     

    Extension:    .*.

    MIME Type:    application/binary

     

     

    If not, instructions on how to set these up are included in the AirGap document.

    (See Step 1: Create Physical and Virtual Directories for Storing Lumension AntiVirus Content on the Air-Gap (Disconnected) Server (Page 46 of the AirGap doc).

     

    Verify that the virtual directories are mapped correctly
    1. In IIS right click on the v1 virtual directory and click  ‘Explore’

    This is where it should be mapped too:

    1. In IIS right click on the v2 virtual directory and click     ‘Explore’

    This is where it should be mapped too:

    1. In IIS right click on the avcontent virtual directory and     click ‘Explore’.

    This is where it should be mapped to:

     

     

    Verify that the MirrorSDK config file is set up ok.
    1. On the EMSS Server, go to the Endpoint Distribution Services folder location
    2. /%Install Path%/Endpoint Distribution Services/providers.cnf
    3. Verify that the WWW_DIR and WWW_DIR64 url set correctly.  The following shows how they should be set when the local URL is: http://localhost:10000/avcontent

    [WWW_DIR]

    name=x86 32-bit anti-malware database

    url=http://localhost:10000/avcontent/av32bit

     

    [WWW_DIR64]

    name=x86-64 64-bit anti-malware database

    url=http://localhost:10000/avcontent/av64bit

     

     

    AV content not coming down to the endpoint

     

    Turn on Trace logging on the endpoint

    1. Edit the Agent Policy Set and set the logging level to Trace.

     

    Verify that the content is accessible from the endpoint

    1. With verbose logging turned on, the URL being polled for  content can be found in LmHost-Main.log

    JanusAv::Bitdef::BitdefController::UpdateAntiMalwareDatabase - Start to update anti-malware database, settings are: [

    --- UpdateOptions ---

        server_:http://AZ-LEMSS-01v.lumension.lcl/UpdateStorage/AntiVirus/Definitions/av64bit

        localPath_:

        debug_:0

        useVersionId_:1

        proxy_:

        proxyAuth_:

        connectTimeout_:10

        ioTimeout_:30

        systemDir_:

    --- UpdateOptions ---

     

    ]."

    Append Versions.id to the url and attempt to access using a browser:

    http://AZ-LEMSS-01v.lumension.lcl/UpdateStorage/AntiVirus/Definitions/av64bit/versions.id

     

    Get the error code from the lmhost-main.log

    1. Find the most recent instance of this message:

    2015-02-09-18:14:29,435ea50,critical,av,"Updater::Update - pUpdateInterface_->PerformUpdate() failed. error:[-2112]"

    The following is a list of error codes and what they mean which gives an indication to what is causing the error.

                                                                         

    Result

    Return Code

    Explanation

    BD_UPD_RET_Invalid_Opt           

    -1000

    An invalid option value was passed to SetOption() -when   COM is not used or to SetOptionInt or SetOptionBSTR-when COM is used , for   the latter two functions this value will be find in retVal parameter not as   returned value.

    BD_UPD_RET_Invalid_Param  

    -1001

    An invalid parameter was passed to SetOption() -when COM   is not used or to SetOptionInt or SetOptionBSTR-when COM is used,  for   the latter two functions this value will be find in retVal parameter not as   returned value.

    BD_UPD_RET_BadServerName 

    -2000

    The server name or the complete location for update is not   correct.For the COM interface this code will not be found as returned value,   but in the retVal parameter.

    BD_UP_RET_BadProxyName           

    -2001

    The proxy set with SetOption was not valid.

    BD_UPD_RET_BadRemoteLocation

    -2002

    The remote location is not valid.For the COM interface   this code will not be found as returned value, but in the retVal parameter.

    BD_UPD_RET_BadLocalPath  

    -2003

    The local path is not valid.For the COM interface this   code will not be found as returned value, but in the retVal parameter.

    BD_UPD_RET_InvalidObject 

    -2004

    The BDUpdateServiceInterface was not properly initialized,   when COM is not used, or internal objects of the instantiated COM object were   not properly initiated.

    For the COM interface this code will not be found as   returned value, but in the retVal parameter.

    BD_UPD_RET_InstallFailed 

    -2005

    The installation process has failed.For the COM interface   this code will not be found as returned value, but in the retVal parameter.

    BD_UPD_RET_InstallCouldNotCreateSymlink

    -2006

    Creating symlink for bdcore failed

    BD_UPD_RET_CouldNotConnect

     

    -2100

    Connection to the specified server failed

    BD_UPD_RET_HttpError202          

    -2101

    The Http status code was 202

    BD_UPD_RET_HttpError403          

    -2102

    The Http status code was 403

    BD_UPD_RET_HttpError404          

    -2112

    The Http status code was 404

    BD_UPD_RET_CouldNotLogin  

    -2103

    The authentication info specified was not valid - the   server/proxy sent a 401/407 code (meaning that the authentication info was   not valid)

    BD_UPD_RET_InvalidUrl            

    -2104

    The url is invalid

    BD_UPD_RET_InvalidWriteParam

    -2105

    The Update SDK was unable to create some file - usually   this happens when the local path does not have the proper directory structure

    BD_UPD_RET_HTTPError             

    -2106

    Some other HTTP error, greater than 400

    BD_UPD_RET_WriteError            

    -2107

    A write error has produced - when receiving a HTTP   response (usually when downloading a file)

    BD_UPD_RET_IOTimeout             

    -2108

    Some I/O operation timed out - when receiving a HTTP   response (usually when downloading a file)

    BD_UPD_RET_IO_RecvError          

    -2109

    This might happen if the Windows HTTP Library is not   functioning properly

    BD_UPD_RET_IO_SendError          

    -2110

    This might happen if the Windows HTTP Library is not   functioning properly

    BD_UPD_RET_InvalidResponse

    -2111

    The response from the server was invalid

    BD_UPD_RET_ServiceStopped 

    -2200

    The service has been stopped

    BD_UPD_RET_Unknown_Error    

    -3000

    An unknown error has occured. The best way is to set debug   on and see what happen.For the COM interface this code will not be found as   returned value, but in the retVal parameter.

     

     

     

     

     

    How to Stop Norman downloads when upgraded Server has all Endpoints upgraded to 8.2.

     

    1. Log onto the Server hosting the DB.
    2. Run the following query on the db:

    USE UPCCommon

    SELECT * FROM ConfigString where ItemId in (SELECT ItemID FROM ConfigItem WHERE ItemName = 'AVConfig')

     

    1. This will return a string SIMILAR to the following:

    <requestlist>

    <request name="AntiVirusEngineAndDefinitions" jobtype="HTTP">

    <schedule intervalminutes="120" daily="false" updateDateTime="2015-02-09T15:00:00.000">

    <filechannel>System.Net.FileAggregator</filechannel>

    <filereadysignal>AntiVirusFileUpdateReady</filereadysignal>

    <goldsignal>AntiVirusGoldFiles</goldsignal>

    </schedule>

    <hashfile>http://cache.lumension.com/antivirus/avfilelist.xml</hashfile>

    <hashfile>http://cache.patchlinksecure.net/antivirus/avfilelist.xml</hashfile>

    </request>

    <request name="BDContent">

    <schedule intervalminutes="120" daily="false" updateDateTime="2015-02-09T15:00:00.000"/>

    <hashfile>http://cache.lumension.com/avcontent</hashfile>

    </request>

    </requestlist>

    1. Remove the request for AntiVirusEngineAndDefinitions leaving an xml string like:

    <requestlist>

    <request name="BDContent">

    <schedule intervalminutes="120" daily="false" updateDateTime="2015-02-09T15:00:00.000"/>

    <hashfile>http://cache.lumension.com/avcontent</hashfile>

    </request>

    </requestlist>

     

    1. Stop EDS
    2. Run the following query on the DB:

    UPDATE configstring

    SET StringValue = '<requestlist>

    <request name="BDContent">

    <schedule intervalminutes="120" daily="false" updateDateTime="2015-02-09T15:00:00.000"/>

    <hashfile>http://cache.lumension.com/avcontent</hashfile>

    </request>

    </requestlist>'

    WHERE ItemId in (SELECT ItemID FROM ConfigItem WHERE ItemName = 'AVConfig')

    (Ensuring that you use the string as extracted from the db, as this contains polling frequency details.)

     

    1. Now only BD content will be pulled down. Also Norman information will be lost from the UI.

    (Subscription Updates -> Configure… -> AntiVirus tab)

     

    How to update definitions on a disconnected Endpoint.

     

    1. On a connected endpoint go to:

    C:\ProgramData\HEAT Software\EMSSAgent\data\persist\live\AV\ScanEngine

    Open the Live.txt and using the number in this file, copy that directory to a USB key.

    1. Stop EMSS Agent on the disconnected endpoint
    2. Turn off hardening on the endpoint.
    3. Go to: C:\ProgramData\HEAT Software\EMSSAgent\data\persist\live\AV\ScanEngine     and make a note of the largest directory number. Create a new directory and increment the largest directory number by one. Delete the smallest     directory number.

     

     

     

    1. Open the USB key and copy the contents of the directory to     C:\ProgramData\HEAT Software\EMSSAgent\data\persist\live\AV\ScanEngine new  directory] on the disconnected endpoint.
    2. Update the Live.txt to reflect the number of the new     directory.
    3. Restart EMSSAgent

    Affected Product(s)

     

    Ivanti Endpoint Security (IES) 8.5U1+

    Heat Endpoint Management and Security Suite (EMSS) 8.5 RTM

    Lumension Endpoint Management and Security Suite (LEMSS)