How to troubleshoot a Patch and Compliance (vulnerability) scan

Version 19

    Verified Product Versions

    LANDESK Management Suite 9.5LANDESK Management Suite 9.6LANDESK Management Suite 2016.x

     

    This document illustrates the files, registry, settings and other information necessary to effectively troubleshoot a vulnerability scan.

    In addition this document walks through the steps that a basic Patch and Compliance scan (otherwise known as a vulnerability scan) takes.

     

    This article will not describe every single step that the Vulnerability Scanner takes, but those steps where a failure can occur.

     

    For the purposes of this document a simple scan is performed by running the following at the client command line:

    vulscan /scan=0 /showui

     

    This command tells the vulnerability scanner to scan Windows vulnerabilities (type 0) and to show a user interface.

     

    The name "LDMS2016" and "LDMS2016_v###" will be seen throughout this document.   This refers to the Core Server name of "LDMS2016" which is the name of the core server that the author had when creating the document.

     

     

    Settings

    The settings that control how the vulnerability scanner will behave are stored in the Distribution and Patch Settings within the Agent Settings tool.

     

    These settings control behaviors such as user input options, Cloud Services Appliances patch options, scanner CPU utilization, etc.

     

    These settings are stored in the LANDESK Database in the AgentSettings table and physically on the core server in the

    \Program Files\LANDESK\ManagementSuite\ldlogon\AgentBehaviors folder.  The Distribution and Patch Settings are stored in the AgentBehavior_(Corename)_v###.xml file within this folder.

     

    AgentBehaviorsXML.jpg

                                                                                Example

     

     

    Product Licensing

    The categories available to scan and repair are controlled by the Product license that has been purchased and activated on the core server.

     

    The following graphic shows the categories available within the Download Updates function within the Patch and Compliance tool for those with a license for all categories.

          DownloadUpdatesCategories.jpg

    Click for full size

    For Product Licensing support Contact LANDESK support and select the Product Licensing option.

     

    Registry Keys

     

    Core

    HLKM\Software\LANDESK\ManagementSuite\PatchManagement\WebServiceMaxThreads

    This key does not exist by default and should only be created with an understanding of how this key works and the full ramifications of creating this key and changing the default value.  This changes the number of default threads

     

    This key is documented here: https://community.landesk.com/docs/DOC-36027#jive_content_id_Increasing_the_Number_of_Web_Process_to_Database_Threads

     

    Client

    HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\landesk\managementsuite\WinClient\Vulscan

     

    This registry key contains the following information:

     

    NameTypeData
    Description
    AgentBehaviorREG_SZLDMS2016_v495The agent behavior that vulscan will use when operating
    AlternateRebootBehavior

    /rebootIfNeeded is called from 3 possible locations during a client configuration.  It is hard for the task-handler version of the caller to know that a one-time-only (client-config-only) reboot override has been specified.  So all installers just call vulscan with the /UseAlternateRebootBehavior.  If vulscan can find the string value of "AlternateRebootBehavior" in the vulscanreg key, it'll act as if the behavior was passed by the command line.

    CommandLineREG_SZThe command line that was used to launch vulscan
    ComputerIdn.LDMS2016REG_DWORD0x00000006When run in a /showui mode the ComputerIDN is accessed locally from the registry rather than needing a separate GetSystemIDN for the UI through a second web service call to the core.  This value matches the ComputerIDN identifier in the LDMS database.
    KLBehaviorREG_SZLDMS2016_v517This refers to the LANDESK Antivirus behavior.  This will exist even if LANDESK Antivirus is not installed on the client.
    LastReportedReboot.LDMS2016REG_DWORD0x00000001
    trustedfilelistREG_SZLDMS2016_v861Trusted file list used for LANDESK Endpoint Security.  This will be present even if EPS is not installed or trusted file lists are not configured for this client.

    Note: The populated "Data' entries are provided as an example.  Yours will differ.

     

    The VulscanReboot key should NOT be modified, deleted or created.   This is a volatile registry key used by the vulnerability scanner.  Creating this key manually will create a persistent registry key that does not go away and will cause reboot loops and/or other undesirable behavior.

     

     

    Gathering information for LANDESK Support

     

    The vulnerability scan log files are located in the C:\ProgramData\LANDESK\log folder.

     

    When in doubt just .ZIP up the entire folder and send it.

     

    Otherwise the following logs should be gathered:

     

    • vulscan*.log
    • statusdlg*.log

     

    It is very useful to turn on Xtrace with the following enabled from the registry key prior to duplicating the problem and gathering logs:

     

    From HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\landesk\managementsuite\LogOptions:

    2016-07-18_11-23-22.jpg

    How To: Enable XTrace Diagnostic Logging for the LANDESK Core and Clients

     

    Tips and Tricks

     

    Vulscan can be used as a shortcut to open various folders.  The following should be run on the client command line:

     

    Vulscan e - Open the folder where Vulscan resides

    Vulscan c - Open the LDClient folder,

    Vulscan log - Open the ProgramData\LANDESK\Log folder

     

    Issue: Cannot open vulscan logs folder from the command line using "vulscan e"

     

    LANDESK Patch and Compliance (vulnerability) Scan Process Flow

     

    It is important to note that the following must all be able to take place:  Client contact to core through IIS and several web services.  Core contact to Database Server  Core Contact to client  Correct permissions on core ManagementSuite\Incoming directory and \ManagementSuite\LDLOGON\VulnerabilityData and VulscanResults folders.

     

    Note that issues can come and go during a vulscan.  This would indicate intermittent issues.  Most of the time this occurs when the server or database has connectivity issues or are too overwhelmed to respond to requests.

     

    Step 1  - Contact the LANDESK Core Server

    The vulscan engine attempts to contact the core server by checking the HKLM\SOFTWARE\Intel\LANDESK\LDWM registry key.  The client tries to contact vulcore.asmx through the WSVulnerabilityCore web service.  Thus the client needs to be able to contact the core, IIS needs to be available, the app pool needs to be running, and the database needs to be able to contact the core. 

     

    Good Vulscan.log entry

    Fri, 15 Jul 2016 11:38:20 Core server name found in HKLM\SOFTWARE\Intel\LANDesk\LDWM: LDMS2016

    Fri, 15 Jul 2016 11:38:20 File C:\Program Files (x86)\LANDesk\Shared Files\ProxyHost.exe version within specified

    Fri, 15 Jul 2016 11:38:20 Attempting to connect to proxyhost

    Fri, 15 Jul 2016 11:38:20 connect to proxy result: 0

    Fri, 15 Jul 2016 11:38:20 Using proxyhost to communicate with the core

    What could go wrong?

    Certificate Based Authentication - New Secure Client information

     

    LDMS 2016 Enhanced Security Mode

     

    If core has been upgraded and you have copied the .CRT, .KEY and

    Client unable to connect to the core server

     

    Error: "Host not found. Retrying"

    Bad vulscan.log entry:

    Fri, 15 Jul 2016 13:50:16 In SendRequest: Action = SOAPAction: "http://tempuri.org/GetHashForFile"

     

    Fri, 15 Jul 2016 13:50:16 SendRequest: SOAPAction: "http://tempuri.org/GetHashForFile"

     

    Fri, 15 Jul 2016 13:50:16 Action SOAPAction: "http://tempuri.org/GetHashForFile" failed, socket error: 0, SOAPCLIENT_ERROR: 5.  Status code: 503, fault string:

    Fri, 15 Jul 2016 13:50:16   Retrying in 0 seconds...

    Fri, 15 Jul 2016 13:50:16 Action SOAPAction: "http://tempuri.org/GetHashForFile" failed, socket error: 0, SOAPCLIENT_ERROR: 5.  Status code: 503, fault string:

    Fri, 15 Jul 2016 13:50:16   Retrying in 9 seconds...

    Fri, 15 Jul 2016 13:50:19 Last status: Retrying in 6 seconds...

    The client makes a SOAP request to the core server webservice and gets HTTP error 503 - Service Unavailable

     

    Note: The default timeout for Vulscan to connect to the core is 10 minutes.   Connection will fail after this time.


    Basic Troubleshooting

      • Does the HKLM\SOFTWARE\Intel\LANDESK\LDWM registry key have the correct core name listed?
      • Can you ping the core server?  Try IP address, netbios name, and FQDN
      • Does the client have connectivity otherwise?
      • Can you browse to http://coreservername/WSVulnerabilityCore/vulcore.asmx from the client browser?
        • Is the LDAppVulnerability application pool running on the core and is is the identity assigned to it correct?
        • Is IIS running on the core?

     

    Useful Articles

    IIS Troubleshooting and LANDESK Management Suite: 101

    How to troubleshoot IIS using Log Parser Studio from Microsoft

     


    Core server unable to talk to the database

     

    This error shows that something is wrong with the core to database communication or web service to database communication.  This can be a simple connectivity issue, database too busy, IIS/ASP.NET, etc.

     

    Error: "Server busy"

     

    Bad Vulscan.log entry:

    Fri, 15 Jul 2016 13:31:07 In SendRequest: Action = SOAPAction: "http://tempuri.org/ResolveDeviceID"

     

     

    Fri, 15 Jul 2016 13:31:07 SendRequest: SOAPAction: "http://tempuri.org/ResolveDeviceID"

     

     

    Fri, 15 Jul 2016 13:31:22 Action SOAPAction: "http://tempuri.org/ResolveDeviceID" failed, socket error: 0, SOAPCLIENT_ERROR: 7.  Status code: 500, fault string: Server was unable to process request. ---> A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - The remote computer refused the network connection.) ---> The remote computer refused the network connection

    Fri, 15 Jul 2016 13:31:22   Retrying in 5 seconds...

    Fri, 15 Jul 2016 13:31:25 Last status: Retrying in 2 seconds...

    Fri, 15 Jul 2016 13:31:26 Last status: Retrying in 1 seconds..

    The client does a SOAP request to the core web service to resolve it's device ID and gets HTTP Error 500 - Internal Server Error

     

      • Can you browse from the client to http://<coreservername>/WSVulnerabilityCore/Vulcore.asmx?
      • Is the core server overloaded or is the database overloaded causing a lack of a timely response?
      • Do other functions that depend on database connectivity work?  (Inventory Scan, doing a search for computers, running a LANDESK query, etc)
      • Is the APP pool assigned to the right version of .NET (4.0)
      • Is ASP.NET 4.0 bound to IIS?
      • Are the database credentials on core correct?  Check in the Configure Services drop-down in the LDMS console.
      • Is the database server up and running?  (Ping the database server, etc)

     


    Useful Articles

    Error: "Server Busy" When Running a Vulnerability Scan

    Step 2  - Core downloads and applies various agent settings

    At this step the core server downloads and applies various agent settings.  If a setting does not apply to the computer the file will be downloaded anyway.  (For example, if you have Endpoint Security in your

     

    Good Vulscan.log entry

    Fri, 15 Jul 2016 14:38:57 Checking whether to unzip 'C:\ProgramData\vulScan\ClientConnectivityBehavior_Apply.zip'.  Force: false

    Fri, 15 Jul 2016 14:38:57 GetFileHash: could not find "C:\ProgramData\vulScan\ClientConnectivityBehavior_Apply.zip"

    Fri, 15 Jul 2016 14:38:57 Calling 'PreApplyBehavior' in 'C:\Program Files (x86)\LANDesk\LDClient\ClientConnectivityBehavior_Apply.dll'

    Fri, 15 Jul 2016 14:38:57 Client connectivity settings pre-apply dll

    Fri, 15 Jul 2016 14:38:57 Allowing to download from the source

    Fri, 15 Jul 2016 14:38:57 Downloading trusted certificates

    Fri, 15 Jul 2016 14:38:57 In SendRequest: Action = SOAPAction: "http://tempuri.org/GetHashForFile"

     

    Fri, 15 Jul 2016 14:38:57 SendRequest: SOAPAction: "http://tempuri.org/GetHashForFile"

     

    Fri, 15 Jul 2016 14:38:57 Success

    Fri, 15 Jul 2016 14:38:57 Self update: files are up to date.

    Fri, 15 Jul 2016 14:38:57 Last status: Done

    Fri, 15 Jul 2016 14:38:57 Calling 'ApplyBehavior' in 'C:\Program Files (x86)\LANDesk\LDClient\ClientConnectivityBehavior_Apply.dll'

    Fri, 15 Jul 2016 14:38:57 Successfully loaded ClientConnectivityBehavior_apply behaviors from 'C:\ProgramData\vulScan\ClientConnectivityBehavior_LDMS2016_v499.xml'.

    The client checks it's file hash for the behavior file and compares it through a SOAP request to the core web service function "GetHashForFile".

    It then applies the behavior to the client.

    What could go wrong?

    Client cannot access the AgentBehaviors folder on the core server

     

    The client needs to be able to access the \LDLOGON\Agentbehaviors folder on the core server.  It then downloads the agent behavior .XML files and applies them if they pertain to the computer, otherwise the settings come down, but they are not applied.

     

    Error: " 'Applying XXX settings failed"

    Bad vulscan entry:

    Fri, 15 Jul 2016 15:20:53 Info: Core did not find file AgentBehaviors/RebootBehavior_LDMS2016_v503.xml

    Fri, 15 Jul 2016 15:20:53 Last status: File not found on core

    Fri, 15 Jul 2016 15:20:53 Info: Core did not find file RebootBehavior_Apply.zip

    Fri, 15 Jul 2016 15:20:53 Last status: File not found on core

    Fri, 15 Jul 2016 15:20:53 Info: Core did not find file AgentBehaviors/RCBehavior_LDMS2016_v511.xml

    Fri, 15 Jul 2016 15:20:53 Last status: File not found on core

    Fri, 15 Jul 2016 15:20:53 Info: Core did not find file RCBehavior_Apply.zip

    Fri, 15 Jul 2016 15:20:53 Last status: File not found on core

     

    Useful Articles

    Issue: Vulscan is not applying agent setting changes or is using an incorrect agent setting

    Error "Unable to get the setting from core" when running security scan (Vulscan.exe)

    Error: "Core could not find a file" when running vulscan on clients

    Error: "Failed to apply compliance settings" during vulnerability scan


     

    Step 3 - Vulscan loads and caches local MSI information

    In order for the vulnerability scanner to scan MSI information, the vulnerability scanner reads and caches the MSI information from the local computer's registry. This calls the MsiEnumProducts and MsiEnumPatches functions.  This depends on the existence of MSI.DLL in the C:\Windows\System32 directory.

     

    Vulscan.log entry:

    Fri, 15 Jul 2016 15:20:56   Loading MSI patch information

    Fri, 15 Jul 2016 15:20:56   product {7A4192A1-84C4-4E90-A31B-B4847CA8E23A}

    Fri, 15 Jul 2016 15:20:56   product {7E8833A1-AF24-4CAE-82DF-CFE14C14B94D}

    Fri, 15 Jul 2016 15:20:56   product {2EDC2FA3-1F34-34E5-9085-588C9EFD1CC6}

    Fri, 15 Jul 2016 15:20:56   product {E7D4E834-93EB-351F-B8FB-82CDAE623003}

    Fri, 15 Jul 2016 15:20:56   product {764384C5-BCA9-307C-9AAC-FD443662686A}

    Fri, 15 Jul 2016 15:20:56   product {5FCE6D76-F5DC-37AB-B2B8-22AB8CEDB1D4}

    Fri, 15 Jul 2016 15:20:56   product {3D6AD258-61EA-35F5-812C-B7A02152996E}

    Fri, 15 Jul 2016 15:20:56   product {45734758-4041-4EA8-8E62-DE661FC3879C}

    Fri, 15 Jul 2016 15:20:56   product {23170F69-40C1-2702-0920-000001000000}

    Fri, 15 Jul 2016 15:20:56   product {60EC980A-BDA2-4CB6-A427-B07A5498B4CA}

    Fri, 15 Jul 2016 15:20:56   product {1F1C2DFC-2D24-3E06-BCB8-725134ADF989}

    Fri, 15 Jul 2016 15:20:56   product {4C5EF2FF-EEA0-4314-8693-2AF565F14525}

    Fri, 15 Jul 2016 15:20:56 Loaded 12 products and/or patches

     

    Step 4 - Client requests vulnerability data information from core

     

    1. Vulnerability Definitions are downloaded from the LANDesk Patch Content servers and stored in the LDMS database connected to the core server.
    2. When a client calls in to scan for particular data, it requests Vulnerability data of a certain type (Windows Vulnerabilities, LANDESK Updates, Custom Definitions, etc) and for the particular OS the client is running.
      1. If the client is close to up to date the client gets the vulnerability data directly from the web service.  If it is not close to up to date it downloads the entire vulnerablity data set from the .XML file(s) mentioned below.
      2. The core server also writes this information to XML files in \Program Files\LANDesk\ManagementSuite\LDLogon\VulnerabilityData
      3. The file that gets written is "type_os-bitlevel_language.timestamp".   So a Windows 7 x64 client requesting Windows Vulnerability Data information would cause the core server to write a file called "0_win7-x64_enu.1315869631.xml" and also a compressed .XMLZ version of the same file.  Only the first requesting client causes the .XML file to be initially written.  Thereafter the other clients will simply receive this .XML file.

        Note: Deleting a definition will cause the entire .XML file to be re-written and all clients will redownload the entire .XML file.

      4. LDZIP.DLL in \Program Files (x86)\LANDesk\ManagementSuite\WSVulnerabilityCore\Bin is responsible for writing the compressed version.
      5. The client then downloads this .XMLZ file, decompresses it and begins parsing it.

       

      Good vulscan.log entry:

      Fri, 15 Jul 2016 15:20:56 -------------------ProcessRules of type 0----------------------

      Fri, 15 Jul 2016 15:20:56 GetData(): agentconfig =

      Fri, 15 Jul 2016 15:20:56 Getting definition data from core LDMS2016

      Fri, 15 Jul 2016 15:20:56 HTTP POST: http://LDMS2016:443/WSVulnerabilityCore/VulCore.asmx

      Fri, 15 Jul 2016 15:20:56 Setting a proxy...

      Fri, 15 Jul 2016 15:20:56 Setting socket timeout to 1000 * 60 * 4

      Fri, 15 Jul 2016 15:20:56 Success

      Fri, 15 Jul 2016 15:20:56 Last status: Done

      Fri, 15 Jul 2016 15:20:56 Parsing information

      Fri, 15 Jul 2016 15:20:56 Decompressing data

       

      What could go wrong?

       

      Error: "0x8db30194" (404) from vulscan

      Error: "Client user does not have administrator rights" when running Vulnerability Scan

      Error: "Failed. Cannot Interpret Data" when running a Security and Compliance scan

       

      Step 5 - Vulnerability scanning results are sent to the core server

      After scanning the results are sent to the core server through http://<corename>:443/WSVulnerabilityCore/vulcore.asmx.  At this point the Web services processes the results and creates a scan result file (in this case ScanResults_{A25894AD-E7E7-C042-86AB-5F8BBD866601}_0.vrz) that goes into the \Program Files\LANDESK\ManagementSuite\VulscanResults folder on the core.  This gets processed into the database and will show up in the Security and Compliance information for the client in the inventory.

       

      Good vulscan.log entry

      Mon, 18 Jul 2016 08:37:40 Sending scan results to core LDMS2016

      Mon, 18 Jul 2016 08:37:40 PutResultsAsFile uncompressed length: 4936

      Mon, 18 Jul 2016 08:37:40 compressed length: 914

      Mon, 18 Jul 2016 08:37:40 HTTP POST: http://LDMS2016:443/incomingdata/postcgi.exe?prefix=vulscanresults\&name=ScanResults_{A25894AD-E7E7-C042-86AB-5F8BBD866601}_0.vrz

      Mon, 18 Jul 2016 08:37:40 Setting a proxy...

      Mon, 18 Jul 2016 08:37:40 Setting socket timeout to 1000 * 60 * 4

      Mon, 18 Jul 2016 08:37:40 Success

      Mon, 18 Jul 2016 08:37:40 In SendRequest: Action = SOAPAction: "http://tempuri.org/PutResultsByFile"

       

      What can go wrong?

      Failures to send the results can come from some of the following issues:

       

      • Incorrect permissions to the \Program Files\LANDESK\ManagementSuite\IncomingData folder.
      • Incorrect permissions to the \Program Files\LANDESK\ManagementSuite\VulscanResults folder.
      • Missing, corrupted or incorrect version of postcgi.exe in the IncomingData folder.
      • Inability to contact the web service to place results.

       

      Failure in vulscan.log

      Mon, 18 Jul 2016 08:49:37 Sending scan results to core LDMS2016

      Mon, 18 Jul 2016 08:49:37 PutResultsAsFile uncompressed length: 4936

      Mon, 18 Jul 2016 08:49:37 compressed length: 913

      Mon, 18 Jul 2016 08:49:37 HTTP POST: http://LDMS2016:443/incomingdata/postcgi.exe?prefix=vulscanresults\&name=ScanResults_{A25894AD-E7E7-C042-86AB-5F8BBD866601}_0.vrz

      Mon, 18 Jul 2016 08:49:37 Setting a proxy...

      Mon, 18 Jul 2016 08:49:37 Setting socket timeout to 1000 * 60 * 4

      Mon, 18 Jul 2016 08:49:37 Failed http://LDMS2016:443/incomingdata/postcgi.exe?prefix=vulscanresults\&name=ScanResults_{A25894AD-E7E7-C042-86AB-5F8BBD866601}_0.vrz on server (0), server status: 404.

      Mon, 18 Jul 2016 08:49:37 HTTP Error 404.  Giving up.

      Mon, 18 Jul 2016 08:49:37 Last status: Failed: No response from core

      Mon, 18 Jul 2016 08:49:37 Failed to put vulnerability results to core as file: 8DB301B1

      Mon, 18 Jul 2016 08:49:37 Skipping repair step because scan errors occurred.

      Mon, 18 Jul 2016 08:49:37 ReleaseMutex 'Global\vulscan_scan' succeeded. Code: 0

      Mon, 18 Jul 2016 08:49:37 Failed

      In this case the postcgi.exe was missing in the incomingdata folder.  The service responded with an HTTP 404 error "File or directory not found".

       

      Additional articles:

      Issue: Vulnerability Scans are not updating on the core

      Error: "HTTP Error 403" Vulscan Return Code 433

       

      Step 6 - Vulnerability scanner checks for autofix patches

      The vulnerability scanner then checks with the core server to see if there are any patches that should be autofixed at this time.  This is done through the http://localhost/wsvulnerabilitycore/vulcore.asmx web service using the GetAllPatches function.  If patches are found that need to be autofixed one of the following methods is called:

       

      • Getallpatches2 -  GetAutofix Patches for all definitions specified
      • GetAutofixPatchesForGroup - If scanning against a group, get all Autofix definitions for that group.
      • GetPatchesForGroup - Get all patches for a group (remember, you can push a repair job against a group and it will be able to scan and repair in one scan)
      • GetPatchesForVulnerability - Get all autofix patches for patches manually selected and scanned.

       

      The core then builds a list of the repair logic that vulscan will follow and it gets sent to the client through the webservice, the client then writes an .XML file to follow as it repairs patches.   This information is all of the repair logic from the definition.