Patching and Applicability - Patches Are Not Showing As Applicable

Version 1

    You may find that one, a few, or all of your endpoints are not showing patches as applicable where they should be. We'll start by analyzing the process by which applicability is determined, and go into some universal troubleshooting steps. If those don't work, we'll delve deeper into your situation on a case by case basis.

     

    1.1: What is a DAU (Discover Applicable Updates) scan?

    In order for your endpoints to report their inventory to your Ivanti Endpoint Security server, they have to perform a DAU scan. The purpose of this DAU scan is to:

         a) Report current inventory to the Ivanti Endpoint Security server, and

         b) Provide information on what patches should be applicable.

     

    DAU scans can be started in a few different ways.

         a) The most common is a normally scheduled time. By default, DAU scans run every 26 hours. You can change this in the Agent Policy sets in Manage -> Agent Policy Sets view. The endpoint will also contain an attribute called "Patch DAU: Scheduling Frequency" which shows the frequency in hours between scans.

         b) By default, the scan also kicks off once your subscription replication completes. This can be configured under Tools -> Options -> Agents -> Discover Applicable Updates (DAU) Options.

         c) You create new content manually using Content Wizard and upload it to the server.

         d) You select "Discover applicable updates..." in the Scan Now wizard on the Information tab on the endpoint (note that the endpoint must check in with the server for this scan to go off. It does not "push" the notification, but rather puts up a flag for that endpoint the next time it requests an update from the server).

         e) You navigate to the endpoint directly, open the Endpoint Security Agent Control Panel, navigate to Patch and Remediation -> Patch Module, and click "Scan" at the top.

     

    **Note that the DAU scan starting can be inhibited by a few different factors. We will go into those in more detail in "Troubleshooting the DAU Scan task."

     

    In short, the DAU scan will cause your client to contact your Ivanti Endpoint Security server, and request patch information (.ospx files) about what's available to be downloaded based on the host OS. It runs through the files from the .ospx downloads, and compares everything that can be downloaded and patched with what is currently installed on the host machine (if your endpoint is in trace level logging, you can actually see this process happen file by file in the lmdetection.log file). It compiles information about any differences in the .ospx file with the current inventory, and when the analysis is complete, it sends a .xml file with that information up to the server and moves on to the next .ospx file, repeating the process until every applicable file has been analyzed.

     

    **The underlying processes on Non-Windows platforms are very similar to Windows platforms, but the engine driving these scripts migrates from the .Net framework to the Java framework. This is why the version and configuration of Java you're using is important for the DAU scan to run correctly on Non-Windows platforms.

     

    Starting in version 8.5, we've added a new feature to DAU scans, called the Delta Scan. This feature is designed to alleviate the strain on large environments by preventing a large amount of information from going to and from your database every time a DAU scan is run. When a DAU scan is first run, a registry key is created at *HKEY_LOCAL_MACHINE\SOFTWARE\PatchLink.com\cache* and a hash of the information in your inventory is stored there under *SYSTEM_HASH* . Every time a DAU scan is run, it hashes the current software inventory, and compares the new SYSTEM_HASH value with the value in your database to determine if there have been any significant software or registry changes that may indicate a new program was installed. If the values are the same, a much more condensed inventory report is sent to the server, and in-depth analysis is not performed on the endpoint. This is the Delta Scan in action, and it saves you a lot of time and system resources every time it occurs. We'll get back to troubleshooting this Delta Scan in "Troubleshooting the DAU Scan task."

     

    1.2: How is applicability determined?

    Patch applicability is determined on a file-by-file basis by analyzing the associated fingerprint of each file. Most vendors use a method of reading a .properties file, a registry key, or a file system object to determine what version of a program you have. They use this application fingerprint in their applicability analysis. We have also utilized this methodology, and all of our patches include a fingerprint for easy analysis. Some patches (particularly SQL-based patches) perform recursive registry key or file system scans, which can slow your DAU scan task significantly, but overall this methodology makes applicability analysis quick and accurate.

     

    The compiled information is sent up to the database, and the Ivanti Endpoint Security server pulls the information using a series of queries and filters to display inventory details, applicable patches, and more.

     

     

     

    2.1: Troubleshooting the DAU scan task (Windows)

     

    1. The DAU scan task is never kicking off on any of my endpoints.

         a) Confirm that the system task isn't disabled

              1. Navigate to Manage -> Deployments and Tasks. Check the "Discover Applicable Updates" System Task. Confirm that the "Disable" button lights up. If "Enable" can be selected, click on that. It should match the image below:

    DAU1.png

         b)  Confirm policy configuration

              1. Navigate to Manage -> Endpoints, and click on one of the affected endpoints.

                   i. Patch Hours of Operation determines when DAU scans and patching tasks can be executed. Ensure that there are no restrictions here for when you would like DAU scans to run.

                   ii. Verify that Patch: Communication Interval is a short enough timespan that several hours do not pass between check-in attempts, which may cause temporary disruptions. 30 minutes is appropriate for most environments.

         c) Confirm that Replication is working

                   i. Navigate to Tools -> Subscription Updates, and confirm that your Subscription Service History is showing Successful for Licenses, System, Patch / Components, and Patch / OS Packs recently. Note that if a replication of any sort has been going on for longer than a few hours, your replication may be stuck. Review Common Replication Questions and Replication stuck downloading updates for more information. A good replication can be seen below:

    DAU2.png

         d) Try to kick off the scan manually

                   i. Select "Discover applicable updates..." in the Scan Now wizard along the top of the endpoint information screen to kick off a scan when the agent checks in next. Alternatively, you can navigate to the endpoint and kick off a DAU scan immediately. Watch the results and continue with troubleshooting as appropriate.

         e) Check the last DAU scan status

                   i. Agent status should show Online, PR status should show online, and Last DAU scan status should show Success. You can click on the status hyperlink for more detailed information on the last DAU scan. If it was successful, continue to "The DAU scan task is successful, but I'm still not seeing any applicable patches on my endpoint(s)." Otherwise, continue to "The DAU scan task is failing."

        

     

    2. The DAU scan task is failing.

         a) A "190C" error is reported

              i. The 190C error is a generic error message, returned by Windows rather than the Ivanti product. There can be several reasons for this error message; confirm that the system has read/write permissions to the entire install drive, there is sufficient space for several downloads (sometimes an excess of 10GB is required for staging OS updates), and the system can reach the Ivanti Endpoint Security server. Missing script error messages can usually be resolved by uninstalling the client, rebooting, and reinstalling the client. You can usually see more specifics about the error message by clicking on it. If you cannot resolve the error on your own, open a ticket with Ivanti support.

         b) An "Unsupported OS" error message is reported, and the OS type is unknown

              i. The initial agent install determines the OS type based on the TTL result of a ping request. This is why the ICMP rule *must* be enabled and allowed in order for the install to complete successfully. Confirm that the TTL for that endpoint is between 64 and 128.

              ii. Confirm that the OS is supported by your version. If possible, upgrade to the newest version of the Ivanti Endpoint Security (a guide for this upgrade process can be found here: HEAT EMSS 8.5 Server Upgrade Guide )

         c) A "190A" error is reported

              i. This is usually caused by a caching problem. Try to run the Content Cleaner using the following switches: contentcleaner.exe -L -M -R -V

              (Visit Content Cleaner Usage For EMSS for more information)

              ii. If you can determine which package is failing, navigate to Review -> Other -> Packages, locate the patch that's causing the failure, delete it and then re-cache the file.

         d) Another error is reported

              i. Try to get information about the error message by clicking on the status hyperlink. Sometimes, a database connectivity issue can cause some error messages; confirm that the database is receiving updates, and that the clientadmin and serviceadmin accounts are not locked in the database, the domain, or the OS on the server your database resides on.

     

    3. The DAU scan task is successful, but I'm still not seeing any applicable patches on my endpoint(s).

         a) Confirm that your server is replicating successfully

              i) Navigate to Tools -> Subscription Updates, and confirm that your Subscription Service History is showing Successful for Licenses, System, Patch / Components, and Patch / OS Packs recently. Note that if a replication of any sort has been going on for longer than a few hours, your replication may be stuck. Review Common Replication Questions and Replication stuck downloading updates for more information. A good replication can be seen below:

              DAU2.png

         b) Confirm that there actually are applicable updates

              i) Try scanning with Windows Update to see if there are actually any applicable patches. Sometimes, a client will update itself using Windows Updater if you have it enabled and set to automatically download patches. Disable the automatic updating if it isn't done already.

         c) Force a full scan (bypassing a Delta Scan)

              i) Navigate to the endpoint you'd like to test and backup then delete the Patchlink Cache registry key at *HKEY_LOCAL_MACHINE\SOFTWARE\PatchLink.com\cache* . You can delete the entire key safely. Kick off a DAU scan and see if the applicable patches list updates. The next DAU scan will be a full scan, rather than a Delta scan.

         d) Ensure that data is flowing between the endpoint and your server

              i) Install Wireshark or Fiddler on your endpoint, and observe the traffic during a DAU scan. Blocked packages or data will appear, and you can configure proxy servers, firewalls or other networking rules to allow this type of traffic.

              ii) You can also navigate to Manage -> Endpoints, click on an affected endpoint, and go to the Inventory page. If a DAU scan was successful, some data about the endpoint should appear under the Software inventory.

    DAU3.png

         e) Verify that the new content exists on the Ivanti Endpoint Security server

              i) Change your filters in the Vulnerabilities/Patch Content to read:

                   Content type: Critical and Not Superseded

                   Vendor release date: After (within 1 month to prevent the query from pulling too much data)

                   Applicability: All

                   State: Enabled

                   Detection Status: All

              Click "Update View" and verify that new content is being downloaded. You can also search for the KB or CVE you know you're missing to ensure it's actually reached your server.