13 Replies Latest reply on Dec 13, 2018 5:23 AM by luke.edmonds

    Device Status not showing correctly

    luke.edmonds Apprentice

      Hi all,

       

      We have a problem with communication between our Ivanti Core Server and our devices. While I have been troubleshooting this issue I can't seem to make any logic of this issue and it is most likely one for support but any help anyone can give me would be much appreciated. The best thing would be if we can't fix it would be to get as much information for support as possible such as logs.

       

      So the problem is that our Ivanti EPM Core server has 800+ devices. A large portion of these devices display incorrect device status and it will display nothing or display devices as being offline when they aren't. I also have a problem where devices online will not pickup anything in a scheduled task for example a new agent package or software, instead they will sit in pending or failed and give a result like "Policy has been made available" or  "Unable to contact the specified machine. The machine may be off or unreachable".

       

      The devices having this issue don't seem to fit any logic and so all logical reasons for this happening seem to have gone out the window at the moment, I can't find a pattern or a trend.

       

      Machines are appearing offline but are still pingable from the core server and from other devices.

      Machines are appearing offline but are not pingable from the core server but are from other devices.

      Machines are appearing online but won't action any schedule tasks and provide results as previously mentioned.

      All machines or at least the ones in the console (being managed that I am aware of) I can see are doing a daily inventory and vulnerability scans so they must be communicating with the core.

      I cannot telnet on the LANDesk ports to some machines that are pingable but can to others. (Ports: 9595, 9535, etc). About Ports used by Ivanti Endpoint Manager ( EPM ) / LANDESK Management Suite ( LDMS ) - full list

      I have also done tests by pinging a device and tried to browse to "http://ClientIP:9595/allowed/ldping" and "http://ClientName:9595/allowed/ldping" and got inconsistent results where some devices have been successful and others not successful as per Device statuses (offline / online) are not displayed correctly on the core server or are not displayed at all - Basic Troubleshooting guide for the LDMS 9.5 Console

      Wondering what to do if I can ping a device from the core and others devices but can't telnet using 9595 and can't browse successfully to "http://ClientName:9595/allowed/ldping" or  "http://ClientIP:9595/allowed/ldping".

      Also to factor out network issues, some of the machines with the issues are on the same subnet, vlan, IP range and physical site. Some of the devices are on desks next to each other.

      The machines having issues are a mixture of Windows 7 and Windows 10 and are in the same OU in AD and get the same Group policy settings.

       

      The only thing I can put this down to is some local settings manually changed and tampered with for example Windows firewall turned on and blocking something like LANDesk inbound services but I don't think this would be the case and will confirm.

       

      Is there any specific log files I should be looking at which might be of some help with this issue. I am aware all the logs I probably need are in C:\Program Files\LANDesk\ManagementSuite\log and C:\ProgramData\LANDesk\Log but am not sure what I am looking for.

       

      I will continue troubleshooting and if I have any new ideas or get any further I will update this discussion.

       

      Kind regards,

      Luke

        • 1. Re: Device Status not showing correctly
          luke.edmonds Apprentice

          Hi,

           

          I should add that I thought maybe this is a problem with the scheduler service somehow or is related to this but I can see that the account this runs as is a service account which happens to be a domain admin and has local administrator access to the machines to do what it needs to do so this shouldn't be a blocker. I am also passing all ports to my network team however this can't be a network issue if some devices are fine but others not on the same vlan and subnet, etc.

           

          Kind regards,

          Luke

          • 2. Re: Device Status not showing correctly
            luke.edmonds Apprentice

            Hi,

             

            I have just checked on a colleagues machine having this issue however his machine is un-managed rather than managed like the rest of the machines having this issue. I managed to do an agentless scan on his machine to get a full inventory of his machine however soon as I look at it in the agentless scan folder is shows the machine as being offline but it is pingable from the core server. Soon as I try use advanced agent deployment it says the policy has been made available and never gets to the machine. Furthermore the http://clientIP/Name:9595/allowed/ldping doesn't load.

             

            This was done with all local firewalls turned off which makes no difference.

             

            Kind regards,

            Luke

            • 3. Re: Device Status not showing correctly
              phoffmann SupportEmployee

              So to help along a bit.

               

              1. Troubleshooting agent status (excellent article) info on here ==> How to troubleshoot Agent Discovery .

              2. A lot of such problems tend to come down to a combination of DNS (Wireshark will help with that) and/or Firewall blocks. (You'd be astounded how rare it is for me to run across actually *RELIABLE* / accurate DNS ...)

              3. Make use of the "DIAGNOSTICS" tool ... that helps a LOT and shows you "expected" versus "actual" values (expected == grey // conflicting actual == red). So Right-click a device & select this option here ...


                ... then click HERE for the "real time discovery" (essentially "Poke CBA & tell me what you get") ... that can be a HUGE help in chasing down DNS / Firewall issues & such.

                Discrepancies (such as "so I pinged this IP and expected device A to respond, but device B responded instead") will be highlighted in RED (so pretty obvious).

               

              ... so depending on the outcome, you can then focus those suckers down. (And yes, you CAN configure a firewall to have a device appear as "offline" when it isn't). Alternatively, you can use the PDS2DIS command line, but the GUI / UI option is nice & breaks things down for you.

               

              USUALLY if you have "inconsistent" results from different devices, my initial suspect is DNS servers / records, based on personal experience.

              • 4. Re: Device Status not showing correctly
                phoffmann SupportEmployee

                Also - don't forget to check out the comments in the article. Useful stuff there too!

                • 5. Re: Device Status not showing correctly
                  luke.edmonds Apprentice

                  Thanks, I have spent a large portion of today looking at this problem and getting the evidence and information together to both contact support but also liaise with the other support teams here, Server Infrastructure and Network Infrastructure. I have ruled out networks almost completely as I have tested all ports require are open and its only some machines that aren't working out of all but DNS that is a very good shout and likely to be the cause by the sounds of it. I can't seem to find anything on the machines themselves blocking this and it wouldn't make sense that they are but DNS messing up that is quite likely especially since I believe the core server having issues was moved to a datacentre a little before I started this role. Thanks for the link to that article I did look through many articles about troubleshooting this kind of thing but didn't come across this article.

                  • 6. Re: Device Status not showing correctly
                    phoffmann SupportEmployee

                    Easiest way to "prove" DNS is the culprit is ... Wireshark.

                     

                    Run Wireshark on the Core & on the client you're testing ... that way you can see "where communications go off to" ... and "that they never arrive" (I would suspect) at your intended target. So that'd be one for Networks to look at.

                     

                    A surprisingly common thing I run into at various accounts ... either routers don't route it (for some reason), route it to the wrong place, and/or DNS records / servers need a kick up the backside.

                     

                    Wireshark will be your friend (when tracing the discovery requests).

                    • 7. Re: Device Status not showing correctly
                      luke.edmonds Apprentice

                      Hey man,

                       

                      Thanks for your help so far, I am still working through that document and troubleshooting but I have found out that this looks to be a DNS issue. I can't ping hostnames but can ping IP addresses and I browsed to an IP address and it is a different machine from the one showing in the console with that IP. Hopefully after troubleshooting, gathering evidence and working through that document I can get to the bottom of this and have it resolved today.

                       

                      Kind regards,

                      Luke

                      • 8. Re: Device Status not showing correctly
                        phoffmann SupportEmployee

                        Yeah no problems.

                         

                        If it makes your networking guys better, it's a (sadly) common problem. Most people just don't tend to notice how (in-)accurate their DNS records are without things like this .

                        • 9. Re: Device Status not showing correctly
                          luke.edmonds Apprentice

                          Okay so here is where I am at, I thought I had this resolved but nope not yet. Definitely some DNS issues happening though:

                           

                          I have a device which is online and can be pinged with the IP fine but not the hostname and http://ClientIP:9595/allowed/ldping works fine with no issues but replacing ClientIP with hostname doesn't work. Soon as I do anything in a schedule task and target this device it says the policy has been made available and just waits forever. I have worked through the document for previous machines but the results and resolution was slightly different with them. Currently working through the document for this machine and machines like this and have found in the PolicyTaskHandler.exe.log that as far as I understand the task/core server has discovered the device but failed to synchronise the policy. I guess the next step is to check policysync.exe and the raxfer log file. (This example is Skype deployment but previously this problem was discovered when doing an advanced agent deployment to existing machines, the content doesn't make any difference to the results).

                           

                             RollingLog : [Task: Skype upgrader - 8.44.0.78 latest(2), TaskID: 549, ProcID: 8536] : ***PolicyTaskHandler run options***

                          12/07/2018 14:43:47 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : Accelerated push: [True]

                          12/07/2018 14:43:47 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : Use DNS/WINS resolution for discovery: [False]

                          12/07/2018 14:43:47 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : Wait for machines to finish: [False]

                          12/07/2018 14:43:47 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : Maximum Push or Policy-supported push run time minutes: [15]

                          12/07/2018 14:43:47 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : Verbose logging: [True]

                          12/07/2018 14:43:47 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : Policy task handler: performing hash calculations on package : Skype upgrader

                          12/07/2018 14:43:52 INFO  8536:1     RollingLog : Redirected \\LANDeskCoreServer\ldlogon\ldms_app\Skype\Skype-8.34.0.78.exe to \\LANDeskCoreServer\ldlogon\ldms_app\Skype\Skype-8.34.0.78.exe

                          12/07/2018 14:43:52 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : Loading machine: [PC02528], IP: [10.60.6.117], MAC: [308D99BD023E], objid: [864], OS: [Windows 7 Professional 64], OSVersion: [6.1]

                          12/07/2018 14:43:52 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : LoadTaskTargets: Loaded target machine, ID is: [864], operating system is: [Windows 7 Professional 64] for machine: [PC02528]

                          12/07/2018 14:43:52 INFO  8536:1     RollingLog : NewTargetApi.LoadAdditionalClientData: Elapsed time to chunk computer idns and load additional data was: [00:00:00.0025059]

                          12/07/2018 14:43:52 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : LoadTaskTargets: There were [1] targets loaded

                          12/07/2018 14:43:52 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : PolicyTaskHandler starting to process task...

                          12/07/2018 14:43:53 INFO  8536:1     RollingLog : ETask.SetMachineStatus: Elapsed time to chunk computer idns and update the database was: [00:00:00.0078096]

                          12/07/2018 14:43:53 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : ProcessTask: Target count before FilterAllTargets is [1]

                          12/07/2018 14:43:53 INFO  8536:1     MDMConsole.MDMTaskHandler : MDMTaskHandler.FilterNodesAndProcess(): Called with 1 computer IDs, Task ID 549

                          12/07/2018 14:43:53 INFO  8536:1     MDMConsole.MDMTaskHandler : Queueing 0 devices to notify pipe.

                          12/07/2018 14:43:53 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : ProcessTask: Target count after FilterAllTargets is [1]

                          12/07/2018 14:43:53 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : DiscoverAndSynchronize: Machine [PC02528] with id [864] was indicated as ONLINE by Agent State or successfully discovered, attempting to synchronize policy...

                          12/07/2018 14:43:53 INFO  8536:1     RollingLog : TargetMachineContainer.MachineTargetOS: Operating System is: [Windows 7 Professional 64] for machine: [PC02528]

                          12/07/2018 14:43:53 INFO  8536:1     RollingLog : TargetMachineContainer.MachineTargetOS: Operating System is: [Windows 7 Professional 64] for machine: [PC02528]

                          12/07/2018 14:43:53 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : SyncPolicyTask: Synchronizing policy with the command: [C:\Program Files (x86)\LANDesk\LDCLient\PolicySync.exe -taskid=549], to machine: [PC02528]

                          12/07/2018 14:44:35 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : SyncPolicyTask: Failed to synchronize policy on machine: [PC02528], RAXfer return code: [-2147481845], RAXfer exitcode: [0]

                          12/07/2018 14:44:36 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : SetFinalTaskStatus: This is a policy-supported push task, final task status is [PULL_AVAILABLE]

                          12/07/2018 14:44:36 INFO  8536:1     RollingLog : [Task: Skype upgrader - 8.34.0.78 latest(2), TaskID: 549, ProcID: 8536] : PolicyTaskHandler finished processing task, setting task status to [PULL_AVAILABLE]..

                           

                          Also found in Wireshark a different result than I got to the other machines I seem to have resolved. This result seems to show ttl=128 no response found from the core to the device but right afterwards a reply from the device to the core and then it goes in a loop.

                           

                          A bit confused but I will muddle through.

                           

                          Luke

                          • 10. Re: Device Status not showing correctly
                            luke.edmonds Apprentice

                            As I dive into this deeper I am finding that there is no one problem here, it seems that I resolved some of the machines earlier and have found the issue and resolution for another machine which I believe is an exception to the general problems that most of the machines have. I still haven't managed to fix the general problem of online machines sitting in waiting/pending on scheduled tasks and not being available via DNS and only by IP.

                            • 11. Re: Device Status not showing correctly
                              luke.edmonds Apprentice

                              Hi,

                               

                              I haven't got this issue resolved overall yet however I found that we had some issues with our setup of Windows 10 devices which I am currently resolving which will resolve this issue I hope. I have engaged our Network and Inf teams just to make sure there is nothing wrong that side of things. I did raise a support ticket however I decided to close it until I have fixed Windows 10 and confirmed with Network and INF that we have no issues network wise and DNS wise. If this is still an issue after all this I will raise a support ticket. I am hoping it's a network or DNS issue and I can get it fixed soon.

                               

                              Kind regards,
                              Luke

                              • 12. Re: Device Status not showing correctly
                                phoffmann SupportEmployee

                                You may find that a handful of devices have other issues (such as overly aggressive configured firewalls, or other things), but resolving "80%" or so of your overall issues with DNS is usually a good endeavour & leaves you with a - hopefully much reduced - burndown list.

                                 

                                Let us know if you need any more help or so.

                                • 13. Re: Device Status not showing correctly
                                  luke.edmonds Apprentice

                                  Agreed that is exactly what I have found so far. Just getting a GPO in place to resolve the overly aggressive configured firewalls on Windows 10. As for the rest of the machines it is most likely a DNS issue. If it is DNS and I get this resolved hopefully as you said 80% or more should be resolved and then it will be down to the few devices that have individual and unique issues.