11 Replies Latest reply on Oct 12, 2018 2:13 AM by phoffmann

    Inventory scan failed :ldiscn32: the inventory server "XXX" did not respond

    BRajam Rookie

      Recently  we are facing issue with inventory scans not working on most of new client machines. We are able to successfully push the Landesk inventory agent  to a new PC. It seems to install fine but the PC is still sitting in Pending Unmanaged Client Deployment. It fails when sending scan results to the core server. When I tried to run a inventory scan manually ,it fails and displays the below error.

       

       

       

      Inventory Services seem to be up and running within the PC and core server.
      I can telnet from the PC to the core with ports 5007.
      Windows firewall is disabled.

       

       

      Appreciate your timely help and support on this

       


      Next, I havent configured anything under cloud services appliances,does it have anything do with this inventory scan fail?

       

        • 1. Re: Inventory scan failed :ldiscn32: the inventory server "XXX" did not respond
          phoffmann SupportEmployee

          As long as you're not using a CSA (Cloud Service Appliance) -- i.e. "are going over the internet", you won't need to have that stuff configued for inventory to work.

           

          Chances are this is -- pure and simple -- a networking problem (usually "shortname is used" / "can't resolve the name").

           

          Your device stays in "pending unmanaged" because the Core hasn't yet received an "actual" inventory scan, so all is logical so far.

           

          So - first thing:

           

          Run the following command line, see if that works:

           

          1 - Open up a CMD prompt (ideally -- "run as admin")

          2 - Go to "C:\Program Files (x86)\LANDesk\LDClient\"

          3 - Run the following command line (colour-coding things you need to replace with the SERVER IP to begin with).

           

          I'm using upper-case switches purely for better readability -- they're actually not case sensitive!

           

               ldiscn32.exe /V /F /SYNC /NTT=123.123.123.123:5007 /S="123.123.123.123" /I=HTTP://123.123.123.123/ldlogon/ldappl3.ldz

          ... where you need to replace the "123.123.123.123"-s with the actual IP of your Core.

           

          I want you to try an IP first, to eliminate your DNS being a problem .

           

          Now - assuming the above works, try replacing the "123.123.123.123"-s with the FQDN of your Core Server.

           

          If THAT works, try replacing the the "123.123.123.123"-s with the NETBIOS / shortname of your Core (at the latest, I would guess this is where it'll fail).

           

          What stuff does:

          /V ==> Run in verbose mode / with GUI enabled

          /F ==> Force a software scan (may as well get you a proper inventory)

          /SYNC ==> force sending the FULL scan file, not a delta file (needed since you didn't reach the Core yet).

          /NTT=... ==> Specifies how to talk to the inventory service.

          /S=... ==> Specifies the Core (overriding agent setting information !)

          /I=... ==> Points to the LDAPPL3 file on the Core which the client needs.

           

          ===============

           

          Super bonus round ... (further troubleshooting)

           

          Check what your agent setting on the Core for CLIENT CONNECTIVITY says "in theory" ...

           

          ... and then cross-check that with the agent behaviour file that's on the client (under -- "C:\ProgramData\vulScan\").

           

          It'll be called something like "ClientConnectivityBehavior_{rest_of_name}.xml" ... you're looking for the "CORESERVER"-tag, as per the below section:

           

          (This is from my lab, so no big secret ).

           

          That allows you to cross-check what reference your agent *DOES* use versus what it SHOULD be using. If those two are the same -- great.

           

          If they are NOT the same, chances are you are in a catch-22 situation (where your agent can't contact your core because it has bad info -- and can't correct the info because it can't contact your core).

           

          How to solve this? Conveniently - very easily.

           

          Run THIS command line (again from the LDCLIENT directory):

          vulscan /coreserver="123.123.123.123" /showui /updatebehavior

          ... that launches the vulnerability scanner with a few override options, specifying which Core to talk to (obviously change the IP address), forcing it to show UI (so you can see stuff as it happens) and specifying that it should ONLY check for updated agent settings (which it can then download).

           

          This isn't "the proper fix", but can get you out of a pinch.

           

          The "proper fix" is for you edit / fix your client connectivity agent setting, and/or re-build your agent installer (with the right config file?) to ensure that it's using up to date stuff.

           

          Couple of tips:

          • Don't use shortnames. DNS has enough issues with being reliable without NETBIOS messing things up.
          • If you can't trust your DNS (common enough, surprisingly), don't be shy about using IP!

            ... sure, you then can't use a DNS-alias when it's convenient, but equally, you won't get scerwed over by iffy DNS (or uses a HOSTS file entry for instance).

           

          Right - that should help .

          • 2. Re: Inventory scan failed :ldiscn32: the inventory server "XXX" did not respond
            BRajam Rookie

            Hi Phoffmann

             

            Many thanks for your detailed information.

             

            As you mentioned tried the command line scan with IP address of core server, but still  getting the same error

             

             

            Same error with FQDN and shortname as well.

             

            Core server is reachable from this client machine. any idea what could be wrong here?

             

             

            Next Super bonus round ... (further troubleshooting)
            I verified the ClientConnectivityBehavior_{rest_of_name}.xml, it does have the same as agent client connectivity  setting. FYI. I used shortname of coreserver in agent settings.

            As you mentioned,will create a agent with core IP address and give a try.

             

            Appreciate your help. Thanks in Advance

            • 3. Re: Inventory scan failed :ldiscn32: the inventory server "XXX" did not respond
              phoffmann SupportEmployee

              Hmm - interesting.

               

              I would argue that "Wireshark it" (from both ends) to see if the client is actually able to reach the Core. Generally, this is a case of "client cannot talk to the inventory service" ... which USUALLY translates as "can't resolve name", but could be blocked by something on the network (hence the suggestion to Wireshark it).

               

              In case the Inventory Service on the Core has gotten into a weird state, you may want to give it a re-start // check from another device whether it can send inventory all right.

               

              Let's start with those and go from there .

              • 4. Re: Inventory scan failed :ldiscn32: the inventory server "XXX" did not respond
                Rick.Smith1 Expert

                phoffmann & BRajam We've been having issues starting (we believe) around the June time frame. So pretty close to May when this was posted.

                - Wireshark trace done both on client\core during a failed attempt - No indication as to exact cause. A couple out of order packets, but appears to have been re transmitted successfully and not root cause.

                - Tried FQDN (our default settings) and shortname and IP

                - No IIS 503 errors that we've been able to find.

                 

                Same core server had been working for a long time before we started having this issue. We've not gone back and removed all server patches for from that time frame, and it probably wouldn't be feasible at this point. We've had a case open with Ivanti since it started, but we have yet to be able to identify root cause and why it started. The biggest impact is that it will cause provisioning to fail when it tries to do the inventory scan. I could sit there and retry the inventory scan over and over and within minutes it will both fail and succeed. No changes to how we call the server. NO debug logs are created even with the debug switch when it fails to connect to the core.

                 

                Rick

                • 5. Re: Inventory scan failed :ldiscn32: the inventory server "XXX" did not respond
                  phoffmann SupportEmployee

                  So - June had "a few issues" with IIS (Microsoft patches caused a bunch of problems) - which I believe are now "more resolved" (i.e. - latest patches should fix things).

                   

                  Keep in mind that - by default - your clients are now talking to a secure WWW-service first ... so that may be your problem (IIS).

                   

                  You can try to change that temporarily by disabling the use of the WWW-service in the agent behaviour ... talking to "just" the inventory service should work fine.

                   

                  That would at least make sense in terms of you not seeing any networking problems & IIS being the "piggy in the middle" with the issues not handing things down to the inventory service proper / into the LDSCAN folder?

                   

                  Give that a try ... if things "magically work" by eliminating the IIS factor here, then you may need to patch up on 'that side // have another IIS issues hopefully?

                  • 6. Re: Inventory scan failed :ldiscn32: the inventory server "XXX" did not respond
                    Rick.Smith1 Expert

                    phoffmann To be clear, I think what you are saying is to uncheck this option in the inventory settings for the client "Post to web service: Rather than copying scan files to a folder on the core server, this option sends scan files directly to a web service. This is more efficient and it is the new default." https://help.landesk.com/docs/help/en_US/LDMS/10.0/Windows/agent-h-inventory-settings.htm?cshid=InventorySettings_Scanne…

                     

                    We did analyze several event logs and the event logs do show that we seem to have been having these types of issues for awhile. Using log parser studio, we noticed we had a huge number of 503 IIS errors on the inventory scan application pool, so we increased the concurrent limits to eliminate the 503 errors.

                     

                    We did encounter some issues in June\July with the patching. Mainly what we were aware of was the inability to restart IIS without rebooting the server itself, nothing more. But around June the problems seem to have increased dramatically.

                     

                    I'll try not using the Post to web service. What i have noticed and traced is that PING tests will vary in the amount of time the results return. Sometimes seconds, other times several minutes. [UPDATE - Still failed with the post to web service unchecked]

                     

                    I also noticed (and reported) a lot of errors in the HTTPERR log that looks like this:

                     

                    Line 2188: 2018-10-02 20:08:08 172.26.2.247 52490 172.24.4.62 80 HTTP/1.1 POST /ScanInstructions/AgentStateServices/Report?deviceID={BLAH BLAH BLAH}&subnet=192.168.1.0--24;192.168.1.0--24;172.26.0.0--21 411 - LengthRequired -

                     

                    Using fiddler, I have attempted to replicate the POST against the core with the length specified (not sure if this is some bug where the clients are not POSTing sporadically with the length to avoid the error). Sometimes the core returns a 200 result in a few seconds, but other times its 10-11 minutes before I get a response back.

                     

                    Rick

                    • 7. Re: Inventory scan failed :ldiscn32: the inventory server "XXX" did not respond
                      phoffmann SupportEmployee

                      Yes - you were correct in regards to the "using secure www-service" that I mentioned. I usually include screenshots, just had my VM-s running maintenance stuff so couldn't. My apologies .

                       

                      So if *PING* bounces from "seconds to minutes", that smells of network issue(s) somewhere down the line (and my first suspect tends to be "web-caching / network caching appliances" usually ... so that'd NOT be something that IIS would affect.

                       

                      If you extended the resourcing on IIS, that's likely a good thing (and you'll want to check on how many 503's you're getting now) ... could be "simply" that IIS didn't have enough to handle your volumne.

                       

                      As a "cheat", copy the last 7 days (or whatever) worth of IIS logs into a separate directory & create ++ run the following batch script to get a feel for "in how much trouble" you are IIS wise (as well as which app-pools / when they have issues):

                      • Copy several days worth of IIS logs to a separate location (i.e. - "C:\XX\").
                      • Create a batch file with the following 2 commands in the same location (you don't want it in the default IIS log location , as you don't want to parse 2+ YEARS' worth of IIS logs ).

                        FINDSTR /S /N /C:" 503 " *.log > 503_errors.TXT
                        FINDSTR /S /N /C:" 500 " *.log > 500_errors.TXT

                      • ... run the batch & look at the file outputs ... and go from there.

                       

                      ... I've not really seen that kind of error in HTTPERR before. You may want to investigate that on IIS fora ... not sure if it's a legitimate error, or a symptomatic one (not all errors are "the honest cause"), so it may be a red herring or may be indicative of something else (I usually check for TimeOut errors in the HTTPERR to get an idea for how unhealthy IIS's resourcing is compared to getting stormed by requests).

                       

                      ====

                       

                      Going "straight through" to the Inventory Service should alleviate any IIS-centric load issues.

                       

                      If things like PING bounce from seconds to minutes, though, this sounds like you've got actual network issues somewhere as well. That won't be helped if there are IIS issues on top, but this would be more of a combination of issues (network latency & IIS load). But now, IIS (at least as far as inventory goes) shouldn't be a factor ... but network latency / response WILL still be a factor (as clients may just fail in talking to the Inventory Service directly).

                       

                      Hope that helps?

                      • 8. Re: Inventory scan failed :ldiscn32: the inventory server "XXX" did not respond
                        Rick.Smith1 Expert

                        Thanks Paul. We do have a clean wireshark trace of both the core and client exhibiting the issue and sent that into engineering to validate.

                         

                        I like your copy the logs to a different folder idea, we didnt need those historical logs, so I just wiped them after each change and monitored the status codes. We are at zero 503 errors at this point. We were at about 1/3 the number of our 200 status code counts, so it was pretty bad before.

                         

                        We are EPM 2017.3 SU3. I know our TRM\TAM saw the same errors on his test core. None of the peers I have reached out to are on the same version of EPM as I am, so it's difficult to validate if it's not something version specific. Of our client devices with the longest event logs still in tact that go that far back, as far as I can tell the clients started posting a lot more of these errors with our upgrade from 2016 to 2017 EPM. But with the IIS application pools with such low concurrent counts before, its

                        also difficult to separate if the counts belonged to one issue vs another.

                         

                        Thanks for the added suggestions and advice.

                         

                        Rick

                        • 9. Re: Inventory scan failed :ldiscn32: the inventory server "XXX" did not respond
                          Peter Massa Expert

                          Hey Rick,

                           

                          We are on 2017.3 SU3 (for a few months), and in the last week we suddenly have started getting a ton of Inventory Server not responding errors like you stated.  Causing provisioning to stop working, etc.

                           

                          Did you end up finding a solution?  I reached out to our TRM but have not heard anything back as to a fix.

                           

                          Peter

                          • 10. Re: Inventory scan failed :ldiscn32: the inventory server "XXX" did not respond
                            Peter Massa Expert

                            Hey Rick,

                             

                            Just wanted to follow up on this.

                             

                            I called into support and we determined that the root cause was that the app pool for Inventory was being over loaded past its queue size, which was causing requests to be dropped.

                             

                            The cause of this overload was due to the Agent State feature.  In a normal deployment of agent state or other csep features - a single device in a subnet gets elected and communicates to the core on a frequent basis.  In our environment for whatever reason every single device in every single subnet was electing itself, unable to tell that another device was already elected.  We had a constant flow of ASA traffic coming in non-stop from basically every node.   We were seeing in the c:\programdata\landesk\log\SelfElectController.log:

                               CSEP_Enabled = true  XDD_ARP_PromoteDesktops = true  Services Enabled:  ASA_SVC

                             

                            Even though in the core - it showed another device for my subnet as the elected device.  Looking at other devices near by me in the same subnet, they also were all showing elected.

                             

                            We did not determine the cause of Agent State not working properly - but plan to - it likely is due to us blocking some multicast features on our LAN.  For now we disabled the Agent State feature in the agent client connection setting and have stopped seeing 503 errors completely for the last 10 hours.  We have also seen the time for inventory scan responses decrease to about 10-20 seconds instead of 2-3 minutes while waiting on the core server to provide inventory instructions.  Additionally we have seen our IIS log sizes dramatically decrease - they had been hitting about 1GB in size per day, for the 10 hours since we disabled ASA - it only reached almost 100MB.

                             

                            Hope this helps - may or may not be the same issue for you,

                            Peter

                            1 of 1 people found this helpful
                            • 11. Re: Inventory scan failed :ldiscn32: the inventory server "XXX" did not respond
                              phoffmann SupportEmployee

                              Hmm - interesting. And yeah - "spamming the Core" (as an unintended side-effect of blocking agent <=> agent comms) will cause issues. Good investigation work there.

                               

                              However, *do* be careful / aware that you may be shooting yourself into your own foot in other respects too by disabling multicast-related things.

                               

                              Things like "peer download" for instance *HEAVILY* rely on TMC working. We use TMC a lot overall for subnet-directed communication (since it's very light on the network).

                               

                              You may want to use this article - How to use PEDownloader.exe to Duplicate / Troubleshoot Software Distribution - to specifically test things like peer download & such in your locked-down networks to make sure whether you do / do not have hobbled those components in a simple, command-line based fashion.