7 Replies Latest reply on Sep 26, 2008 2:58 AM by phoffmann

    Core Server Lockup

    Apprentice

      Hi:

       

       

       

      Our server locked up this morning, We could bypass the logons screen but the desktop does not appear.  After awhile we hard booted the Server, everything seems to be working again.  I'm checking the Event ID so see if what possibly caused the problem.

       

       

       

      Under Application, I see these messages:

       

       

       

      Warning: 9/15/2008 4:16AM W3SVC-WP
      ISAPI 'C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll' reported itself as
      unhealthy for the following reason: 'Deadlock detected'.

      Error: 9/15/2008 5:45AM LANDesk Inventory Server
      The description for Event ID ( 0 ) in Source ( LANDesk Inventory Server ) cannot be found.
      The local computer may not have the necessary registry information or message DLL files to
      display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to
      retrieve this description; see Help and Support for details. The following information is
      part of the event: NULL recordset: TCP.

      Error: 9/15/2008 5:45AM
      Database exception: SCAC105.SCN, LDInv.dll
      -2147467261
      Invalid pointer
      Update Table:Computer Column:SWNUMFILES PK:, Thread ID: 7548.

       

       

       

      Error: 9/15/2008 5:52AM
      The description for Event ID ( 0 ) in Source ( LANDesk Inventory Server ) cannot be found.
      The local computer may not have the necessary registry information or message DLL files to
      display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to
      retrieve this description; see Help and Support for details. The following information is
      part of the event: Invalid pointer.

      Error: 9/15/2008 7:56AM
      The script started from the URL '/incomingdata/postcgi.exe' with parameters '' has not
      responded within the configured timeout period.  The HTTP server is terminating the script.
      For more information, see Help and Support Center at
      http://go.microsoft.com/fwlink/events.asp.

      Warning: 9/15/2008 9:55AM
      The COM+ Event System failed to create an instance of the subscriber
      {58FC39EB-9DBD-4EA7-B7B4-9404CC6ACFAB}.  StandardCreateInstance returned HRESULT 8000401A.
      For more information, see Help and Support Center at
      http://go.microsoft.com/fwlink/events.asp.

       

       

       

      At about 9:56 AM until 9:57AM there were excessive Warning messages (over 300) from the LANDesk Inventory server with the Description "Failed to start a TCPIP Thread."

       

       

       

      Has any one encountered this issue, and know what caused it?

       

       

       

      Thank you.

        • 1. Re: Core Server Lockup
          phoffmann SupportEmployee

           

          Warning: 9/15/2008 4:16AM W3SVC-WP

          ISAPI 'C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll' reported itself as

          unhealthy for the following reason: 'Deadlock detected'.

           

           

          ==> Not run across this one, but it seems like a .NET problem. Maybe your Core's IIS is overstressed?

           

          I would recommend looking at the (mainly) SQL performance whitepaper, as it also mentions IIS and gives performance counters for that: You can find it here:

          http://community.landesk.com/support/docs/DOC-2356

           

          =====================

           

          Regarding the Inventory errors, first I'd make sure you've got the latest Inventory Service.

           

          For 8.8 that's included in SP2 - for 8.7 you need to be on SP5 an can request it from support.

           

          The "failed to start a TCP thread" error is often caused by performance problems (again, a reason to look at the white paper above I linked to carefully). Quite possible your Core is just dying under the load you put it under - I can't say without knowing more of your environment (# of nodes, hardware, blah blah blah) - but most of this info you can find in the white paper.

           

          =====================

           

          POSTCGI throwing an error is again going to be WebConsole/IIS related.

           

          COM+ stuff MIGHT be along the same lines ... and should be. COM+ can be a nightmare to fix (I don't think even Microsoft have a real process for that) - so I'm hoping at the moment that it's more symptomatic than anything else :).

           

          Hope this helps nudge you in the right direction a bit :).

           

          Paul Hoffmann

          LANDesk EMEA Technical Lead

          • 2. Re: Core Server Lockup
            Apprentice

            Hi Paul:

             

            Thank you for the response.  Another thread also indicated an issue with .net and IIS.  In my environment I have no access to the SQL Server (for performance tuning), I can only make suggestions.  I'm waiting for SP2 (the 24th), hopefully the update will go without any problems.

             

            My Core Server is:

            2x Quadcore Xeon @ 2.5GHZ

            8GB of RAM

            Windows 2003 R2 Enterprise with SP2

             

            LDMS 8.8 (not SP1)

             

            We have about 4800 nodes.  Following white paper, the server shouldn't be over stressed.

            • 3. Re: Core Server Lockup
              phoffmann SupportEmployee

              The numbers themselves "should be" OK - but there's a lot of things that can change on account of configuration / what you actually do with the Core - so run the numbers / stat-counters and see what you come up with. That'll give you a more realistic image of what load is actually going on.

               

              The numbers in the paper are a goalmarker - there's plenty of ways of killing IIS with fewer nodes (we've run into one incident recently where vulscans were being sent every hour and the Core-server was shy of thermal meltdown because of the load caused) quite easily - so closer inspection is always needed.

               

              Looking at the IIS log, have a look what gets mentioned the most ... how big is your IIS log / day, and so on. Then there's fact of actually (re-)configuring IIS to use multiple application pools and so on and so on and so on - which should be done with the LANDesk partner ideally.

               

              With some creative (mis-)configuring, I'm pretty sure I could get 1,000 nodes to bring a Core to its knees that normally serves over ten thousand - there's lots of ways to do these things - it all involves finding out what causes the problem, and once you the "what", you can find out the "why" - i.e. "did we accidentally configure our clients in such a way that they send a vulscan every minute" (for example).

               

              Paul Hoffmann

              LANDesk EMEA Technical Lead

              • 4. Re: Core Server Lockup
                Apprentice

                I will run the number check and also research the IIS issues...

                 

                Thank you.

                • 5. Re: Core Server Lockup
                  Tom Farrugia SupportEmployee

                  As for the invalid pointer error, this article helped me resolve this with a client recently.

                   

                  Error: "Event ID 0: Invalid Pointer Error in Application Log on the Core Server"

                   

                  Hope that helps.

                  • 6. Re: Core Server Lockup
                    Rookie

                    Hello,

                     

                    Keos any updates on your research?

                     

                    We have the same problems in our environment!

                     

                    Especially the "Failed to start TCPIP Thread" Error Message, will occur very often since beginning of september on our server!

                     

                    But we do not run 8.8, we run 8.7 SP2.

                     

                    The Inventory Server will stop and then there are plenty of this messages in the eventlog.

                     

                    Regards Peter

                    • 7. Re: Core Server Lockup
                      phoffmann SupportEmployee

                      Paider - I would recommend you run the same checks that I've recommended to Keos -- his checks will only be valid for HIS/HER environment.

                       

                      His results will not likely have much meaning towards yours.

                       

                      Paul Hoffmann

                      LANDesk EMEA Technical Lead