IIS Troubleshooting and LANDESK Management Suite: 101

Version 10

    Verified Product Versions

    Endpoint Manager 9.5Endpoint Manager 9.6Endpoint Manager 2016.xEndpoint Manager 2017.x

    Description: IIS is a web server that can contain a library of web technology. This document will not cover all of IIS but instead focus on pieces that LANDESK uses, how they are used, and suggestions on troubleshooting them. LANDESK is very IIS intensive so this can be a critical part of the functionality and performance of a core server.

     

    Basics and Structure:

     

         IIS Application - Most of what you see in the Default Website (and beyond) are applications in IIS. Applications house files, folders, etc.

     

    IIS1.png

     

         Application Pools - These are threads of execution that allow IIS to be segregated and managed more efficiently. It's quite similar to Task Manager for windows. Applications in the website can be added to their own application pool or be included in a single application pool with a group of other applications. One of the benefits of this is that resources on the web server can be adjusted according to how hard the particular component of a website is getting hit.

     

    IIS2.png

     

    IIS Configuration:

     

         Web.config - This is a configuration file used for a particular section of the overall website. You will have many of these throughout IIS on a LANDESK core however their scope is limited to the location in which they are      located. That is...a web.config located in C:\inetpub\wwwroot for example won't have any connection to one located in C:\inetpub\wwwroot\LANDESK\LDMS.

         Location: Web directory in question.

     

    TIP: Renaming a specific web.config and replacing it from a working server is often an effective step. Sometimes though the server name is located in places within the web.config and may need to be changed.

     

        ApplicationHost.config - This is the primary configuration file for all of IIS. The settings in this file apply to everything however individual web.config files can override the settings.

        Location: C:\Windows\System32\inetsrv\config

     

         Restoring a Backup ApplicationHost.config file - This process can be very useful when the problem in IIS is difficult to find and a recent change caused it to stop functioning. However, it can also cause problems. Especially version changes and sometimes Service Packs etc can change the IIS configuration for LANDESK. Rolling back the configuration BEFORE the version or service pack was applied can cause problems.

     

         IIS keeps the last 10 configuration changes stored in the location below. Look at the date/time stamp to determine which one you may want to try.

     

         IIS3.png

     

         Below is the applicationhost.config file in it's production location. This is the copy that IIS is currently using. Rename the production file, copy the desired backup, then run IISRESET at a command prompt. It's recommended to then test to make sure everything works. If problems happen then the process can be reversed to roll back the changes or to try another backup point.

     

         IIS4.png

     

    Analyzing IIS Logs:The logs outlined below are the primary logs for IIS. Extended logging can be used but most of what you need can be found here. The time stamp in the logs is in UTC though so it can be misleading. Each connection to a website, web service, and other items in IIS contains a return code. A return code of anything in the 200 range will indicate a success while everything else resulted in an error or problem. See the link below for a list of IIS Error codes. However, all non-200 return codes don't necessarily indicate something needs to be fixed. See screen shot below. This is when a Remote Control takes place. The service is initially attempted without credentials. The attempt is denied with a 401.2 code...then the attempt is made again and this time returns a 200 when the credentials are included indicating a success. It's followed by 2 more connections to the remote control log that are also successful.

     

    IIS16.png

     

    TIP: Perform the same action on a working lab LDMS core as a broken core. Then compare the entries and return codes to see which ones are problems and which ones are normal functionality.

    Note: IIS sometimes takes an extra couple of minutes to log an action. Refresh and check the logs below repeatedly and give it some time.

     

         Primary Log -C:\Inetpub\logs\LogFiles\W3SVC1 (then date stamped)

         HttpError Log - C:\Windows\System32\LogFiles

     

         IIS Error Codes Link

     

         HTTPError Log Example: "Timer_ConnectionIdle" is what you want and basically indicates no problems. If IIS is overwhelmed other errors will be logged that correspond to the location that's causing the error.

         IIS5.png

     

    Performance Issues: The first thing to know about performance and IIS is that if you tell 25,000 managed devices to contact the core server...then 25,000 devices will contact the core server. In other words...you can overload the core with a scheduled task very easily as a lot of what LANDESK does relies on IIS. If IIS is having performance issues then checking when those issues are occurring and if they happen to be as a result of scheduled tasks or other functions. Another method is to get a good picture of the logs and see which pieces are getting hit a lot. An example is Extended Device Discovery (XDD). XDD is designed to be deployed to 1 client per subnet. If XDD is enabled on every agent then every agent will be contacting IIS on the core to report it's findings and generates A LOT of unneeded traffic.

     

    TIP: If a particular application within an Application Pool is causing problems it can affect all other applications running within the pool. This is especially problematic when it's the LDApplMain pool as that pool holds about 28 applications. A good step is to temporarily create a new application pool and move the problematic application to that pool. The pool can then be isolated and if necessary be shut down without affecting all other functions. A good suggestion is to compare the properties of the new application pool with the original one and make sure they match in configuration before moving the application. The application in question can be moved by clicking on the application and choosing it's Advanced Settings.

     

    Right-click a tab item (in this case Memory) and then select "Command line". This process is slightly different in older versions of Task Manager.

    IIS6.png

     

    IIS Application Pools are identified after the Command Prompt was enabled. This allows you to see what resources each pool is taking up.

    IIS7.png

     

    Web Services: Web services are programs based in the web that perform functions. Sometimes it's as simple as returning C when it's passed A and B in A+B=C. Other times it's much more complicated. Web services are used heavily in Management Suite and they perform functions from authentication, to scheduling tasks in LDMS, etc. A lot of web services can be browsed and tested with controls in the web service but not all of them. A good example is the MBSDK web service. This service performs many functions for LDMS including aiding in the authentication of the console. You can test the web service by browsing to the following path. Screen shots are below for running a test of the service and an example of the output. If these tests don't work then settings for this application can be compared to a working server to help determine the cause.

     

    To test a web service click the desired service and select browse (Remember...not all web services will load with a friendly interactive screen)

    IIS8.png

     

    The blue banner is typical of a web service but some may not look so friendly. All web services will contain different options.

    IIS9.png

    Clicking a link within the web service will provide more details. In this case there is an "Invoke" button. Others will have options for parameters to pass to the service.

    IIS10.png

    Clicking the invoke button produces a return of information in this case.

    IIS11.png

     

    Miscellaneous Troubleshooting Steps: Below are some general steps for troubleshooting IIS.

     

    Browse the default website - This is a great trick/tip. The idea is that...you cannot get to any rooms in the house if you cannot first get in the front door. In other words...if you can't browse the default website and get the standard IIS banner then most likely nothing underneath the Default Website will work. All IIS versions after LANDESK is installed will display the default IIS banner for the particular version. If you cannot browse the default website here are some settings to check:

     

    Right-click the default website and click "Edit Bindings":

    IIS12.png

     

    Highlight a binding (usually 443) and click "Edit"

    IIS13.png

     

    It is important that the settings below stay just like this. Changing any of these will change how the website works and results in breaking LDMS in one fashion or another.

    IIS14.png

     

    Clearing .NET Temporary Cache - Below is the location for the cache in 9.6. Older versions use a 32-bit version of .NET and will be located in a similar location under the 2.0 version of .NET. All of the sub-folders in the location below are temporary. Removing them hurts nothing and they will be rebuilt when needed. Various problems with the SLM Console, Web Console, and many other portions of IIS have been fixed by clearing this cache.

     

    Location: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files

     

    Process:

    1- At a command prompt run "IISReset /STOP"

    2- Delete all the sub-directories in the location listed above.

    3- At a command prompt run "IISReset /START"

     

    Increase the Permissions on an Application Pool - For security reasons most application pools run under the Network Service account. However, sometimes patches, AD Group Policies, or other software may lock down the Network Service account. In doing so the account cannot access folders and files that it needs to do it's job. In these cases you can either find the permission problem manually or change the user running the application pool to an account that has greater permissions. The Local System account usually is enough for most problems but the setting can even be set to a domain administrator. Keep in mind...the configuration was set this way to help prevent hackers from getting access to the server through the website so there is a security give and take with changing these.

     

    IIS15.png

     

    Test the website remotely and with different browsers - How a website performs on the core may be different than how it performs remotely and may show a different set of problems. The same goes for the way the website is displayed. Testing different browsers is always a good idea as well as clearing the browsers cache and forcing it to load a new copy of the data.