How to troubleshoot kernel memory leaks

Version 11

    Issue


    System runs out of resources over time and becomes unresponsive.

     

    The System Event Log shows one or more of the following error:

    Event ID: 2019
    Source: Srv
    Description: The server was unable to allocate from the system nonpaged pool because the pool was empty.

    Event Type: Error

    Event Source: Srv

    Event ID: 2020

    Description: The server was unable to allocate from the system paged pool because the pool was empty.

     

    Cause

    A driver installed on the system has a memory leak and is consuming all available resources.


    Resolution

     

    To determine the specific driver causing this memory leak the following steps can be taken:

    1. Download the Windows Driver Kit .ISO
      (http://www.microsoft.com/downloads/en/details.aspx?displaylang=en&FamilyID=36a2630f-5d56-43b5-b996-7633f2ec14ff)
    2. Extract or mount the Windows Driver Kit .ISO and run “KitSetup.exe” from the root.
    3. Uncheck “Debugging Tools for Windows” and check “Tools” and click OK.
    4. Select a directory to install to and click OK.
    5. From a CMD prompt, navigate to [Directory Installed To]\Tools\Other and then to the appropriate directory for the Operating System you are trying to troubleshoot (amd64 for x64 systems, i386 for 32-bit systems) and run “poolmon /p".
    6. Press "D" to sort by "Diff".   This will display at the top of the list the process with highest differential between handles allocated, and handles freed.
    7. Under the Tag column will be a 4 digit “Pool Tag” identifier.   This can be used to find the malfunctioning driver.
    8. To find the malfunctioning driver, go to C:\Windows\System32\drivers and C:\Windows\SysWow64\drivers and run “findstr /m /l [tag] *.sys”  (Where [tag] is the 4 digit tag identifier from the log file in step 7.
    9. The malfunctioning driver (.sys file) should now be displayed.   A Google search should now help you determine who that .SYS file belongs to.

     

    The registry key change is as follows:

     

    HKLM\System\CurrentControlSet\Control\Session Manager\GlobalFlag

     

    Change the value of Global Flags to 1024 in decimal and reboot the computer.

     

    If the driver is allocating pool space rapidly and not giving it back, it will be quickly apparent which driver is the culprit.   However at times it may be necessary to wait a while before it becomes noticable.

     

    Sometimes a driver may only start acting up when certain events occur.   In this instance it is necessary to have poolmon run at regular intervals and log to a file.   The following batch file (also attached) will run poolmon and write to a log file with the current time and date.  (Poolmon.exe will need to be copied to C:\ for this to work properly)

     

    @echo off
    SET POOLMON="C:\Poolmon.exe"
    SET OUTDIR=C:\
    SET YEAR=%DATE:~10,4%
    SET MONTH=%DATE:~4,2%
    SET DAY=%DATE:~7,2%
    SET HOUR=%TIME:~0,2%
    IF /I %HOUR% LEQ 9 SET HOUR=0%HOUR:~1,1%
    SET MINUTE=%TIME:~3,2%
    SET SECOND=%TIME:~6,2%
    SET ISODATE=%YEAR%-%MONTH%-%DAY%_%HOUR%-%MINUTE%-%SECOND%
    %POOLMON% -d -n %OUTDIR%\poolmon.%ISODATE%.log
    This will create a log file in the root of C: called poolmon.currentdateandtime.log

     

    You can then create a Windows Local Scheduler job to run the batch file regularly:

     

    To schedule a task that runs every N minutes

     

    Minute Schedule Syntax

    schtasks /create /tn TaskName /tr TaskRun /sc minute [/mo {1 - 1439}] [/st HH:MM] [/sd StartDate] [/ed EndDate] [{/et HH:MM | /du HHHH:MM} [/k]] [/it] [/ru {[Domain\]User [/rp Password] |System}] [/s Computer [/u [Domain\]User [/p Password]]]

     

    Example to have poolmon run every 30 minutes:

     

    schtasks /create /sc minute /mo 30 /tn "Run Poolmon Every 30 Minutes" /tr C:\runpoolmon.bat /ru system

     

    This command line is included in the attached "AddPoolmonTask.bat".   To run this command as displayed here and add a task to the task scheduler that runs Poolmon every 30 minutes, simply run this batch file.

     

    Further information regarding the schtasks command: Schtasks: Management Services

     

    Note: See the following articles for further information regarding this process:


    How to use Memory Pool Monitor (Poolmon.exe) to troubleshoot kernel mode memory leaks
    http://support.microsoft.com/default.aspx?scid=kb;EN-US;177415


    How to find pool tags that are used by third-party drivers
    http://support.microsoft.com/kb/298102


    Understanding Pool Consumption and Event ID: 2019 or 2020
    Understanding Pool Consumption and Event ID: 2020 or 2019 – Ntdebugging Blog

     

    PoolMon Examples

    PoolMon Examples