1 2 Previous Next 17 Replies Latest reply on Sep 29, 2016 10:32 AM by brett.chadwick

    Device Control logging & database growth

    pierrejvr Rookie

      I'm finding that the Device Control Global Policy default setting of logging all events is causing my database to grow very rapidly when assigning it to high traffic servers.

      How do I limit logging of events to only events related to media and externally connected devices, i.e. USB, Bluetooth, CD/DVD? I can't see an effective way of doing it with the Global Policy's log exclusion list.

        • 1. Re: Device Control logging & database growth
          brett.chadwick Apprentice

          The log events are normally very small and won't increase storage requirements in the database by any significant amount. However if you are seeing a increase or have a very high device traffic which is normal for your company you may choose to exclude certain event types that make up most of the events. As an example you may choose to exclude events generated by Local System as these are not critical for audits and would be normally system generated events that could be trusted.

           

          The other options around this would be filtering events based on user, device model, machines or collections.

          Filtering of log events will occur before hitting the server and reduce the amount of logging that is occurring.

           

          Here are some of the options in the screenshot.Screen Shot 2016-03-24 at 8.35.14 AM.png

          1 of 1 people found this helpful
          • 2. Re: Device Control logging & database growth
            brett.chadwick Apprentice

            As a follow up on this it might be helpful to understand what exactly these events look like.

            -------- 2016.03.24 16:55:03 --------
            Intel(R) 82574L Gigabit Network Connection #2, Intel(R) 82574L Gigabit Network Connection, Intel
            PCI\VEN_8086&DEV_10D3&SUBSYS_07D015AD&REV_00
            21
            1
            256
            34
            256
            23
            \Device\NDMP5
            \Driver\e1qexpress
            {4D36E972-E325-11CE-BFC1-08002BE10318}
            
            

             

            This is an event from a device connection, this was extracted from the WLD.log found inside C:\Windows\sxdata\ on a endpoint.

            This is an example of an event from a endpoint. This is the logged data that the global policy, and above exclusion is referring to so referencing these events can help you consider the right filter criteria.

             

            The SDC Event files are found inside C:\Windows\sxdata\Shadow\

            This directory has security settings implemented and could be protected if you have agent hardening enabled so be sure to disable agent hardening on your test machine before attempting to access the above Shadow directory. This directory contains shadow events, these can be a logged event, from a device connection, and logged data like what process tried to access what, what access level was granted to an endpoint and time/date stamp of the event.

             

            {
            
            
            {EventLog}{ServerLog}{SysLog}2016-03-24 16:51:54.0270000 -0700|WRITE-DENIED||S-1-5-18||\Device\HarddiskVolume2|||C:\Windows\SxData\Shadow\sdcevent-97C4E2832786D101.log.final|lmhost.exe||||||
            
            
            }{ServerLog}{SysLog}2016-03-24 16:51:54.0270000 -0700|WRITE-DENIED||S-1-5-18||\Device\HarddiskVolume2|||C:\Windows\SxData\Shadow\sdcevent-97C4E2832786D101.log.final|lmhost.exe||||||
            
            
            

             

            Shadow events are another event type that could be generating additional data and could be filling the database. Shadow events come in two forms the first is a file name only, and the second is a full file copy of data which is stored as flat file on the file system of the server. So if shadow events are the cause it would be file name only events and not full file shadowing events.

            2 of 2 people found this helpful
            • 3. Re: Device Control logging & database growth
              mproberts Rookie

              Brett,

               

              We're having a similar problem where the sqlserv.exe process is maxing out the memory on the server.  Looking on the server in the Program Files (x86)\Lumesion\EMSS\Content\BroadbandFileUpload\logmanager\Complete folder I have lots of folders with log files with an .UN or a .CL extension.  Opening one of .UN files, I see lots of entries like your second screen shot.

               

              I don't have shadowing turned on, so I'm a little confused to why these logs are being generated?

               

              We've only recently started using the device control module, so it's very possible that I configured something wrong.

               

              Thanks in advance!

               

              v/r

               

              Mike

              • 4. Re: Device Control logging & database growth
                brett.chadwick Apprentice

                When you say maxing out the memory how bad is the utilization?

                The normal behavior for SQL Server is to use as much memory as possible to load the database into memory.

                 

                So to combat this behavior you can specify allotted memory for a SQL Server instance. If you are finding memory use to 95-100% of available memory on the system it could be that background processing of log data is causing a build up of files on the file system waiting to be processed.

                 

                The directory [Program Files (x86)\Lumesion\EMSS\Content\BroadbandFileUpload\logmanager\Complete ] you mention where you see files is for obviously completed files which I would not expect to see a significant amount of data being stored. However this of course depends on the settings, number of endpoints with device control and the policies defined for those machines.

                 

                How many folders/files are you finding there?

                 

                Here is an example from my test server which to be fair has only a handful (8) registered agents and only a few with device control installed.

                A review/Health Check of your server might be a good idea. In what country are you based? If the US what state are you based in?

                I can see if I can get someone from either our SE team or support to reach out to you and check over the server.

                 

                1 of 1 people found this helpful
                • 5. Re: Device Control logging & database growth
                  mproberts Rookie

                  Brett,

                   

                  The Physical memory utilization on the server is 93%.  This server has 32GB RAM and sqlserv.exe is using 28GB.

                   

                  Both the InProgress and Complete folders have 200 folders in them.

                   

                  I'm in the US and I already have an open ticket on this issue (30211045).  We're running in an airgapped environment, so it very difficult to provide log files.

                   

                  We're running EMSS on multiple stand-alone networks and so far we've seen the issue pop up on 2 of them.

                  • 6. Re: Device Control logging & database growth
                    brett.chadwick Apprentice

                    Understood, no problem.

                    93% would seem to be a bit high, and with that amount of memory it should handle the load.

                    I have reviewed the case, it does sound like some log files may be getting backed up processing.

                    The number of endpoints and how the policies are assigned can add to this.

                    In some cases it could be duplicate policies that are generating the additional workload.

                     

                    Are the policies assigned at the AD level or using Custom Groups/System Groups? Or just to specific machines?

                    How many endpoints in total have the DC module installed?

                    Is the SQL server remote or installed on the same server as Heat EMSS / LEMSS?

                    The EDS process on the EMSS server is doing all log processing. How much memory/CPU is this process using?

                     

                    Also I will move this data into the case too so we all have the same information to get this resolved ASAP.

                    • 7. Re: Device Control logging & database growth
                      mproberts Rookie

                      Brett,

                       

                      The only policy that I created are assigned to some of the system groups that I built in EMSS.

                      I have 287 endpoints with the DC module installed.

                      The SQL server is on the same server as EMSS

                      The LM.EDS.exe process is using is using 0% of the CPU and 1.158 GB of Physical Memory.

                       

                      The server has 32GB of storage and dual Xeon E5-2470v2 processors @ 2.40GHz.

                       

                      Thanks for all of the help!

                       

                      Mike

                      • 8. Re: Device Control logging & database growth
                        brett.chadwick Apprentice

                        Okay, does not sound like it is the issue I was thinking of.

                        Strange, support can provide some advice on the EDS configuration changes required to increase the log processing speed.

                        If you don't get to a satisfactory solution let me know and we can get into some more detailed troubleshooting.

                        A little harder as you can imagine without being able to look at the server, but just takes a little more time to get to the root cause.

                        • 9. Re: Device Control logging & database growth
                          reneg Apprentice

                          Hello Mike,

                          The symptoms that you are describing sounds a lot like a known issue with EMSS 8.2.  Can you log into the EMSS Console > Manage > Agent Policy Set > Global Policy > is Agent Hardening activated?

                           

                          There was a known bug with the DC Module + Agent Hardening activated that would fill up the database with useless logging events and fill up [Program Files (x86)\Lumesion\EMSS\Content\BroadbandFileUpload\logmanager\ folder.

                           

                          Our technical support team should be fully aware of this problem and provide you with a manual solution to fix this issue if you cannot upgrade to version 8.3.

                          • 10. Re: Device Control logging & database growth
                            mproberts Rookie

                            Rene,

                             

                            We're running version 8.3, but it was an upgrade from 8.2.

                             

                            I had turned off the agent hardening (Agent Uninstall Protection) earlier when we first started having the issue as part of my troubleshooting.

                             

                            As a workaround, we've limited the amount of memory that SQL can use down to 16GB (The server has 32GB).  That *seems* to help with the sluggishness of the console, but we're still getting lots of log files generated.  The LM.EDS.exe process is using around 1.7GB of memory to process the logs.

                             

                            I've been in contact with technical support, but due to our security constraints, it's very hard to almost impossible to provide log files that are collected using the log collector utility.  I don't mind being the eyes of the tech if they tell me what to look at in these logs.

                             

                            v/r

                             

                            Mike

                            • 11. Re: Device Control logging & database growth
                              reneg Apprentice

                              Mike,

                              Turning off Agent Hardening should mitigate future problems but now let's take care of the current problem.  Try this: 

                               

                              Warning: You will lose some log data!

                               

                              1.  Run services.msc > locate the service name EDS Server > stop this service.

                              2.  Go to <installpath>\EMSS\Content\BroadbandFileUpload\logmanager > delete the subfolders.

                              3. Restart the Windows Server.

                              4. Monitor the performance and share the feedback.

                               

                              Also, how good are your SQL skills?  I can provide you with a SQL Script to purge the database of useless log files if your database size (UPCCOMMON DATABASE) is getting too big.

                               

                              --Rene A Gonzalez

                              • 12. Re: Device Control logging & database growth
                                pierrejvr Rookie

                                Rene,

                                 

                                I would like to get a copy of the SQL script to purge old/useless log files, as I'm still getting a lot of database growth due to DC.

                                 

                                Thanks,

                                Pierre

                                • 13. Re: Device Control logging & database growth
                                  reneg Apprentice

                                  Pierre, I just remembered that if you upgrade the EMSS Server to v-8.3, there is a feature called "Database Maintenance" (Tools > Database Maintenance). Create a new Maintenance Plan to purge out old data.   If you are not running v8.3, I'll send you the SQL Scripts directly.

                                  • 14. Re: Device Control logging & database growth
                                    mproberts Rookie

                                    Rene,

                                     

                                    I followed your instructions yesterday and when I checked this AM, all of the endpoints were off-line, so I rebooted the server.  After the reboot the endpoints came back on line.

                                     

                                    The sqlsevr.exe process is currently using around 956 MB of RAM and the LM.EDS.exe process is using around 1.75 GB of RAM.

                                     

                                    The \Content\BroadbandFileUpload\logmanager\Complete and InProgress folders are continuing to pop in.  Right now there are 59 folders in each folder, but they're all empty, so that's a change.

                                     

                                    The UPCOMMON database is currently 260 GB...  Not sure if that to big or not.

                                    1 2 Previous Next