10 Replies Latest reply on Apr 2, 2009 7:22 AM by defuseme2k

    Antivirus Updates Killing Bandwidth

    scots Apprentice

      We're in the middle of deploying LANDesk Antivirus.  From what I've read, definitions are attempted to be downloaded from the following sources in this order:

       

      - Peer

      - Preferred Server

      - Core

      - LANDesk

      - Kaspersky

       

      According to our network guys, the AV definition updates are killing the bandwidth.  They're telling me that in one of our larger locations about half the PC's are downloading from the Preferred Server (an Active Directory server at that location), and the other half are downloading from the Core (which is here in my building).  They're telling me that even the local traffic going to the Preferred Server is killing the network.

       

      My current settings for definition updates are:

       

      - Every day

      - Between 3:00pm to 8:00am

      - Start at 3:30pm

      - Random delay up to 1 hour

       

      Most of the people work 8:00am to 5:00pm.  Delaying until 5:00pm or later would be ideal since there wouldn't be enough people around to complain about network slowness.  The problem is that a big chunk of them power off their PC's when they leave for the day (even if you ask them to leave them on), so delaying to 5:00pm or later will mean that a bunch of them will never get their AV updated.  We never had any network problems like this with Symantec AV when the clients would download definitions, and all 7200+ of our PC's would come back here to my building for the updates.  Why are the LDAV updates hogging up so much more bandwidth than the Symantec updates?  Is there anything that we can do about this?  It's really starting to cause major problems with other network applications timing out and whatnot.

        • 1. Re: Antivirus Updates Killing Bandwidth
          Employee

              AV-1265688.2     Essentially what this update does is allows you to download just the changes and not the enire bases.cab file.  It also gives you the ability to turn off some of the options as to where you can download the file from.

           

          http://community.landesk.com/downloads/patch/AV-1265688.2.zip

           

          It adds this UI so you can configure changes in the Agent Configuration.

           

          Picture1.png

          • 2. Re: Antivirus Updates Killing Bandwidth
            Rookie

            Hello Ty,

             

            can I use this UI on our Landesk Server 8.7 SP5 ?  Or only on 8.8 Server?

             

            Regards

            hjschwe

            • 3. Re: Antivirus Updates Killing Bandwidth
              phoffmann SupportEmployee

              The patch is only available for LANDesk Management Suite 8.8 -- 8.7 is not really viable, as there's a lot of "behind the scenes" stuff that isn't immediately visible that would need to change too, which would in essence migrate things into 8.8 so ... this is 8.8 only, in short.

               

              Paul Hoffmann

              LANDesk EMEA Technical Lead

              • 4. Re: Antivirus Updates Killing Bandwidth
                scots Apprentice

                OK...new question.  I've received this patch and installed it on my core and the console on my PC.  Do I need to push anything out to the clients for this to work?  The readme only mentioned installing on the core and additional consoles.

                • 5. Re: Antivirus Updates Killing Bandwidth
                  phoffmann SupportEmployee

                  Yes, they will need an update (a new AVSERVICE is part of the patch).

                   

                  Vulscan will take care of itself (since there's a vulscan.dll) but the rest would be better to upgrade.

                   

                  The update is necessary, as once you've made these changes in the settings on your Core, the client-behaviour XML's will be changed (they get additional parameters for those new options) ... which will mean nothing to an AV-client that's not updated.

                   

                  Apologies for the oversight in the readme - I'll see if I can get that fixed.

                   

                  Paul Hoffmann

                  LANDesk EMEA Technical lead

                  1 of 1 people found this helpful
                  • 6. Re: Antivirus Updates Killing Bandwidth
                    scots Apprentice

                    So...I need to install a patch on the clients too?  Or just do a change settings?

                     

                    Also...what's a good number for the "Download updates as a single file if changed file count is greater than" setting?

                    • 7. Re: Antivirus Updates Killing Bandwidth
                      Rookie

                      Without coming across the wrong way; I'm glad someone else is having problems with this too.  I've been working with the LANDesk AV PSE on this particular issue for some time now.  I'm in the process of updating my clients to this patch.  The problem with the patch is that when I install the new version of the Antivirus client on my MDR for instance, it still pulls down the bases.cab from my core server.  Interestingly enough, a client install afterward does NOT pull from there, it still goes to my core.  Multicasting works great for me on everything else I do, so this is an AV specific problem.  I don't have any preferred servers set up yet, and it really doesn't make any sense to do so since most of my WAN circuits are directly coming back here anyway, so there isn't much point.  I've set the number of files to be 999 behind before downloading hte bases.cab file, this should prevent updated clients from pulling down the entire thing -- ever.  I haven't gotten that far yet.  Part of the installation process of the new AV Client pulls down the bases.cab regardless of your new settings.  The av behavior settings (you know the 999 files out of date setting) are not used until after the install is complete, so I've got so much hell to go through before I catch back up.

                       

                      I'd like the guy who started this thread to please check the sdmcache on some of the clients which are pulling from your preferred server and core server.  See if you see an avp.klb.bad, or any other files with the .bad extention.  Perhaps the bases.cab.bad will be there.  From what I've seen, the clients are not able to successfully multicast the avp.klb file from other peers and it receives a bad copy and it does NOT update.  Intermittently it will acheive a successful download of the avp.klb file, and when it does it is typically over 15 files behind, which causes the client to get the MOST up to date bases.cab at the time.  Depending on the frequency of the core server going out to download updates from the internet -- you can have a problem with the client not finishing the update in time, causing a viscious cycle that can go on for hours.  My circuits aren't very big, usually half T1.  So 30 clients pulling the bases.cab from my core KILLS my circuits.  I've got other fish to fry now, but there is obviously a problem with the peer multicasting, even on the newer patched client.  You may reduce the problem by not allowing the bases.cab downloads (individual file downloads only), but you're not addressing the avp.klb.bad issue which causes clients to only be sparadically up to date.

                       

                      The install process needs to be changed so that it follows the settings specified in the config at deployment time.  It doesn't, it just pulls that stupid bases.cab down regardless unless the client is already up to date, which obviously isn't going to be the case.  I like the LANDesk support guys, I don't want to chide them too hard, but this Antivirus product feels like a 1.0.

                      • 8. Re: Antivirus Updates Killing Bandwidth
                        Rookie

                        Does this patch need to be applied to a Core Server running LD8.8 SP2A?

                         

                        We are experiencing the same issue with AV killing the bandwidth - but don't ever seem to get on top of it.

                        • 9. Re: Antivirus Updates Killing Bandwidth
                          phoffmann SupportEmployee

                          I hope the following will clarify a few things:

                           

                          * The Patch contains updates for both Core and Clients.

                          * Therefor, it's necessary to update BOTH (so you can plan ahead for this).

                          * The patch is a POST SP2/SP2A patch, and not included in those service packs. It will be included in SP3.

                           

                          At this point, it may be potentially worth while waiting for SP3 to come out and use that, since you'll be needing to run a client-side update as well. If the problem is a big issue for you, LANDesk support will be happy to provide you with the patch (open a support ticket requesting it), no problem.

                           

                          I hope that this clarifies some of the apparent confusion lingering.

                           

                          Paul Hoffmann

                          LANDesk EMEA Technical Lead.

                          • 10. Re: Antivirus Updates Killing Bandwidth
                            Rookie

                            The problem in my environment turned out to be an issue with IIS serving up an old version of some of the files after an update, and it would not stop doing this until an iisreset was performed.  You could actually go into IIS and recycle the worker thread for the app pool, which worked too, but an iisreset was just as fast.  My core updates every hour, so this fix was NOT anything more than a bandaid and very manual.

                             

                            I worked for a long while with David Holland (LANDesk AV PSE) on this issue (weeks), it was escalated and finally resolved to at least a non-manual process.  I suggest trying some of the tests below with your core server, since the issue turned out to be an issue with IIS in particular, not really LANDesk.

                             

                            First, go ahead and iisreset at the cmd line (google is your friend if you haven't done this before), then force an update of your virus definitions (they do NEED to update to a new rev for this test to provide any valueable data).  Once it completes, navigate to the UNC path \\corename\ldlogon\antivirus\bases, and grab a copy of avp.klb, and name it avp1.klb.  Then visit the same path via http://corename/ldlogon/antivirus/bases/ and download avp.klb to the same folder.  Then at the command line, navigate to the folder and run this:  fc /b avp1.klb avp.klb.  If they DO NOT match (you should see a message that says there are no differences in the files), then what is actually happening at the client level is really disturbing.  Hashes aren't matching which leads to the client not properly updating and the files start to get behind.  The 15 file limit problem, or potentially higher if you have the patch mentioned in this thread and don't have it cranked up enough can become an issue.  Once in a while, the worker thread on the app pool recycles on its own (something like 17400 seconds by default), and the problem is fixed for a short spurt, and your bandwidth can be saturated with clients all pulling the bases.cab from the core server.

                             

                            You can try this registry key if you can reproduce the issue above with the avp.klb not matching because IIS is serving older copies of the files.  Navigate in the registry to HKLM\SYSTEM\CurrentControlSet\Services\InetInfo\Parameters\ and add the dword value DisableMemoryCache, and set it to a decimal value of 1.  You will need to restart your core server (easier than catching all the service restarts) to make this take effect.  You then can force another definitions update, and again it will need to actually update the revisions for the test to give you real results.  You can then do the above test again with fc from a client.  If this does in fact cause the files to match consistently after every update, then it might be worth leaving the registry key in place, just don't forget that you did it.  What this does is completely remove any caching in memory at all for IIS, causing all files to be served from disk.  This so far hasn't made a huge difference for me, but it could for someone else depending on the environment.  I'm just happy my client's av patterns are up to date and my bandwidth saturation issues are gone.

                             

                            My understanding, since this was at least partially reproduced at LANDesk support is that something will be coming out sometime in the future to address the problem.  The suspicion is that this is happening more often than is realized.  I wanted to post it here just in case anyone else is having the same problem which I suspect is the case.