LDMS is using the sdclient to download your drivers. How come you don't have the drivers out on your preferred server? wouldn't you want to download the drivers from there instead of always coming back to your core.
This network is pretty much a 100mb+ LAN - largely confined to a single site. To be honest, I'm only using Preferred servers to get the benefit of having Software distribution and patching being authenticated to the share rather than having to use sysshares or allow anonymous access (or even domain computers for that matter)
I've done what you suggested and put the files on the preferred server and while it has got rid of the 20 second wait, there still seems to be a massive overhead when SDClient checks the preferred server each time and then copies the files one by one. Copydrivers.exe from Jan Buelens has been used with the 8.8 Core this one has replaced and CopyDrivers still takes a fraction of the time.
I still use copydrivers.exe for my HII needs even in 9.0. I still find it to be the superier method.