I am assuming that you want to install the gateway on a VM that is running on the ESX server? This should not be a problem. What is the version of the ISO that you are running? Also the area where you are getting the failure is not a common one. I seem to remember that you need to have at least 10 Gb available on the disk... Also you should have a look here for the BKM if you have not already...
LANDesk also provide a Management Gateway VMware image which can be downloaded from this link:
That page needs to input some information, include the Management Gateway in the interested things.