Hi,
Microsoft NLB multicast is fully supported in vmware and you don't need to take any special steps to configure your ESX host.
So 2 options :
-1 The issue is located on your Guest os. Check windows log.
-2 Maybe ARP is rejected by the switch.
You can check here : VMware KB: Microsoft Network Load Balancing Multicast and Unicast operation modes
Regards,
Julien.