Hi...as you know, we cannot perfrom direct upgrade from esx 3.5 to esxi 5.1 and before that you have to do upgrade of esx 3.5 to esx/esxi 4.x.
Also upgrade requires reboot of your host. So if you do not want any downtime for your VM's then you must migrate the VM's to other host.
And the host upgrade will not impact the existing VMFS3 datastore. So you can perfrom online upgrade of your datastore to VMFS5 after the host upgrade even while VM's are running on that.
Host Upgrade - https://www.vmware.com/pdf/vsphere4/r40/vsp_40_upgrade_guide.pdf
Datastore Upgrade - http://www.vmware.com/files/pdf/techpaper/VMFS-5_Upgrade_Considerations.pdf