Here's some more info, in response to a private message. Might as well post this in the forum for others...
The GPU I'm using is a PowerColor Radeon HD 7750 Eyefinity4 with 2 GB and 4 Mini-DisplayPort outputs. Since I'm running it in a Supermicro SC847 chassis, I needed a low-profile card, and since the server is sitting in the basement (noise!) and the video cable runs through a hole in the floor, I wanted Mini-DP because it’s the smallest connector. This was the only card I could find that fit all the requirements. It ships with adapters from Mini-DP to DP and Mini-DP to Single-Link DVI (i. e., max. 1080p).
The card magically started working in one of the later betas of 10.9.3, see my earlier posts, and runs OOB with no kext hacks.
Aside from OS X, I'm running SmartOS (a Solaris derivative) as a guest. It didn't seem to be too happy with the E1000 emulation provided by ESXi, so I currently have a supported Intel 1GbE card in passthru to SmartOS. There are also two LSI 9211-8i-IT controllers in passthru.
Something has not been quite right, unfortunately, as the SmartOS VM has been stalling out every few weeks. I haven't been able to debug it, but my impression is that disk I/O is stalled. If I have shells open to the SmartOS global zone or any its zones, it still takes text input, but nothing else happens. Scrubs of the ZFS file system never show any errors on reboot, but my Time Machine backups via netatalk tend to get damaged if they are running when a stall happens.
I can reboot the SmartOS VM via ESXi, but then the problem appears to recur fairly quickly. Rebooting the entire ESXi host clears the problem, and things will be fine for weeks afterwards.
It feels like a poor interaction between ESXi, PCI passthru and SmartOS. The OS X VM is not affected by the stalls.
I have since tried two different USB 3.0 cards via passthru for OS X: the Orico Fresco Logic FL1100 that I thought would work OOB, and a VIA that I was trying to use via the GenericUSBXHCI kext. Neither worked (KP with the FL, dead USB ports on the VIA), but I didn't pull the VIA until yesterday. Part of the problem may have been that I did not connect auxiliary power to the USB cards, though I did only want to connect a self-powered hub.
Perhaps the USB cards were causing issues for ESXi.
I've also been thinking about making use of 10GbE, since there is no VMXNET3 driver for VMware OS X guests or SmartOS (though it might be possible to get the open-source Solaris driver to work), and the VMware E1000 emulation is limited to 1 Gbit/s throughput. After thinking about it, I hope to do something like this:
- one Intel X520-DP1 card in passthru to SmartOS
- one Intel X520-DP1 card configured in ESXi
- crossover connection between the two with an SFP+ direct attach cable
- motherboard Intel GbE uplink to the rest of the LAN
This would allow me to create iSCSI volumes on the SmartOS ZFS and provide them to ESXi as a VM backing store for my OS X volumes, giving high performance local storage to my Mac VM with the ability to do ZFS snapshots and backups as protection against HFS+ corruption.
Alternatively, I would put a Small Tree or ATTO 10GbE card in passthru to OS X and store my Mac data on a fast netatalk AFP share on SmartOS, but the cost for those 10GbE cards is really just a bit too much at this stage. Small Tree and ATTO sell re-branded Intel X520 and X540 cards with custom firmware that enables their Mac drivers, so you're mostly paying for the drivers. Things should change once Apple ships 10GbE, which I would expect to see sometime around Intel's next 10GbE chip revision.
None of this makes sense until I can cure the SmartOS hangs. Onwards!