I use Nvidia GT 740 (Kepler GPU) which works OOB (disabled grapic injection in chameleon, not needed) . Before that i had an GT 640 - also OOB, 740 is about 20% faster.
Works with OS X drivers and! NvidiaWeb.
I have also old 775 GA Mainbord with Quad 8400 cpu.
I would avoid Fermi (working but some freezes, more power consumtion) and Maxwell GPU (you need NvidiaWeb driver - problem of updates and not full compatible - we wait for Apple Maxwell/pascal drivers)
So look for an Kepler GPU type - depends of speed and price you want to pay. Used ones are also available
Highend Kepler GPU type would be not a good deal, because you would need then also much faster CPU to get full speed advantage.
GT 640 (Kepler) / GT 740.
GTX 750 Ti maybe also good , faster than GT 740 and not oversized for the low end CPU.
Here List of GPu type 600 , 700 - because its really difficult to guess which card type has which gpu type (Fermi/Kelper).