Before I ask you guys for help I googled for hours and found nothing.
I got myself new/old board: GA-EP45T-UD3LR and Xeon X5472. I modified 775 socket for 771 and Xeon cpu with sticker.
Under Windows everything works perfectly, I tested CPU with Intel utility and run various stress tests for GPU/CPU and memory. Not a single glitch. System running stable at stock 3Ghz, 1600FSB and DDR3 1600.
I plugged my OSX HDD and it booted but system was behaving weirdly, windows were half opening and sometimes it wasn't displaying anything in finder, some icons froze on desktop, proper screen artifacts. I have an GTX-760 and HD5770 to swap, both used to work natively on my old motherboard.
Right I thought new hardware needs new system. I Installed Niresh Mavericks distro as only thing working was a PC. I had the same problems with displaying when installing, it sometimes become unresponsive on the welcome screen, sometimes freeze on grey screen. I managed to install but glitches are still there. I tested with 2 different graphics card, same thing.
System works but display freeze sometimes or showing stuff with delay. I tried all the boot flags and no difference.
Many thanks for any input. I will post pictures tomorrow.