Jump to content
Welcome to InsanelyMac Forum

Register now to gain access to all of our features. Once registered and logged in, you will be able to contribute to this site by submitting your own content or replying to existing content. You'll be able to customize your profile, receive reputation points as a reward for submitting content, while also communicating with other members via your own private inbox, plus much more! This message will be removed once you have signed in.


*NEAR-NATIVE* Linux Gentoo QEMU+KVM+High Sierra/Win10+AMD RX560 GPU Passthrough w/ Asus Maximus IX Code+Intel i7-7700K

6 posts in this topic

Recommended Posts


"Hardware and Software Hybridization of Guest Operating Systems"


by rev0lt


Experiment's Goals:


(1) High Sierra at near-native speed on Linux QEMU KVM with AMD GPU Passthrough (Success);


(2) Win10 at near-native speed on Linux QEMU KVM with AMD GPU Passthrough (Success);


(3) To achieve (1) and (2) using Nvidia GPU (Not yet successful); and


(4) To achieve (1) and (2) _simultaneously_. (In theory achievable; but would require purchase of a second AMD card).


Rationale ("Why?"):


Can you feel it?


Jokes aside, a bit of a brief background -- this whole trouble started with my Apple Magic Mouse (Series 1). I really love this mouse -- it has been with me almost 8 full years now, and yes, it still looks beautiful; and I really wanted this wonderful mouse to work in an acceptable manner in Gentoo Linux. I managed to get it to work, but somehow the scrolling and movement in X Window just does not "feel right," even if I tried tuning it with xinput.


So, being OCD'd, I tried to get the mouse to work in an acceptable manner in Windows 10 too. It does work somewhat okay-ish, using Apple's Bootcamp driver for the mouse which I got using my MacBook Pro. But the scrolling and "feel" are still somewhat "off".


Which brings us to this point. From my OCD perspective, this is all done just to get the mouse to work "right" on my setup below.


Hardware Setup:


Apple Magic Mouse (Series 1) <3 <3 <3


Asus Maximus Code IX

Intel i7-7700K

EVGA Nvidia GTX 1080 Hybrid

ASUS Strix AMD RX 560 (purchased for testing this setup)

G.Skill TridentZ DDR4-3200 16G

Samsung NVMe SSD 960 EVO M.2 250GB

Samsung SSD 850 PRO 256GB

EVGA Supernova 850w G2 Gold

Dell P4317Q 4K Monitor (43-Inch)

CoolerMaster MasterKeys Pro L (Cherry MX Red)

Sony Playstation 4 PRO

Thermaltake Core X71

Thermaltake Water 3.0

Apple MacBook Pro


Software Setup:


The SSD 850 Pro is the drive of interest here, since it is where I store the Linux host for learning computer science and programming as a hobby.  (The NVMe M.2 drive is installed with Windows 10 as my primary OS for daily use, so it is irrelevant here.)


I compiled Linux Gentoo 4.13.8 on the SSD 850 Pro as the host OS, with KVM, IOMMU, VFIO functions enabled in the kernel. I also compiled QEMU 2.10.0.




To get High Sierra (10.13 release) working on Linux with QEMU, I followed the instructions at <https://github.com/kholia/OSX-KVM>. The two OVMF files (OVMF_CODE-pure-efi.fd and OVMF_VARS-pure-efi-1024x768.fd) and also the Clover.qcow2 file there all worked out of the box. I downloaded those 3 files onto the Linux host, then prepared the requisite High Sierra USB installer by using the usual USB+Clover method that most folks use to test this (selecting the UEFI option under Clover, not the Legacy option).  After that, I used the script below (commenting out the 2 VFIO GPU passthrough lines -- the lines mentioning 03:00.0 and 03.00.1 -- and use gtk or vnc to output video since GPU passthrough is yet to be done) to get High Sierra installed and running with "soft" video output through gtk, vnc, spice, etc. for the moment, at least until GPU is passthrough later.  (Was I repeating myself there?  That is the trouble with OCD, there is this irresistible compulsion to do a certain thing in a specific way).  I have tried many other methods, but presently it seems that the instructions in the above GitHub link are the only ones that worked.


It is a bit trickier to get High Sierra to run with GPU passthrough to obtain near-native speed. To achieve that, once I installed and booted into High Sierra with soft video output, I [a] patched the AMD9500Controller.kext in /System/Library/Extensions in the High Sierra guest's hard disk by using the instructions at <http://www.insanelym...over-injection/>; and then  installed the Lilu.kext and WhateverGreen.kext into /System/Library/Extensions. It seems that the binary needed to drive the AMD RX 560 is already included in High Sierra, inside the AMD9500Controller.kext folder. All that is needed is to hex-modify the binary so that the hardware layout of the RX 560 is correctly reflected the binary file in that kext. I modified the "Acre" personality entry in the binary in AMD9500Controller.kext for convenience sake because it has 3 connectors at the back, the same number of connectors as the RX 560. (It is unclear whether [a] is needed if  is done -- I have not tested this scenario.)


Specifically, for example, this is what I changed in the binary with xxd:


For the "Acre" personality located at 0x121f80 in the binary file, I changed the hex from






to this






All the connectors (DP, HDMI, DVI) at the back of the card now work perfect. 


At any rate, I did both [a] and then , and High Sierra boots successfully with AMD RX 560 passthrough, using the following Linux QEMU command line script adapted from <https://github.com/kholia/OSX-KVM>:






export QEMU_AUDIO_DRV=alsa &&

qemu-system-x86_64 \

-enable-kvm \

-m 8192 \

-cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,$MY_OPTIONS\

-machine pc-q35-2.9 \

-smp cpus=8,sockets=1,cores=4,threads=2 \

-device isa-applesmc,osk="<i inserted the key from my Apple Macbook Pro here>" \

-drive if=pflash,format=raw,readonly,file=OVMF_CODE-pure-efi.fd \

-drive if=pflash,format=raw,file=OVMF_VARS-pure-efi-1024x768.fd \

-smbios type=2 \

-device ich9-intel-hda -device hda-duplex \

-device ide-drive,bus=ide.2,drive=Clover \

-drive id=Clover,if=none,snapshot=on,format=qcow2,file=./'Clover.qcow2' \

-device ide-drive,bus=ide.1,drive=MacHDD \

-drive id=MacHDD,if=none,file=./high.img,format=qcow2 \

-netdev user,id=usr0 -device e1000-82545em,netdev=usr0,id=vnet0 \

-balloon none \

-device vfio-pci,host=03:00.0,multifunction=on \

-device vfio-pci,host=03:00.1 \

-vga none \

-monitor unix:/tmp/monitor.sock,server,nowait \

-nographic \

-device vfio-pci,host=00:14.0,bus=pcie.0 \





Once the above is done, the separate AMDRadeonX4250.kext (responsible for 3D acceleration, etc) seems to be loaded by High Sierra and the RX 560 card should be functioning perfectly, directly connected to the guest OS with metal support.


Note that using the above command line, I have passthrough my USB controller as well (vfio-pci,host=00.14.0). The result is that bluetooth and all my USB ports (I have not tested the 3.1 one) worked out of the box, with my Apple Magic Mouse (Series 1) working perfectly with that tight, smooth buttery feel.  Also working flawlessly are my CoolerMaster L keyboard and all other USB devices, including USB external drives, etc). I would then control the Linux host underneath via ssh from the High Sierra guest. When the Dell monitor is dedicated to the High Sierra guest at full 4K, the linux host basically becomes transparent and invisible to the user.


Performance-wise, High Sierra runs _buttery smooth_ and _beautifully_ with the AMD RX560 passthrough in QEMU KVM. It is impressively silky, fast and responsive, with QE working and no glitches or hangs or crashes. Apps open almost instantaneously (split second). Ethernet works out of the box and the sound works perfect via the audio output jack of the Dell monitor which is connected to the AMD RX 560 via Display Port. In fact, the setup is so near-native that I'd speculate that a layperson would not notice the difference compared to say, a 2017 iMac (Geekbench 4 benchmark that I ran seem to suggest similar scores) unless the setup is revealed to him. Personally and anecdotally, I do not notice the difference even if I look for them. I mean, this thing is bat-out-of-cave fast. Certainly, it is a whole different league and at a whole different level from the usual slowish virtual box, parallels, vanilla vmware experience. Even compared to my MacBook Pro (also running High Sierra), this setup feels substantially smoother, faster and more responsive.


There are only two very minor noticeable glitches. First, flac audio playback on Fidelia would intermittently "tear" for split seconds if I concurrently run very heavy compile tasks in the Linux Gentoo OS underneath the QEMU/HighSierra. But this is expected. I have not tried CPU-pinning to dedicate specific CPUs to High Sierra yet, but I suppose using CPU-pinning, the lag can be removed since High Sierra would not then have to compete for CPU with the Gentoo Linux host running underneath. Netflix video playback on Chrome/Safari runs smooth, without any lag even under heavy load. Also, the sound in High Sierra via the AMD graphics card works perfectly -- does not suffer from the slight lag as in the case of running Win 10 in QEMU with the same card passed through. Second, in the High Sierra boot screen, the progress bar under the logo would tear slightly during boot up and appear to freeze (but it is still booting underneath) for say 5-6 seconds, before booting into the login/password screen.


With more fine-tuning, I think I can get this experimental set-up to run High Sierra even faster -- but as it is now, it feels like a native machine already. I am super, super impressed with the performance.


Windows 10:


For those who are interested, Windows 10 Enterprise (90-trial version) also works with this QEMU KVM GPU passthrough setup. Everything works out of the box without any patching. All that is needed a download and install of the AMD Radeon video drivers. Performance is very smooth too and near native -- except that the audio output lags behind the video output during Netflix playback by a second or two. I feel that Win 10 in this setup is not as impressive (the "Wow" factor") as getting High Sierra to work at near-native with GPU passthrough. Both are near-native, but High Sierra just "feels" (to me anecdotally) better, tighter, more finely tuned, more "buttery" and smoother in this setup as compared to Win10 under the same. Maybe it is just a function of graphical user interface design generally. I don't really know why.


Further Experiment:


Note that all of the above is done despite the fact that I have plugged the AMD RX 560 only in the PCIE x4 slot on the motherboard. This is because I already have the Nvidia GTX 1080 installed in the PCIE x8_1 slot. The AMD RX 560 can't be placed in the x8_2 slot because I use the GTX 1080 for the Linux OS and those two x8 slots are in the same IOMMU hardware group, and hardware in the same IOMMU group cannot be passthrough to two different OSes.


Thus, theoretically, performance in such experimental setup above could be further improved if I were to use the Intel HD630 onboard graphics for Linux, disconnect the power to the Nvidia GTX, and place the AMD RX 560 in the x8_2 slot for the passthrough. Additionally, I can try moving the Linux host to the NVMe .M2 drive to test for a significant disk speed improvement.


A much more interesting next experiment would be to procure a second AMD RX 560 card, and place it in a x4 slot, with the first AMD RX 560 card in the x8_2 slot, have Linux running on the Intel HD 630, and then passthrough both AMD cards to High Sierra and Win 10 simultaneously, such tha Linux Gentoo, High Sierra and Win10 all run on the same machine at the same time and all at native or near native-speed! 


Alternatively, if anyone here knows how to get the Nvidia GTX 1080 to passthrough to High Sierra or Windows via QEMU, I would appreciate it if you could share your knowledge! I have tried to do so many times, but it all ends in black screen with the Nvidia card.


I intend to achieve this as the final goal of this experiment -- with the Dell P4317Q displaying the 4 machines all at the same time -- four split screens at 1920x1080 resolution each, each split screen for each of linux, high sierra, windows, and also my PS 4 Pro.




With all this running on top of Gentoo, there seems no need to reboot or to troubleshoot incompatible hardware. There are no crashes in the host or the guest whatsoever except that the guest OS sometimes refuses to fire up.  I suspect that this is due to the GPU needing to power-cycle with a shutdown or reboot before it can be fire up again.  But I am unsure.  Also, backup of guest OSes seems easy -- I just copy the qcow2 hard disk file into storage. And I can fire up and power off the guest anytime I wish without interfering with the Gentoo host running underneath, which has basically become transparent. Also, this setup seems, in theory, more resistant to problems when upgrading the OS. And by the way, the messages app seems to work out of the box too.


It is interesting to note that with KVM allowing guest to access many aspects of the host hardware directly, in addition to passing through control of the GPU and USB controllers, discs, other devices etc. to the guest directly, plus pinning CPUs to the guest, the guest is arguably a hybrid of hardware and software, rather than just software. In fact, to me at least, it feels more bare metal than software.


Feel the POWER of Gentoo.




*All for fair-use, learning, experimental testing only*




<https://imgur.com/lxjIFUV(High Sierra GPU Passthrough at full 4K glory)


<https://imgur.com/Y66Yd8a> (High Sierra + Linux + PS4 Pro)


<https://imgur.com/yOSIQIg> (Win 10 + Linux + PS4 Pro)


<https://imgur.com/y6IgTAm> (Apple Magic Mouse (Series 1)) 





D. Kholia (https://github.com/kholia)

Share this post

Link to post
Share on other sites

Nice work, man. I'm planning to do something similar in the future (Virtualize macOS, Windows and a NAS-oriented OS under a KVM) but the only preocupation I had is the USB passtrought in my hardware of choice (A C612 based Supermicro Motherboard). I would hate to learn that the USB don't work or they work only in USB 2.0 mode cause High Sierra is incompatible with the chipset.


Did you try to hide KVM from the virtual machine to get that NVIDIA card working? I could be wrong, but I cant see any line that could hide the hypervisor to the virtual machine. Nvidia Drivers are know to refuse to work under VMs unless they're driving a Quadro card.

Share this post

Link to post
Share on other sites

Did you try to hide KVM from the virtual machine to get that NVIDIA card working? I could be wrong, but I cant see any line that could hide the hypervisor to the virtual machine. Nvidia Drivers are know to refuse to work under VMs unless they're driving a Quadro card.

I remember asking someone about this before, it doesn't seem like Nvidia does any check on MacOS, only on Windows VM (and maybe Linux VM too, not sure tbh)

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.