Got it working now. Had to use the Radeon framebuffer driver instead of Baladi. So important lessons for me were:
- make sure to have a UEFI vbios on the graphics card (use the dual bios toggle so you can easily swap between UEFI and Legacy vbios when necessary, the Z170X uefi setup menu doesn't always work with the UEFI vbios)
- disable the networking stack in the UEFI settings, as well as IOAPIC and CSM
- enable XHC Hand Off
- don't use fakeid and use the correct framebuffer
Now only problem is that sleep doesn't work; the screen goes black but the computer doesn't go to sleep and just stops responding entirely.