Popular Post Andy Vandijck Posted August 16, 2014 Author Popular Post Share Posted August 16, 2014 I decided to look into the workings of the algorithms that decompress the kernel cache (in boot.efi) of Apple's new LLVN compression algorithm and found a way to make a new usable version (code for LZVN decompression written in assembly). I then added up a test program in C for handling of the proper handling of the decompression itself. The assembly file is 64bit (x86_64) written in a .s file. In order to use and call the routine of LZVN decompression a minor piece of code in assembly is needed. Allso for the return value you'll need a minor piece of assembly. Enjoy LZVN decompression is finally here A minor rewrite of the .s file and the .c file would be required for 32-bit (i386) support but it is easily possible... You can test it if you like decomplzvn /Volumes/Secondary/System/Library/Caches/com.apple.kext.caches/Startup/kernelcache kernel_decomp /Volumes/Secondary/System/Library/Caches/com.apple.kext.caches/Startup/kernelcache: Initial compressed size -> 3247493122, Reported uncompressed size -> 11565063 kernel_decomp: Actual decompressed size -> 11565063 EDIT: Update - added also LZSS decompression and put together also all info I have on the LZVN decompression parts. EDIT2: Added C function compatibility and preliminary LZVN compression algorithms EDIT3: Used the wrong function from kextcache from Yosemite, it wasn't the compression function. I did some cleanup of the assembly code and I generated a proper C header to use the decompression function without assembly code. It works identical to the decompress_lzss function as used in kextcache (current 10.9.4 sources). It's called lzvn_decode, same arguments. Also haven't been able to locate the 32-bit function yet, but there is one as there is a 32-bit binary of kextcache for Yosemite (could use some help there to find it, then I could make a full library for decompression). I also built updated kext_tools for Mavericks with adapted code using my latest cleaned up version of the LZVN decompressor (in case of a 64-bit build, the LZVN decode functions are used). Sources and binaries attached EDIT4: Minor update in mkextunpack: option to specify compression type (-c follow by lzvn or lzss) EDIT5: Another update -> added WIN64 version (updated sources, compiled with MinGW64 and added RT manifest, exe icon and rc script with version info. EDIT6: New WIN64 version built with MSVC2013. No fully correct decompile yet, working on it... EDIT7: Added nasm version (64-bit) of the lzvn decode algorithm, no 32-bit version yet nor any succesful decompile yet... EDIT8: Tested and optimized the lzvn.c file for any OS and tested it with full warnings on the compiler and it worked beautifully Reallt BIG THANKS TO MinusZwei for doing this. I hope I can call onto your talents when I have the compression function working because very beautiful code with a bit of edits and extensions (even for MSVC any version) With this new development I CAN add LZVN booting to chameleon, (and will do so after I've built WIN32 and WIN64 version of decomplzvn, with MSVC2013 no less ) Big salute to MinusZwei, you turned my asm 64 bit stolen LZVN decompression code into fully functional C code! I only optimized it to have types better human readable, did a size_t cast somewhere to size_t (was required or warning) and added a cross platform build able OSSwapInt64 version for Windows/Linux or other platforms and so) and it was perfect! Very big thank you from all of us! New Mac versions of the tools are attached and Win32 and Win64 version of decompkernelcache will soon follow! AND THEN... the big wanted Chameleon source mods ... hang tight! I'm very excited! EDIT9: Final bug fixes and optimizations to LZVN and decompkernelcache itself. Now should be able to build optmized on virtually everything... V1.8 is the optimized final version (fully working) for virtually any OS... Prebuilt binaries for Windows (32+64) standalone optimized version... Optimized binary for OS X built for running on Mac OS X 10.5 or newer, 32 or 64 bit Intel EDIT9: I fixed the last issues with Windows systems. I also improved some other things and it now fully builds for both 32 and 64 bit Windows (after resolving more casting issues). Since this is the final version I've also compiled the latest version with full optimizations on (and tested, very, very fast! ) I also decided with the -MT option (link in static library) and minimize the dependencies the Win32 and Win64 version need. The only dll referenced in the .exe files is "KERNEL32.DLL" (which is, for Windows at least, the absolute minimum use of dependencies... I tested both of them against a kernelcache file (one from a default install image) and in less than a second the output files where made! The sizes checked out and the data was correct so it is perfect now. I used MSVC 2013 Ultimate (the latest MS C/C++ compiler) with most warnings open... It builds excellently (With /W3 and /WX). Next up.... Chameleon source mod... (YEAH!) Lets make sure our "legacy" good old loader can load Yosemite's kernelcache (Both LZSS and LZVN...) The new sources can thus also be built with VIRTUALLY ANY OS, AND OPTIMIZED! Good... next code mod of the day: Chameleon! Keep you posted... Binaries and sources of decompkernelcache V1.7 (Win and Mac, both 32 and 64 bit, optimized fully) are attached to this post! EDIT: Added Github repo: https://github.com/andyvand/decompkernelcache kext_tools-326.90.2-AnV_R2.zip decompkernelcache-V1.8-MacOSX_WIN32_WIN64.zip 25 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2052235 Share on other sites More sharing options...
joe75 Posted August 16, 2014 Share Posted August 16, 2014 man, the hits keep coming with you! 3 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2052238 Share on other sites More sharing options...
mendietinha Posted August 16, 2014 Share Posted August 16, 2014 superb, mate! 1 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2052248 Share on other sites More sharing options...
Slice Posted August 16, 2014 Share Posted August 16, 2014 Andy, Thanks for your great work! I send it to revision 2819 but your diff file in a nightmare so I can't check if I apply it correctly. I corrected some places to do the project compilable and now I want to hear if some more corrections needed. 2all. Test, please! 1 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2052250 Share on other sites More sharing options...
Pene Posted August 16, 2014 Share Posted August 16, 2014 I've decided to port the efifs drivers to EDK2 fully (instead of good old GNU EFI) because I wanted a lot more filesystems supported under UEFI. I've now ported them all (all drivers from GRUB). Great work Andy! 1 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2052257 Share on other sites More sharing options...
magnifico Posted August 16, 2014 Share Posted August 16, 2014 I know that there would be improvements with these drivers? I can only change hfsplus, I do not see other drivers that use usually in folder 64 (UEFI) @ pene , how are you ? 1 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2052264 Share on other sites More sharing options...
Fabio1971 Posted August 16, 2014 Share Posted August 16, 2014 @ Andy Great job, in my case I can try HFS-64.efi Fabio Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2052298 Share on other sites More sharing options...
magnifico Posted August 16, 2014 Share Posted August 16, 2014 Andy, Thanks for your great work! I send it to revision 2819 but your diff file in a nightmare so I can't check if I apply it correctly. I corrected some places to do the project compilable and now I want to hear if some more corrections needed. 2all. Test, please! Please attached here 2819 ..i want to try ok Sergey work perfect 0:100 0:000 Starting Clover rev 2819 on American Megatrends EFI Only curiosity what is this ? 0:120 0:000 Kernel and Kext Patches: 0:120 0:000 Allowed: y 0:120 0:000 Debug: n 0:120 0:000 KernelCpu: n 0:120 0:000 Lapic: n 0:120 0:000 AICPUPM: n 0:120 0:000 AppleRTC: n 0:120 0:000 KernelPm: n 0:120 0:000 FakeCPUID: 0x0 0:120 0:000 ATIController: <null string> 0:120 0:000 ATIDataLength: 0 0:120 0:000 0 Kexts to load 0:120 0:000 0 Kexts to patch Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2052327 Share on other sites More sharing options...
Pene Posted August 16, 2014 Share Posted August 16, 2014 Andy, Thanks for your great work! I send it to revision 2819 but your diff file in a nightmare so I can't check if I apply it correctly. I corrected some places to do the project compilable and now I want to hear if some more corrections needed. 2all. Test, please! Slice, note that with r2819 you changed in fdf to VBoxHfs (instead of HFSPlus that was used previously). This will break in CloverEFI the possibility to boot installers created by 'createinstallmedia' (no links support in VBoxHfs). Also, just wondering, which diff file are you referring to? I did not see any diff file... 1 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2052465 Share on other sites More sharing options...
Neso Posted August 16, 2014 Share Posted August 16, 2014 Please attached here 2819 ..i want to try ok Sergey work perfect 0:100 0:000 Starting Clover rev 2819 on American Megatrends EFI Only curiosity what is this ? 0:120 0:000 Kernel and Kext Patches: 0:120 0:000 Allowed: y 0:120 0:000 Debug: n 0:120 0:000 KernelCpu: n 0:120 0:000 Lapic: n 0:120 0:000 AICPUPM: n 0:120 0:000 AppleRTC: n 0:120 0:000 KernelPm: n 0:120 0:000 FakeCPUID: 0x0 0:120 0:000 ATIController: <null string> 0:120 0:000 ATIDataLength: 0 0:120 0:000 0 Kexts to load 0:120 0:000 0 Kexts to patch Apianti did it when he was fixing kexttopatch frorm 2797 to 2806. He left expanded log for kext patching in case of troubles. Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2052472 Share on other sites More sharing options...
magnifico Posted August 16, 2014 Share Posted August 16, 2014 Apianti did it when he was fixing kexttopatch frorm 2797 to 2806. He left expanded log for kext patching in case of troubles. ah ..ok ok Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2052474 Share on other sites More sharing options...
Slice Posted August 17, 2014 Share Posted August 17, 2014 Slice, note that with r2819 you changed in fdf to VBoxHfs (instead of HFSPlus that was used previously). This will break in CloverEFI the possibility to boot installers created by 'createinstallmedia' (no links support in VBoxHfs). Also, just wondering, which diff file are you referring to? I did not see any diff file... O-oh! Sorry for the fdf. In this long series I didn't see this. The diff file Andy gave me in PM. 1 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2052759 Share on other sites More sharing options...
Pike R. Alpha Posted August 18, 2014 Share Posted August 18, 2014 Great work Andy! See PM for 32-bit pointers 3 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2052959 Share on other sites More sharing options...
Micky1979 Posted August 19, 2014 Share Posted August 19, 2014 Wow, big big big big work! congratulations for this superb job thank you very much Andy! 2 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2053245 Share on other sites More sharing options...
Micky1979 Posted August 19, 2014 Share Posted August 19, 2014 Last login: Tue Aug 19 12:16:24 on ttys000 new-host-2:~ Micky1979$ cd /Users/Micky1979/Desktop/lzvn/decomplzvn new-host-2:decomplzvn Micky1979$ make clean rm -f *.o decomplzvn new-host-2:decomplzvn Micky1979$ make gcc -arch x86_64 -O2 -c decomplzvn.c -o decomplzvn.oas -arch x86_64 -o lzvn.o lzvn.s gcc -arch x86_64 -O2 -o decomplzvn *.o new-host-2:decomplzvn Micky1979$ /Users/Micky1979/Desktop/lzvn2/decomplzvn/decomplzvn /Users/Micky1979/Desktop/kernelcache /Users/Micky1979/Desktop/kernel_decomp /Users/Micky1979/Desktop/kernelcache: Initial compressed size -> 1116599296, Reported uncompressed size -> 14711298 /Users/Micky1979/Desktop/kernel_decomp: Actual decompressed size -> 14711298 new-host-2:decomplzvn Micky1979$ decomplzvn.zip decompkernelcache.zip after compiling in Lion (segmentation fault 11 otherwise), it work in Lion Also (yes, I'm on 10.7.5) test result: Last login: Tue Aug 19 12:19:14 on ttys000 new-host-2:~ Micky1979$ hexdump -s 0x3E8050 -C /Users/Micky1979/Desktop/kernel_decomp | grep " [0-9][0-9]\.[0-9]\.[0-9]: " | awk -F "|" '{ print $2 }' | perl -pi -e 's/[a-z]*\ //' | awk -F : '{ print $1 }' 14.0.0 new-host-2:~ Micky1979$ Yosemite kernel 14.0.0 kernel_decomp.zip Andy, thanks again 1 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2053260 Share on other sites More sharing options...
Extreme™ Posted August 19, 2014 Share Posted August 19, 2014 To Andy Vandijck Hello!.... Possible to have a simple explanation of what you have made? Sorry, i don't uderstand what is it and how is it used your files... I'm still inexperienced of hack. Thank you sir! 1 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2053310 Share on other sites More sharing options...
Dmos Posted August 20, 2014 Share Posted August 20, 2014 Same here ....looks Rocket science to me..... is it possible ... De-compile .. your thoughts little bit and make simple writing so we noob(me).. Can Understand it like wise Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2053545 Share on other sites More sharing options...
Andy Vandijck Posted August 21, 2014 Author Share Posted August 21, 2014 I'm currently doing an attempt to decompile the assembly file. Keep you posted 2 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2053830 Share on other sites More sharing options...
jamiethemorris Posted August 21, 2014 Share Posted August 21, 2014 What is the benefit of LZVN vs LZSS? Does it compress/decompress faster or something? Boot times seem to be the same for me in Yosemite vs. Mavericks. Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2053848 Share on other sites More sharing options...
magnifico Posted August 21, 2014 Share Posted August 21, 2014 What is the benefit of LZVN vs LZSS? Does it compress/decompress faster or something? Boot times seem to be the same for me in Yosemite vs. Mavericks. Consider that for this reason the legacy bootloader they needed to have the kernel in the installer, downloaded here and there, to boot the installer, but no longer because with the tool, of AnV, you can remove it from the kernelcache! PS : should be so, at least I think .... or not Andy? 1 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2053856 Share on other sites More sharing options...
Micky1979 Posted August 21, 2014 Share Posted August 21, 2014 What is the benefit of LZVN vs LZSS? Does it compress/decompress faster or something? Boot times seem to be the same for me in Yosemite vs. Mavericks. Frankly, we do not even know if in Yosemite GM Apple will remove the "preferred compression" entry in /usr/standalone/bootcaches.plist. Thinking about it does not make much sense (for Apple) to keep it. Speed of compression and decompression in memory maybe it was the goal using lzvn: I think Apple is using lzvn to compress a lot of its "Payload".....to save space, memory and then speed. Micky Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2053980 Share on other sites More sharing options...
Pene Posted August 23, 2014 Share Posted August 23, 2014 Speaking of the HFS driver, did anyone test if this grub HFS driver properly supports links (aka can boot media created by 'createinstallmedia')? I'm away from home for a while, so I can't really test things. If it properly reads 'createinstallmedia' disk, I think it might be a good idea to switch the driver used in Clover to this one instead of VBoxHfs (if it is OK with Andy, of course). 2 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2054666 Share on other sites More sharing options...
Andy Vandijck Posted August 24, 2014 Author Share Posted August 24, 2014 Speaking of the HFS driver, did anyone test if this grub HFS driver properly supports links (aka can boot media created by 'createinstallmedia')? I'm away from home for a while, so I can't really test things. If it properly reads 'createinstallmedia' disk, I think it might be a good idea to switch the driver used in Clover to this one instead of VBoxHfs (if it is OK with Andy, of course). Sure, no problem, I made these public for everyone. Use it as you see fit 2 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2054892 Share on other sites More sharing options...
Pene Posted August 25, 2014 Share Posted August 25, 2014 Hi Andy,Thanks.I tried to test this HfsPlus driver, under Qemu with OvmfClover firmware compiled with DEBUG mode ( ./ebuild.sh -D CSM_ENABLE -b DEBUG).This allows me to see debugging information printed by edk2 code.Edk2 Asserts when Grub HfsPlus loads with: InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7F170A40 Loading driver at 0x0007EC17000 EntryPoint=0x0007EC172AF HFSPLUS.efi InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7F174218 ASSERT /Users/pene/Desktop/CloverGrowerPro/edk2/MdeModulePkg/Core/Dxe/Mem/Pool.c(425): CR has Bad Signature This happens both if I use the binary you uploaded, and also if I compile it myself.Not sure what's wrong yet, but something is.If I had tried without DEBUG, it may have passed this point (with RELEASE asserts are disabled), but this issue needs to be ironed out.The assertion is at CoreFreePoolI (), so this is most probably a memory management issue.I didn't really try to look into it deeply yet to figure out why, but if you have any ideas, they are welcome 1 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2055192 Share on other sites More sharing options...
dmazar Posted August 25, 2014 Share Posted August 25, 2014 Assert can happen in FreePool if you pass it something that is not allocated with AllocatePool. For example: possible issue in grub_driver.c, GetFSGuid(): FreePool(FsName). 2 Link to comment https://www.insanelymac.com/forum/topic/294634-different-solutions/page/23/#findComment-2055201 Share on other sites More sharing options...
Recommended Posts