Jump to content

Different solutions

Andy Vandijck

By Andy Vandijck,
in Developers Corner

 Share
925 posts in this topic

Recommended Posts

  • Popular Post
Andy Vandijck
Andy Vandijck
Posted
  • Author
  • Popular Post
Posted

I decided to look into the workings of the algorithms that decompress the kernel cache (in boot.efi) of Apple's new LLVN compression algorithm and found a way to make a new usable version (code for LZVN decompression written in assembly).

I then added up a test program in C for handling of the proper handling of the decompression itself.

The assembly file is 64bit (x86_64) written in a .s file.

In order to use and call the routine of LZVN decompression a minor piece of code in assembly is needed.

Allso for the return value you'll need a minor piece of assembly.

 

Enjoy :D

 

LZVN decompression is finally here ;)

A minor rewrite of the .s file and the .c file would be required for 32-bit (i386) support but it is easily possible...

 

You can test it if you like :)

decomplzvn /Volumes/Secondary/System/Library/Caches/com.apple.kext.caches/Startup/kernelcache kernel_decomp
/Volumes/Secondary/System/Library/Caches/com.apple.kext.caches/Startup/kernelcache: Initial compressed size -> 3247493122, Reported uncompressed size -> 11565063
kernel_decomp: Actual decompressed size -> 11565063

B)

 

EDIT: Update - added also LZSS decompression and put together also all info I have on the LZVN decompression parts.

EDIT2: Added C function compatibility and preliminary LZVN compression algorithms

 

EDIT3:

Used the wrong function from kextcache from Yosemite, it wasn't the compression function.

I did some cleanup of the assembly code and I generated a proper C header to use the decompression function without assembly code.

It works identical to the decompress_lzss function as used in kextcache (current 10.9.4 sources).

It's called lzvn_decode, same arguments.

Also haven't been able to locate the 32-bit function yet, but there is one as there is a 32-bit binary of kextcache for Yosemite (could use some help there to find it, then I could make a full library for decompression).

I also built updated kext_tools for Mavericks with adapted code using my latest cleaned up version of the LZVN decompressor (in case of a 64-bit build, the LZVN decode functions are used).

Sources and binaries attached ;):D

 

EDIT4: Minor update in mkextunpack: option to specify compression type (-c follow by lzvn or lzss)

EDIT5: Another update -> added WIN64 version (updated sources, compiled with MinGW64 and added RT manifest, exe icon and rc script with version info.

EDIT6: New WIN64 version built with MSVC2013. No fully correct decompile yet, working on it...

EDIT7: Added nasm version (64-bit) of the lzvn decode algorithm, no 32-bit version yet nor any succesful decompile yet...

 

EDIT8: Tested and optimized the lzvn.c file for any OS and tested it with full warnings on the compiler and it worked beautifully

Reallt BIG THANKS TO MinusZwei for doing this.

I hope I can call onto your talents when I have the compression function working because very beautiful code with a bit of edits and extensions (even for MSVC any version)

With this new development I CAN add LZVN booting to chameleon, (and will do so after I've built WIN32 and WIN64 version of decomplzvn, with MSVC2013 no less :D)

Big salute to MinusZwei, you turned my asm 64 bit stolen LZVN decompression code into fully  functional C code!

I only optimized it to have types better human readable, did a size_t cast somewhere to size_t (was required or warning) and added a cross platform build able OSSwapInt64 version for Windows/Linux or other platforms and so) and it was perfect!

Very big thank you from all of us! ;)

New Mac versions of the tools are attached and Win32 and Win64 version of decompkernelcache will soon follow!

AND THEN... the big wanted Chameleon source mods :D... hang tight! I'm very excited!

 

EDIT9: Final bug fixes and optimizations to LZVN and decompkernelcache itself.

Now should be able to build optmized on virtually everything...

V1.8 is the optimized final version (fully working) for virtually any OS...

Prebuilt binaries for Windows (32+64) standalone optimized version...

Optimized binary for OS X built for running on Mac OS X 10.5 or newer, 32 or 64 bit Intel

 

 

EDIT9: I fixed the last issues with Windows systems.

I also improved some other things and it now fully builds for both 32 and 64 bit Windows (after resolving more casting issues).

Since this is the final version I've also compiled the latest version with full optimizations on (and tested, very, very fast! :D) 

I also decided with the -MT option (link in static library) and minimize the dependencies the Win32 and Win64 version need.

The only dll referenced in the .exe files is "KERNEL32.DLL" (which is, for Windows at least, the absolute minimum use of dependencies...

I tested both of them against a kernelcache file (one from a default install image) and in less than a second the output files where made!

The sizes checked out and the data was correct so it is perfect now.

I used MSVC 2013 Ultimate (the latest MS C/C++ compiler) with most warnings open...

It builds excellently (With /W3 and /WX).

Next up.... Chameleon source mod... (YEAH!)

Lets make sure our "legacy" good old loader can load Yosemite's kernelcache (Both LZSS and LZVN...)

The new sources can thus also be built with VIRTUALLY ANY OS, AND OPTIMIZED!

Good... next code mod of the day: Chameleon!

Keep you posted...

Binaries and sources of decompkernelcache V1.7 (Win and Mac, both 32 and 64 bit, optimized fully) are attached to this post!

EDIT: Added Github repo:

https://github.com/andyvand/decompkernelcache

kext_tools-326.90.2-AnV_R2.zip

decompkernelcache-V1.8-MacOSX_WIN32_WIN64.zip

  • Like 25
Link to comment
Share on other sites
Slice
Slice
Posted
Posted

Andy,

Thanks for your great work! I send it to revision 2819 but

your diff file in a nightmare so I can't check if I apply it correctly.

I corrected some places to do the project compilable and now I want to hear if some more corrections needed.

 

2all. Test, please!

  • Like 1
Link to comment
Share on other sites
Pene
Pene
Posted
Posted

I've decided to port the efifs drivers to EDK2 fully (instead of good old GNU EFI) because I wanted a lot more filesystems supported under UEFI.

I've now ported them all (all drivers from GRUB).

Great work Andy!
  • Like 1
Link to comment
Share on other sites
magnifico
magnifico
Posted
Posted

Andy,

Thanks for your great work! I send it to revision 2819 but

your diff file in a nightmare so I can't check if I apply it correctly.

I corrected some places to do the project compilable and now I want to hear if some more corrections needed.

 

2all. Test, please!

Please attached here 2819 ..i want to try

ok Sergey work perfect

 

 

0:100  0:000  Starting Clover rev 2819 on American Megatrends EFI

 

Only curiosity what is this ?

 

 

0:120  0:000  Kernel and Kext Patches:
0:120  0:000      Allowed: y
0:120  0:000      Debug: n
0:120  0:000      KernelCpu: n
0:120  0:000      Lapic: n
0:120  0:000      AICPUPM: n
0:120  0:000      AppleRTC: n
0:120  0:000      KernelPm: n
0:120  0:000      FakeCPUID: 0x0
0:120  0:000      ATIController: <null string>
0:120  0:000      ATIDataLength: 0
0:120  0:000      0 Kexts to load
0:120  0:000      0 Kexts to patch
Link to comment
Share on other sites
Pene
Pene
Posted
Posted

Andy,

Thanks for your great work! I send it to revision 2819 but

your diff file in a nightmare so I can't check if I apply it correctly.

I corrected some places to do the project compilable and now I want to hear if some more corrections needed.

 

2all. Test, please!

Slice, note that with r2819 you changed in fdf to VBoxHfs (instead of HFSPlus that was used previously).

This will break in CloverEFI the possibility to boot installers created by 'createinstallmedia' (no links support in VBoxHfs).

 

Also, just wondering, which diff file are you referring to? I did not see any diff file... :)

  • Like 1
Link to comment
Share on other sites
Neso
Neso
Posted
Posted

Please attached here 2819 ..i want to try

ok Sergey work perfect

0:100  0:000  Starting Clover rev 2819 on American Megatrends EFI

Only curiosity what is this ?

0:120  0:000  Kernel and Kext Patches:
0:120  0:000      Allowed: y
0:120  0:000      Debug: n
0:120  0:000      KernelCpu: n
0:120  0:000      Lapic: n
0:120  0:000      AICPUPM: n
0:120  0:000      AppleRTC: n
0:120  0:000      KernelPm: n
0:120  0:000      FakeCPUID: 0x0
0:120  0:000      ATIController: <null string>
0:120  0:000      ATIDataLength: 0
0:120  0:000      0 Kexts to load
0:120  0:000      0 Kexts to patch

Apianti did it when he was fixing kexttopatch frorm 2797 to 2806. He left expanded log for kext patching in case of troubles.

Link to comment
Share on other sites
Slice
Slice
Posted
Posted

Slice, note that with r2819 you changed in fdf to VBoxHfs (instead of HFSPlus that was used previously).

This will break in CloverEFI the possibility to boot installers created by 'createinstallmedia' (no links support in VBoxHfs).

 

Also, just wondering, which diff file are you referring to? I did not see any diff file... :)

O-oh! Sorry for the fdf.  In this long series I didn't see this.

The diff file Andy gave me in PM.

  • Like 1
Link to comment
Share on other sites
Micky1979
Micky1979
Posted
Posted 

Last login: Tue Aug 19 12:16:24 on ttys000
new-host-2:~ Micky1979$ cd /Users/Micky1979/Desktop/lzvn/decomplzvn
new-host-2:decomplzvn Micky1979$ make clean
rm -f *.o decomplzvn
new-host-2:decomplzvn Micky1979$ make
gcc -arch x86_64 -O2 -c decomplzvn.c -o decomplzvn.oas -arch x86_64 -o lzvn.o lzvn.s
gcc -arch x86_64 -O2 -o decomplzvn *.o
new-host-2:decomplzvn Micky1979$ /Users/Micky1979/Desktop/lzvn2/decomplzvn/decomplzvn /Users/Micky1979/Desktop/kernelcache /Users/Micky1979/Desktop/kernel_decomp
/Users/Micky1979/Desktop/kernelcache: Initial compressed size -> 1116599296, Reported uncompressed size -> 14711298
/Users/Micky1979/Desktop/kernel_decomp: Actual decompressed size -> 14711298
new-host-2:decomplzvn Micky1979$ 

 

 decomplzvn.zip

 decompkernelcache.zip

after compiling in Lion (segmentation fault 11 otherwise), it work in Lion Also (yes, I'm on 10.7.5)  :thumbsup_anim:

 

test result:

Last login: Tue Aug 19 12:19:14 on ttys000
new-host-2:~ Micky1979$ hexdump -s 0x3E8050 -C /Users/Micky1979/Desktop/kernel_decomp  | grep " [0-9][0-9]\.[0-9]\.[0-9]: " | awk -F "|" '{ print $2 }' | perl -pi -e 's/[a-z]*\ //' | awk -F : '{ print $1 }'
14.0.0
new-host-2:~ Micky1979$  

Yosemite kernel 14.0.0  :thumbsup_anim:  :thumbsup_anim:  :thumbsup_anim: 

 kernel_decomp.zip

 

Andy, thanks again

  • Like 1
Link to comment
Share on other sites
Dmos
Dmos
Posted
Posted

Same :( here ....looks Rocket science to me..... is it possible  ... De-compile .. your thoughts little bit  and make simple writing so we noob(me).. Can Understand  it  like wise :)

Link to comment
Share on other sites
Andy Vandijck
Andy Vandijck
Posted
  • Author
Posted

I'm currently doing an attempt to decompile the assembly file.

Keep you posted ;)

  • Like 2
Link to comment
Share on other sites
magnifico
magnifico
Posted
Posted

What is the benefit of LZVN vs LZSS? Does it compress/decompress faster or something? Boot times seem to be the same for me in Yosemite vs. Mavericks.

 

Consider that for this reason the legacy bootloader they needed to have the kernel in the installer, downloaded here and there, to boot the installer, but no longer because with the tool, of AnV, you can remove it from the kernelcache!

 

PS : should be so, at least I think .... or not Andy?

  • Like 1
Link to comment
Share on other sites
Micky1979
Micky1979
Posted
Posted

What is the benefit of LZVN vs LZSS? Does it compress/decompress faster or something? Boot times seem to be the same for me in Yosemite vs. Mavericks.

Frankly, we do not even know if in Yosemite GM Apple will remove the "preferred compression" entry in  /usr/standalone/bootcaches.plist.

Thinking about it does not make much sense (for Apple) to keep it.

Speed ​​of compression and decompression in memory maybe it was the goal using lzvn:

I think Apple is using lzvn to compress a lot of  its "Payload".....to save space, memory and then speed.

 

Micky

Link to comment
Share on other sites
Pene
Pene
Posted
Posted

Speaking of the HFS driver, did anyone test if this grub HFS driver properly supports links (aka can boot media created by 'createinstallmedia')? I'm away from home for a while, so I can't really test things.

 

If it properly reads 'createinstallmedia' disk, I think it might be a good idea to switch the driver used in Clover to this one instead of VBoxHfs (if it is OK with Andy, of course).

  • Like 2
Link to comment
Share on other sites
Andy Vandijck
Andy Vandijck
Posted
  • Author
Posted

Speaking of the HFS driver, did anyone test if this grub HFS driver properly supports links (aka can boot media created by 'createinstallmedia')? I'm away from home for a while, so I can't really test things.

 

If it properly reads 'createinstallmedia' disk, I think it might be a good idea to switch the driver used in Clover to this one instead of VBoxHfs (if it is OK with Andy, of course).

Sure, no problem, I made these public for everyone.

Use it as you see fit ;)

  • Like 2
Link to comment
Share on other sites
Pene
Pene
Posted
Posted

Hi Andy,

Thanks.

I tried to test this HfsPlus driver, under Qemu with OvmfClover firmware compiled with DEBUG mode ( ./ebuild.sh -D CSM_ENABLE -b DEBUG).
This allows me to see debugging information printed by edk2 code.

Edk2 Asserts when Grub HfsPlus loads with:

InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7F170A40
Loading driver at 0x0007EC17000 EntryPoint=0x0007EC172AF HFSPLUS.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7F174218
ASSERT /Users/pene/Desktop/CloverGrowerPro/edk2/MdeModulePkg/Core/Dxe/Mem/Pool.c(425): CR has Bad Signature

This happens both if I use the binary you uploaded, and also if I compile it myself.
Not sure what's wrong yet, but something is.
If I had tried without DEBUG, it may have passed this point (with RELEASE asserts are disabled), but this issue needs to be ironed out.

The assertion is at CoreFreePoolI (), so this is most probably a memory management issue.
I didn't really try to look into it deeply yet to figure out why, but if you have any ideas, they are welcome :)

  • Like 1
Link to comment
Share on other sites
 Share
Go to topic listing
×
×
  • Create New...