NVMeFix

978b029efa981d618699b09868 · March 24, 2020

18 hours ago, Mrfalcon7 said:

Yes it panicked. It boots sometimes but freezes while using the system. Yea i will try to switch to opencore tonight.

See the discussion here as well: https://github.com/acidanthera/bugtracker/issues/780

FredWst · April 1, 2020

Hi,

Just to report that KBG40ZNS256G NVMe KIOXIA 256GB seems to be unsupported.

Optiplex dell 3070.


2020-04-01 22:50:05.793492+0200 0x73       Default     0x0                  0      0    kernel: (Lilu) Lilu       api: @ (DBG) got load request from NVMeFix (102)

2020-04-01 22:50:05.793543+0200 0x73       Default     0x0                  0      0    kernel: (Lilu) NVMeFix      init: @ (DBG) NVMeFix bootstrap DBG-102-2020-04-01

2020-04-01 22:50:05.793603+0200 0x73       Default     0x0                  0      0    kernel: (Lilu) NVMeFix     nvmef: @ (DBG) Registered for matching notifications

2020-04-01 22:50:06.207963+0200 0xcb       Default     0x0                  0      0    kernel: (IONVMeFamily) AppleNVMe Assert failed: ( 0 != data )

2020-04-01 22:50:06.219908+0200 0xcb       Default     0x0                  0      0    kernel: (IONVMeFamily) ReleaseIDNode

2020-04-01 22:50:06.220359+0200 0xcb       Default     0x0                  0      0    kernel: (IONVMeFamily) file: /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-470.100.17/IONVMeController.cpp

2020-04-01 22:50:06.220808+0200 0xcb       Default     0x0                  0      0    kernel: (IONVMeFamily) line: 5478

2020-04-01 22:50:06.228397+0200 0xcb       Default     0x0                  0      0    kernel: (IONVMeFamily) 

2020-04-01 22:50:06.230241+0200 0xcb       Default     0x0                  0      0    kernel: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=0 entrysize=64

2020-04-01 22:50:06.253665+0200 0xcb       Default     0x0                  0      0    kernel: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=0 entrysize=64

2020-04-01 22:50:06.253672+0200 0xcb       Default     0x0                  0      0    kernel: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=1 entrysize=64

2020-04-01 22:50:06.268809+0200 0xcb       Default     0x0                  0      0    kernel: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2861:SQ index=1 entrysize=64

2020-04-01 22:50:07.078935+0200 0xfb       Default     0x0                  0      0    kernel: (Lilu) NVMeFix     nvmef: @ (DBG) matchingNotificationHandler for IOMediaBSDClient

2020-04-01 22:50:07.088659+0200 0xff       Default     0x0                  0      0    kernel: (Lilu) NVMeFix     nvmef: @ (DBG) matchingNotificationHandler for IOMediaBSDClient

2020-04-01 22:50:19.406532+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) Lilu   patcher: @ (DBG) discovered the right kext com.apple.iokit.IONVMeFamily at 0xFFFFFF7F84052000, invoking handler

2020-04-01 22:50:19.423048+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix     nvmef: @ (DBG) processKext com.apple.iokit.IONVMeFamily

2020-04-01 22:50:19.430814+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) Lilu      mach: @ (DBG) found symbol __ZN16IONVMeController20IssueIdentifyCommandEP18IOMemoryDescriptorP16AppleNVMeRequestj at 0xffffff7f84063f5e (non-aslr 0x11f5e)

2020-04-01 22:50:19.442415+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) Lilu      mach: @ (DBG) found symbol __ZN16IONVMeController22ProcessSyncNVMeRequestEP16AppleNVMeRequest at 0xffffff7f8406415e (non-aslr 0x1215e)

2020-04-01 22:50:19.452995+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) Lilu      mach: @ (DBG) found symbol __ZN16IONVMeController10GetRequestEj at 0xffffff7f84067b64 (non-aslr 0x15b64)

2020-04-01 22:50:19.462617+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) Lilu      mach: @ (DBG) found symbol __ZN16AppleNVMeRequest23BuildCommandGetFeaturesEh at 0xffffff7f8406d742 (non-aslr 0x1b742)

2020-04-01 22:50:19.472796+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) Lilu      mach: @ (DBG) found symbol __ZN16AppleNVMeRequest29BuildCommandSetFeaturesCommonEh at 0xffffff7f8406d776 (non-aslr 0x1b776)

2020-04-01 22:50:19.483187+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) Lilu      mach: @ (DBG) found symbol __ZN16IONVMeController13ReturnRequestEP16AppleNVMeRequest at 0xffffff7f84067d2c (non-aslr 0x15d2c)

2020-04-01 22:50:19.493788+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) Lilu      mach: @ (DBG) found symbol __ZN16AppleNVMeRequest9GetStatusEv at 0xffffff7f8406c2e8 (non-aslr 0x1a2e8)

2020-04-01 22:50:19.504503+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) Lilu      mach: @ (DBG) found symbol __ZN16AppleNVMeRequest9GetOpcodeEv at 0xffffff7f8406c2ce (non-aslr 0x1a2ce)

2020-04-01 22:50:19.514618+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) Lilu      mach: @ (DBG) found symbol __ZN16AppleNVMeRequest20GenerateIOVMSegmentsEyy at 0xffffff7f8406c8f6 (non-aslr 0x1a8f6)

2020-04-01 22:50:19.525218+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) Lilu      mach: @ (DBG) found symbol __ZN16IONVMeController22FilterInterruptRequestEP28IOFilterInterruptEventSource at 0xffffff7f840663f2 (non-aslr 0x143f2)

2020-04-01 22:50:19.536895+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) Lilu      mach: @ (DBG) found symbol __ZTV16IONVMeController at 0xffffff7f8407da08 (non-aslr 0x2ba08)

2020-04-01 22:50:19.547232+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix     nvmef: @ (DBG) handleControllers for 1 controllers

2020-04-01 22:50:19.557092+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix     iokit: @ (DBG) getOSData vendor-id has 1E0F value

2020-04-01 22:50:19.566529+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix     iokit: @ (DBG) getOSData vendor-id has 1E0F value

2020-04-01 22:50:19.575993+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix     iokit: @ (DBG) getOSData device-id has 1 value

2020-04-01 22:50:19.585361+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix    quirks: @ (DBG) Reading OEM info from IODT

2020-04-01 22:50:19.594576+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix    quirks: @ (DBG) Failed to find IODeviceTree:/efi/platform OEMProduct

2020-04-01 22:50:19.604549+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix    quirks: @ (DBG) Failed to find IODeviceTree:/efi/platform OEMVendor

2020-04-01 22:50:19.614491+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix    quirks: @ (DBG) Failed to find IODeviceTree:/efi/platform OEMBoard

2020-04-01 22:50:19.624416+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix    quirks: @ (DBG) Reading OEM info from NVRAM

2020-04-01 22:50:19.643204+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix    quirks: @ (DBG) Failed to find LiluVendorGuid:oem-product

2020-04-01 22:50:19.662779+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix    quirks: @ (DBG) Failed to find LiluVendorGuid:oem-vendor

2020-04-01 22:50:19.685081+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix    quirks: @ (DBG) Failed to find LiluVendorGuid:oem-board

2020-04-01 22:50:19.696994+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix     nvmef: @ (DBG) Property ps-max-latency-us not found for parent service

2020-04-01 22:50:19.710603+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix     nvmef: @ (DBG) Identified model KBG40ZNS256G NVMe KIOXIA 256GB          (vid 0x1e0f)

2020-04-01 22:50:19.723127+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix      apst: @ (DBG) APST status 0

2020-04-01 22:50:19.733277+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix      apst: @ (DBG) Configuring APST

2020-04-01 22:50:19.743375+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix      apst: @ (DBG) Set entry 3 to 0x6d620

2020-04-01 22:50:19.753640+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix      apst: @ (DBG) Set entry 2 to 0x6418

2020-04-01 22:50:19.763752+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix      apst: @ (DBG) Set entry 1 to 0x6418

2020-04-01 22:50:19.773425+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix      apst: @ (DBG) Set entry 0 to 0x6418

2020-04-01 22:50:19.782591+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix      apst: @ (DBG) APST enabled: max PS = 4, max round-trip latency = 35000us

2020-04-01 22:50:19.802921+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix      apst: @ (DBG) APST status 1

2020-04-01 22:50:19.814129+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix      apst: @ (DBG) entry 4 : 0x0

2020-04-01 22:50:19.823317+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix      apst: @ (DBG) entry 3 : 0x6d620

2020-04-01 22:50:19.832636+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix      apst: @ (DBG) entry 2 : 0x6418

2020-04-01 22:50:19.841918+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix      apst: @ (DBG) entry 1 : 0x6418

2020-04-01 22:50:19.851204+0200 0x507      Default     0x0                  0      0    kernel: (Lilu) NVMeFix      apst: @ (DBG) entry 0 : 0x6418

2020-04-01 22:50:22.483684+0200 0x505      Default     0x0                  0      0    kernel: (Lilu) Lilu       dev: @ (DBG) found IONVMeController on pci bridge

2020-04-01 22:50:22.505108+0200 0x505      Default     0x0                  0      0    kernel: (Lilu) Lilu       dev: @ (DBG) found IONVMeBlockStorageDevice on pci bridge

2020-04-01 22:50:22.512706+0200 0x505      Default     0x0                  0      0    kernel: (Lilu) Lilu       dev: @ (DBG) found KBG40ZNS256G NVMe KIOXIA 256GB Media on pci bridge

Fred

978b029efa981d618699b09868 · April 2, 2020

12 hours ago, FredWst said:

Just to report that KBG40ZNS256G NVMe KIOXIA 256GB seems to be unsupported.

What makes you think so? The log does not hint at anything like that.

FredWst · April 2, 2020

1 hour ago, 978b029efa981d618699b09868 said:

What makes you think so? The log does not hint at anything like that.

Yes, I was wrong.

Fred

Zach.Gao · April 10, 2020

A silly question: Where to specify 'ps-max-latency-us' property?

I tried to add to I tried to add this to NVRAM under GUID '7C436110-AB2A-4BBB-A880-FE41995C9F82' but in vain.

APST state is still 'true'.

Toroop · April 15, 2020

Hi

I installed NVMEFix in /L/E but I don't get the NVMePMProxy in IOReg. The kext seems to load ok though and apst is set to true.

I tried to install the debug version of NVMEFix and added -nvmefdbg to Clover's boot args but the only messages I got are:

2020-04-15 15:24:26.734776+1000 localhost kernel[0]: (Lilu) <Lilu`lilu_os_log> NVMeFix init: @ (DBG) NVMeFix bootstrap DBG-102-2020-04-06

2020-04-15 15:24:26.737030+1000 localhost kernel[0]: (Lilu) <Lilu`lilu_os_log> NVMeFix nvmef: @ (DBG) Registered for matching notifications

I used this query:

log show --predicate 'process == "kernel" AND (eventMessage CONTAINS "NVMe")' --style syslog --source

Any idea?

Thanks

davidm71 · April 17, 2020

This kext has me curious. If I was to use Clover lets say to boot up my Laptop running Windows with NVMEFix would my laptop power consumption go down and get longer battery life??

Thanks

MacKonsti · April 19, 2020

On 4/15/2020 at 7:40 AM, Toroop said:

I installed NVMEFix in /L/E but I don't get the NVMePMProxy in IOReg. The kext seems to load ok though and apst is set to true.

Hi everyone, I know this sounds like a n00b question, despite following many years in the Hackintosh developments, but how can we check if NVMeFix.kext works? What is the expected output on IOReg? Same as @Toroop

the kext is run in /L/E/ and seems to load fine but I am not sure how to confirm that it's running as it should. Mind you, had no problems with my Samsung EVO 970 NVMe drive, before discovering this kext.

With IORegistryExplorer, I get the following results:

And when running log on Terminal (I am using the Release version of NVMeFix v1.0.2):

log show --predicate 'process == "kernel" AND (eventMessage CONTAINS[c] "nvme")' --style syslog --source --last boot

Filtering the log data using "process == "kernel" AND composedMessage CONTAINS[c] "nvme""
Skipping info and debug messages, pass --info and/or --debug to include.
Timestamp                       (process)[PID]    
2020-04-19 13:47:34.162867+0200  localhost kernel[0]: (IONVMeFamily) <IONVMeFamily`IONVMeDebugAssert> AppleNVMe Assert failed: ( 0 != data )
2020-04-19 13:47:34.162872+0200  localhost kernel[0]: (IONVMeFamily) <IONVMeFamily`IONVMeDebugAssert> file: /BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-387.270.1/IONVMeController.cpp
2020-04-19 13:47:34.163951+0200  localhost kernel[0]: (IONVMeFamily) <IONVMeFamily`IONVMeController::CreateSubmissionQueue(unsigned short, unsigned char)> virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=0 entrysize=64
2020-04-19 13:47:34.163956+0200  localhost kernel[0]: (IONVMeFamily) <IONVMeFamily`IONVMeController::CreateSubmissionQueue(unsigned short, unsigned char)> virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=0 entrysize=64
2020-04-19 13:47:34.163967+0200  localhost kernel[0]: (IONVMeFamily) <IONVMeFamily`IONVMeController::CreateSubmissionQueue(unsigned short, unsigned char)> virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=1 entrysize=64
2020-04-19 13:47:34.163972+0200  localhost kernel[0]: (IONVMeFamily) <IONVMeFamily`IONVMeController::CreateSubmissionQueue(unsigned short, unsigned char)> virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=1 entrysize=64
2020-04-19 13:47:34.193885+0200  localhost kernel[0]: (apfs) <apfs`log_debug> er_state_obj_get_for_recovery:4214: No ER state object for volume Samsung NVMe - rolling is not happening, nothing to recover.
2020-04-19 13:47:34.194831+0200  localhost kernel[0]: (apfs) <apfs`log_debug> apfs_vfsop_mount:1536: mounted volume: Samsung NVMe

Based on the above and the fact that a) no trace of NVMePMProxy either and b) APST value is TRUE in IOReg, I am not sure what to believe.

With the above results, do we eventually need to keep NVMeFix.kext running and loaded? Thank you all...

System: Latest Mojave ; computer: Intel NUC8i7BEH2 ; disk Samsung 970 EVO 1TB Firmware: 2B2QEXE7 ; Clover r5107 ; DMAR dropped in Clover config; kexts loaded from /L/E/ with correct CsrActiveConfig ; renamed PXSX device to SSD0 via Clover device patch.

Edited April 19, 2020 by MacKonsti

roooooland · April 19, 2020

Hello! Thank you for all your hard work on these tools! I have installed an unsupported NVMe SSD (1TB Crucial P1) in my 2013 MacBook Air, and it has mostly been working, but I've been seeing kernel panics increasingly frequently and am hoping that the failure mode is one that NVMeFix can help with. This is on Mojave, 10.14.6. I have also installed a "matt card" with a patched EFI firmware image to work around hibernation issues, as detailed in this forum thread, but I don't think that's relevant to the behavior under macOS post-boot.

The panics all appear to be triggered at the same offset and have the same call stack:

*** Panic Report ***
panic(cpu 0 caller 0xffffff7f990fbb30): nvme: "Fatal error occurred. CSTS=0xffffffff US[1]=0x0 US[0]=0x4 VID/DID=0x2263c0a9
. FW Revision=P3CR013\n"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-387.270.1/IONVMeController.cpp:5334
Backtrace (CPU 0), Frame : Return Address
0xffffff91246bba10 : 0xffffff8015faf58d
0xffffff91246bba60 : 0xffffff80160eb145
0xffffff91246bbaa0 : 0xffffff80160dc87a
0xffffff91246bbb10 : 0xffffff8015f5c9d0
0xffffff91246bbb30 : 0xffffff8015faefa7
0xffffff91246bbc50 : 0xffffff8015faedf3
0xffffff91246bbcc0 : 0xffffff7f990fbb30
0xffffff91246bbe20 : 0xffffff801665de27
0xffffff91246bbe90 : 0xffffff801665dd49
0xffffff91246bbec0 : 0xffffff8015fee7a5
0xffffff91246bbf40 : 0xffffff8015fee345
0xffffff91246bbfa0 : 0xffffff8015f5c0ce
      Kernel Extensions in backtrace:
         com.apple.iokit.IONVMeFamily(2.1)[F4A8E2AC-36EF-3F94-A170-72D6D2B98E4A]@0xffffff7f990ee000->0xffffff7f9912dfff
            dependency: com.apple.driver.AppleMobileFileIntegrity(1.0.5)[2C186818-6D29-3A4B-BC2D-A810FC514FEA]@0xffffff7f96d8d000
            dependency: com.apple.iokit.IOPCIFamily(2.9)[F9449D20-0DF4-3A92-9532-7E6917DFF1A6]@0xffffff7f96895000
            dependency: com.apple.driver.AppleEFINVRAM(2.1)[C5CD122A-519C-3041-B0EC-469BD1C3CC4E]@0xffffff7f96f68000
            dependency: com.apple.iokit.IOStorageFamily(2.1)[F6337348-E16C-3E80-811B-E69D51B0517A]@0xffffff7f96b83000
            dependency: com.apple.iokit.IOReportFamily(47)[F9CF3B44-D9FA-3470-ABA5-79877BD0B46C]@0xffffff7f970b7000

BSD process name corresponding to current thread: kernel_task
Boot args: kext-dev-mode=1

What information can I gather to provide more information about what is triggering these crashes? I have the Lilu and NVMeFix kexts installed - they show up like this in the kextstat output:

  173    1 0xffffff7f83b3c000 0x7c000    0x7c000    as.vit9696.Lilu (1.4.3) AE058C68-7243-3FD6-9AA2-E62DBD41B72D <8 6 5 3 2 1>
  174    0 0xffffff7f83bb8000 0xf000     0xf000     org.acidanthera.NVMeFix (1.0.2) D03F5399-7DE5-31D3-9C42-6C29D9695341 <173 8 6 5 3 2 1>

My boot-args set up as follows:

$ sudo nvram boot-args
boot-args	kext-dev-mode=1 keepsyms=1 -nvmefdbg

I've also attached a screenshot of the IONVMeController's entry in IORegistryExplorer in case that is helpful. Please let me know if there's anything else I can provide that will help narrow down the root cause here. Thank you!

Toroop · April 19, 2020

7 hours ago, MacKonsti said:

Mind you, had no problems with my Samsung EVO 970 NVMe drive, before discovering this kext.

@MacKonsti I have the same drive as you, so maybe the kext does not support it

I also have no problems but I was hoping this would align battery life more towards what I get on Windows

MacKonsti · April 19, 2020

Thanks mate, that's why I wanted to ask the main developers as to how we can verify/check what our NVMe drives are (positively) affected. What should be the output of IOReg or value to look for? Only NVMePMProxy for example?

But I tend to agree with you @Toroop that EVO 970 (not the PRO) seems to work smoothly... at least on our motherboards.

Edited April 19, 2020 by MacKonsti

roooooland · April 20, 2020

Hit that crash again after enabling keepsyms=1 in my boot args, here's that call stack again with symbols:

*** Panic Report ***
panic(cpu 0 caller 0xffffff7f8f30cb30): nvme: "Fatal error occurred. CSTS=0xffffffff US[1]=0x0 US[0]=0x5 VID/DID=0x2263c0a9
. FW Revision=P3CR013\n"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-387.270.1/IONVMeController.cpp:5334
Backtrace (CPU 0), Frame : Return Address
0xffffff913d60ba10 : 0xffffff800c1af58d mach_kernel : _handle_debugger_trap + 0x47d
0xffffff913d60ba60 : 0xffffff800c2eb145 mach_kernel : _kdp_i386_trap + 0x155
0xffffff913d60baa0 : 0xffffff800c2dc87a mach_kernel : _kernel_trap + 0x50a
0xffffff913d60bb10 : 0xffffff800c15c9d0 mach_kernel : _return_from_trap + 0xe0
0xffffff913d60bb30 : 0xffffff800c1aefa7 mach_kernel : _panic_trap_to_debugger + 0x197
0xffffff913d60bc50 : 0xffffff800c1aedf3 mach_kernel : _panic + 0x63
0xffffff913d60bcc0 : 0xffffff7f8f30cb30 com.apple.iokit.IONVMeFamily : __ZN16IONVMeController13FatalHandlingEv + 0x10e
0xffffff913d60be20 : 0xffffff800c85de27 mach_kernel : __ZN18IOTimerEventSource15timeoutSignaledEPvS0_ + 0x87
0xffffff913d60be90 : 0xffffff800c85dd49 mach_kernel : __ZN18IOTimerEventSource17timeoutAndReleaseEPvS0_ + 0x99
0xffffff913d60bec0 : 0xffffff800c1ee7a5 mach_kernel : _thread_call_delayed_timer + 0xef5
0xffffff913d60bf40 : 0xffffff800c1ee345 mach_kernel : _thread_call_delayed_timer + 0xa95
0xffffff913d60bfa0 : 0xffffff800c15c0ce mach_kernel : _call_continuation + 0x2e
      Kernel Extensions in backtrace:
         com.apple.iokit.IONVMeFamily(2.1)[F4A8E2AC-36EF-3F94-A170-72D6D2B98E4A]@0xffffff7f8f2ff000->0xffffff7f8f33efff
            dependency: com.apple.driver.AppleMobileFileIntegrity(1.0.5)[2C186818-6D29-3A4B-BC2D-A810FC514FEA]@0xffffff7f8cf8d000
            dependency: com.apple.iokit.IOPCIFamily(2.9)[F9449D20-0DF4-3A92-9532-7E6917DFF1A6]@0xffffff7f8ca95000
            dependency: com.apple.driver.AppleEFINVRAM(2.1)[C5CD122A-519C-3041-B0EC-469BD1C3CC4E]@0xffffff7f8d168000
            dependency: com.apple.iokit.IOStorageFamily(2.1)[F6337348-E16C-3E80-811B-E69D51B0517A]@0xffffff7f8cd83000
            dependency: com.apple.iokit.IOReportFamily(47)[F9CF3B44-D9FA-3470-ABA5-79877BD0B46C]@0xffffff7f8d2b7000

Any ideas on whether NVMeFix could help here?

978b029efa981d618699b09868 · April 20, 2020

1 hour ago, roooooland said:

Hit that crash again after enabling keepsyms=1 in my boot args, here's that call stack again with symbols:


*** Panic Report ***
panic(cpu 0 caller 0xffffff7f8f30cb30): nvme: "Fatal error occurred. CSTS=0xffffffff US[1]=0x0 US[0]=0x5 VID/DID=0x2263c0a9
. FW Revision=P3CR013\n"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-387.270.1/IONVMeController.cpp:5334
Backtrace (CPU 0), Frame : Return Address
0xffffff913d60ba10 : 0xffffff800c1af58d mach_kernel : _handle_debugger_trap + 0x47d
0xffffff913d60ba60 : 0xffffff800c2eb145 mach_kernel : _kdp_i386_trap + 0x155
0xffffff913d60baa0 : 0xffffff800c2dc87a mach_kernel : _kernel_trap + 0x50a
0xffffff913d60bb10 : 0xffffff800c15c9d0 mach_kernel : _return_from_trap + 0xe0
0xffffff913d60bb30 : 0xffffff800c1aefa7 mach_kernel : _panic_trap_to_debugger + 0x197
0xffffff913d60bc50 : 0xffffff800c1aedf3 mach_kernel : _panic + 0x63
0xffffff913d60bcc0 : 0xffffff7f8f30cb30 com.apple.iokit.IONVMeFamily : __ZN16IONVMeController13FatalHandlingEv + 0x10e
0xffffff913d60be20 : 0xffffff800c85de27 mach_kernel : __ZN18IOTimerEventSource15timeoutSignaledEPvS0_ + 0x87
0xffffff913d60be90 : 0xffffff800c85dd49 mach_kernel : __ZN18IOTimerEventSource17timeoutAndReleaseEPvS0_ + 0x99
0xffffff913d60bec0 : 0xffffff800c1ee7a5 mach_kernel : _thread_call_delayed_timer + 0xef5
0xffffff913d60bf40 : 0xffffff800c1ee345 mach_kernel : _thread_call_delayed_timer + 0xa95
0xffffff913d60bfa0 : 0xffffff800c15c0ce mach_kernel : _call_continuation + 0x2e
      Kernel Extensions in backtrace:
         com.apple.iokit.IONVMeFamily(2.1)[F4A8E2AC-36EF-3F94-A170-72D6D2B98E4A]@0xffffff7f8f2ff000->0xffffff7f8f33efff
            dependency: com.apple.driver.AppleMobileFileIntegrity(1.0.5)[2C186818-6D29-3A4B-BC2D-A810FC514FEA]@0xffffff7f8cf8d000
            dependency: com.apple.iokit.IOPCIFamily(2.9)[F9449D20-0DF4-3A92-9532-7E6917DFF1A6]@0xffffff7f8ca95000
            dependency: com.apple.driver.AppleEFINVRAM(2.1)[C5CD122A-519C-3041-B0EC-469BD1C3CC4E]@0xffffff7f8d168000
            dependency: com.apple.iokit.IOStorageFamily(2.1)[F6337348-E16C-3E80-811B-E69D51B0517A]@0xffffff7f8cd83000
            dependency: com.apple.iokit.IOReportFamily(47)[F9CF3B44-D9FA-3470-ABA5-79877BD0B46C]@0xffffff7f8d2b7000

Any ideas on whether NVMeFix could help here?

Unfortunately, it's a very generic timeout error that has almost no relevant information. You may want to check if there are any kernel log messages just before the crash about the timed out request, but it is still unlikely we will be able to fix this, as only a few PM981-specific crashes could be reproduced on our machines.

In any case, compatibility with third-party NVMe drives is also needed for genuine Apple computers, so I would expect improved support in newer OS versions. The kext from 10.15.4 already hints at that.

18 hours ago, MacKonsti said:

Thanks mate, that's why I wanted to ask the main developers as to how we can verify/check what our NVMe drives are (positively) affected. What should be the output of IOReg or value to look for? Only NVMePMProxy for example?

But I tend to agree with you @Toroop that EVO 970 (not the PRO) seems to work smoothly... at least on our motherboards.

If you use DEBUG build, kernel log should contain APST enable status and table (see README). In practice you need to measure battery life yourself to tell if it helps.

On 4/18/2020 at 12:25 AM, davidm71 said:

This kext has me curious. If I was to use Clover lets say to boot up my Laptop running Windows with NVMEFix would my laptop power consumption go down and get longer battery life??

Thanks

No, because it is only for Mac. On Windows you may want to use StorNVMe.sys (Microsoft NVMe driver) to get improved power management (ships with the OS, but there also exist vendor-specific drivers). StorNVMe documentation describes power management here: https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/power-management-for-storage-hardware-devices-nvme

Edited April 20, 2020 by 978b029efa981d618699b09868

MacKonsti · April 20, 2020

1 minute ago, 978b029efa981d618699b09868 said:

If you use DEBUG build, kernel log should contain APST enable status and table (see README). In practice you need to measure battery life yourself to tell if it helps.

Thanks for your reply. I am on a desktop machine, the Intel NUC is a desktop system despite using mobile components (I think). Should desktop users stop using NVMeFix.kext altogether you think? Thanks

978b029efa981d618699b09868 · April 20, 2020

3 minutes ago, MacKonsti said:

Thanks for your reply. I am on a desktop machine, the Intel NUC is a desktop system despite using mobile components (I think). Should desktop users stop using NVMeFix.kext altogether you think? Thanks

It's up to you. If your machine works well enough without it, then there's no reason to use it.

roooooland · April 20, 2020

Thanks 978b029efa981d618699b09868. The only intriguing thing in the log around the latest crash is:

$ sudo log show --debug --info --start '2020-04-20 16:20:00' | grep -i nvme
2020-04-20 16:27:59.816933-0400 0x10b      Default     0x0                  0      0    kernel: (IONVMeFamily) AppleNVMe Assert failed: ( 0 != data )
2020-04-20 16:27:59.816935-0400 0x10b      Default     0x0                  0      0    kernel: (IONVMeFamily) ReleaseIDNode
2020-04-20 16:27:59.816937-0400 0x10b      Default     0x0                  0      0    kernel: (IONVMeFamily) file: /BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-387.270.1/IONVMeController.cpp
2020-04-20 16:27:59.816940-0400 0x10b      Default     0x0                  0      0    kernel: (IONVMeFamily) line: 5492
2020-04-20 16:27:59.816941-0400 0x10b      Default     0x0                  0      0    kernel: (IONVMeFamily)
2020-04-20 16:27:59.818136-0400 0x10b      Default     0x0                  0      0    kernel: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=0 entrysize=64
2020-04-20 16:27:59.818142-0400 0x10b      Default     0x0                  0      0    kernel: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=0 entrysize=64
2020-04-20 16:27:59.818151-0400 0x10b      Default     0x0                  0      0    kernel: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=1 entrysize=64
2020-04-20 16:27:59.818157-0400 0x10b      Default     0x0                  0      0    kernel: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=1 entrysize=64

I imagine that doesn't help narrow it down much...I'll try upgrading to 10.15 and see if that helps at all.

Toroop · April 20, 2020

4 hours ago, 978b029efa981d618699b09868 said:

If you use DEBUG build, kernel log should contain APST enable status and table (see README). In practice you need to measure battery life yourself to tell if it helps.

On 4/18/2020 at 8:25 AM, davidm71 said:

@978b029efa981d618699b09868 Is there a reason why I do not see any debug message while using Debug build and boot arg?

Please see my post above. Thanks.

Edited April 20, 2020 by Toroop

978b029efa981d618699b09868 · April 24, 2020

On 4/20/2020 at 11:57 PM, Toroop said:

@978b029efa981d618699b09868 Is there a reason why I do not see any debug message while using Debug build and boot arg?

Please see my post above. Thanks.

No idea, either you are doing something wrong, or os_log swallows the lines. In the latter case https://github.com/acidanthera/DebugEnhancer might help

Toroop · April 26, 2020

On 4/25/2020 at 2:21 AM, 978b029efa981d618699b09868 said:

No idea, either you are doing something wrong, or os_log swallows the lines. In the latter case https://github.com/acidanthera/DebugEnhancer might help

I did try installing the debug enhancer, and re-installing the debug version of NVMEFix but no luck...No NVMePMProxy

and still only this in the logs

2020-04-26 09:56:43.456769+1000  localhost kernel[0]: (Lilu) <Lilu`lilu_os_log> NVMeFix      init: @ (DBG) NVMeFix bootstrap DBG-102-2020-04-06

2020-04-26 09:56:43.456774+1000  localhost kernel[0]: (Lilu) <Lilu`lilu_os_log> NVMeFix     nvmef: @ (DBG) Registered for matching notifications

I guess it just isn't loading properly in my system.

tmbt · May 19, 2020

Hi,

I'm using this kext but I'm facing a problem : when the laptop wake from sleep the kext doesn't work anymore.

Here is what I did to isolate the problem.

Installed a clean version of Catalina and without the NVMEFix I read around 1.8W in Intel Power Gadget.

If I run this kext after a clean boot I read around 0.7W (which is similar to when I had a normal SSD on my laptop)

But when the laptop enter in sleep mode and then wake I read 1.8W again as if the kext is not active anymore.

I also tried to clean install HighSierra and did the same test. Without this kext I got the same 1.8W. After installed IONVMEFamily kext from Mojave and self compiled NVMEFix to allow to run on HighSierra Kernel I read 0.7W after a boot. But as soon as I wake from sleep I got the same 1.8W reading...

I'm using the debug version and booting using the -nvmefdbg but the only entries I can see in the log are :

Mattias-MacBook-Pro:/ mattia$ log show --style syslog --last boot --predicate  "process == \"kernel\" and eventMessage CONTAINS[c] \"nvme\""
Filtering the log data using "process == "kernel" AND eventMessage CONTAINS[c] "nvme""
Skipping info and debug messages, pass --info and/or --debug to include.
Timestamp                       (process)[PID]    
2020-05-19 14:04:16.693917+0200  localhost kernel[0]: (IONVMeFamily) AppleNVMe Assert failed: ( 0 != data )
2020-05-19 14:04:16.693922+0200  localhost kernel[0]: (IONVMeFamily) file: /BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-387.270.1/IONVMeController.cpp
2020-05-19 14:04:16.695104+0200  localhost kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=0 entrysize=64
2020-05-19 14:04:16.695109+0200  localhost kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=0 entrysize=64
2020-05-19 14:04:16.695118+0200  localhost kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=1 entrysize=64
2020-05-19 14:04:16.695122+0200  localhost kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=1 entrysize=64

Could someone give me some advices on how to fix this problem ?

EDIT : I've also installed Mojave on my Hack using the same EFI partition and got the same behavior (around 0.7W at first boot and around 1.8W after the wake from sleep) BUT i've noticed a strange thing (not sure if it is happening only now or was happening also before). IF i force the laptop to sleep closing the lid (i wait until i hear the cpu fan stop rotating and the power led blink as when the laptop is in sleep mode) then when i wake it opening the lid i still got 0.7W so the NVMEFix is still working. BUT if i put the laptop to sleep clicking on the Apple->Sleep or i wait for the laptop to go to sleep by itself (after 2 minutes as i've configured in system preference) then when it wakes NVMEFix is not working anymore and i got 1.8W.

Is this helpfull in some way ?

Thanks

Mattia

Edited May 19, 2020 by tmbt
Update

978b029efa981d618699b09868 · May 21, 2020

On 5/19/2020 at 2:17 PM, tmbt said:
Hi,

I'm using this kext but I'm facing a problem : when the laptop wake from sleep the kext doesn't work anymore.

Here is what I did to isolate the problem.

Installed a clean version of Catalina and without the NVMEFix I read around 1.8W in Intel Power Gadget.

If I run this kext after a clean boot I read around 0.7W (which is similar to when I had a normal SSD on my laptop)

But when the laptop enter in sleep mode and then wake I read 1.8W again as if the kext is not active anymore.

I also tried to clean install HighSierra and did the same test. Without this kext I got the same 1.8W. After installed IONVMEFamily kext from Mojave and self compiled NVMEFix to allow to run on HighSierra Kernel I read 0.7W after a boot. But as soon as I wake from sleep I got the same 1.8W reading...

I'm using the debug version and booting using the -nvmefdbg but the only entries I can see in the log are :
Mattias-MacBook-Pro:/ mattia$ log show --style syslog --last boot --predicate  "process == \"kernel\" and eventMessage CONTAINS[c] \"nvme\""
Filtering the log data using "process == "kernel" AND eventMessage CONTAINS[c] "nvme""
Skipping info and debug messages, pass --info and/or --debug to include.
Timestamp                       (process)[PID]    
2020-05-19 14:04:16.693917+0200  localhost kernel[0]: (IONVMeFamily) AppleNVMe Assert failed: ( 0 != data )
2020-05-19 14:04:16.693922+0200  localhost kernel[0]: (IONVMeFamily) file: /BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-387.270.1/IONVMeController.cpp
2020-05-19 14:04:16.695104+0200  localhost kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=0 entrysize=64
2020-05-19 14:04:16.695109+0200  localhost kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=0 entrysize=64
2020-05-19 14:04:16.695118+0200  localhost kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=1 entrysize=64
2020-05-19 14:04:16.695122+0200  localhost kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=1 entrysize=64
Could someone give me some advices on how to fix this problem ?

EDIT : I've also installed Mojave on my Hack using the same EFI partition and got the same behavior (around 0.7W at first boot and around 1.8W after the wake from sleep) BUT i've noticed a strange thing (not sure if it is happening only now or was happening also before). IF i force the laptop to sleep closing the lid (i wait until i hear the cpu fan stop rotating and the power led blink as when the laptop is in sleep mode) then when i wake it opening the lid i still got 0.7W so the NVMEFix is still working. BUT if i put the laptop to sleep clicking on the Apple->Sleep or i wait for the laptop to go to sleep by itself (after 2 minutes as i've configured in system preference) then when it wakes NVMEFix is not working anymore and i got 1.8W.

Is this helpfull in some way ?

Thanks

Mattia

Thanks for reporting, it looks like 1.0.2 broke it by disabling NVMePMProxy, which was previously used to reenable APST. Can you please test if the problem occurs with 1.0.1?

tmbt · May 22, 2020

On 5/21/2020 at 3:21 AM, 978b029efa981d618699b09868 said:

Thanks for reporting, it looks like 1.0.2 broke it by disabling NVMePMProxy, which was previously used to reenable APST. Can you please test if the problem occurs with 1.0.1?

Hi,

i've done more tests. It seems that in 1.0.2 version i don't have NVMePMProxy anywhere in ioREG. If i use 1.0.1 or 1.0 version i've that key in ioREG but after sleep Intel Power Gadget still read more power consumption than before the sleep.

What could i do more ?

978b029efa981d618699b09868 · May 22, 2020

26 minutes ago, tmbt said:

Hi,

i've done more tests. It seems that in 1.0.2 version i don't have NVMePMProxy anywhere in ioREG. If i use 1.0.1 or 1.0 version i've that key in ioREG but after sleep Intel Power Gadget still read more power consumption than before the sleep.

What could i do more ?

Are you sure the increased consumption is related to SSD and not e.g. graphics?

What does the debug log say (for 1.0.1)? At start it should say "Registering power change interest", and whenever it enables APST (that includes after waking), "Configuring APST"

tmbt · May 22, 2020

Here is what i got from 1.0.1 in terminal :

Mattias-MacBook-Pro:~ mattia$ log show --style syslog --last boot --predicate  "process == \"kernel\" and eventMessage CONTAINS[c] \"nvme\""
Filtering the log data using "process == "kernel" AND composedMessage CONTAINS[c] "nvme""
Skipping info and debug messages, pass --info and/or --debug to include.
Timestamp                       (process)[PID]    
2020-05-22 23:00:00.768217+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:00.775692+0200  localhost kernel[0]: (IONVMeFamily) AppleNVMe Assert failed: ( 0 != data )
2020-05-22 23:00:00.775699+0200  localhost kernel[0]: (IONVMeFamily) file: /BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-387.270.1/IONVMeController.cpp
2020-05-22 23:00:00.776100+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:00.777077+0200  localhost kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=0 entrysize=64
2020-05-22 23:00:00.777083+0200  localhost kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=0 entrysize=64
2020-05-22 23:00:00.777668+0200  localhost kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=1 entrysize=64
2020-05-22 23:00:00.777674+0200  localhost kernel[0]: (IONVMeFamily) virtual IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2784:SQ index=1 entrysize=64
2020-05-22 23:00:00.839630+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:00.849132+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:00.854123+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:00.854521+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:00.855145+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:00.889561+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:04.544131+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:04.590963+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:04.593725+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:04.731494+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:04.732725+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:04.782933+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:05.035721+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:05.293303+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:05.408251+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:05.779398+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:05.790284+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:05.790814+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:06.402274+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:06.518829+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:09.019979+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:13.063902+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:14.024124+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"
2020-05-22 23:00:14.032251+0200  localhost kernel[0]: Couldn't alloc class "NVMeFix"

Even using the Debug version with the boot argument still the same output. But if i check under ioREG i can see the NVMEPmProxy (as picture attached).

I don't have an normal SSD m.2 type (i've sold it some weeks ago) so i can't replace my actual NVME drive to check power consumption ... but i'm quite confident that with the normal SSD i wasn't having this increase power consumption problem after sleep.

I will attach 2 screenshot from Intel power Gadget too :

Before

After

Edited May 22, 2020 by tmbt
Update

978b029efa981d618699b09868 · May 22, 2020

I will fix re-enabling APST after sleep anyway, because it is a legitimate bug, but if older version of the kext does not help you, then this won't help you either.

Debug log should be working, without it I can't tell what is happening after waking.

NVMeFix

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites