I originally wrote this "How To" in the following topic "Workstation 8/9, Player 4/5 and Fusion 4/5 Mac OS X Unlocker" started by Donk, see link below, since there are now some 500+ replies in the topic, I thought it was time to extract the post and start a new topic.
Although a lot of alternative methods are available, the following "How To" based on the work by Donk, Zenith432 and others on "InsanelyMac" is a proven method for installing a "Vanilla / Retail" version of OS X 10.6 (Snow Leopard), 10.7 (Lion), 10.8 (Mountain Lion), 10.9 (Mavericks), 10.10 (Yosemite) and 10.11 (El Capitan) under VMware Workstation 10/11, Player 6/7, Workstation Pro/Player 12 and ESXi 5/6, using physical media (Snow Leopard) or an email with a content code for the Mac App Store (Lion and Mountain Lion) purchased from the Apple Store or media created from the App Store download. If you have access to an existing OS X machine physical or virtual you can download OS X 10.9 (Mavericks), 10.10 (Yosemite) or 10.11 (El Capitan) for free. It is important for the method detailed below to work, you must be running a relatively recent Intel processor with VT-x (Hardware Virtualisation) support which has been enabled in BIOS.
Install the "Unlocker" following the included readme.txt, version 1.3.x for Workstation 10, Player 6 or ESXi 5 and "Unlocker" version 2.0.x for Workstation 11, Player 7, Workstation Pro/Player 12 and ESXi 6 created by Donk based on work by Donk, Zenith432 and Sam B, see the links below, to enable support for OS X. If you have access to an OS X machine, Redeem your purchase and download OS X from the Mac App Store under Purchases or use Updates to get OS X 10.9 (Mavericks), 10.10 (Yosemite) or 10.11 (El Capitan) once downloaded on your OS X machine extract the InstallESD.dmg from your App Store download of "Install OS X Lion" or "Install OS X Mountain Lion" in the Applications folder, right click the icon and select "Show Package Contents" open folder "SharedSupport" and copy the InstallESD.dmg to your desktop.
Link to the "Unlocker" download to support Workstation 10.x, Player 6.x, Fusion 6.x and ESXi 5.x:
Link to the "Unlocker" download to support Workstation 11.x, Workstation Pro 12.x, Player 7.x, Workstation Player 12.x, Fusion 7.x, Fusion 8.x and ESXi 6.x:
Please note this media creation method will NOT work for OS X 10.9 (Mavericks), OS X 10.10 (Yosemite) or OS X 10.11 (El Capitan) an apple script is required to create a bootable DMG file from the "Install OS X Mavericks" or "Install OS X Yosemite" app downloaded from the Mac App Store, see the second post in this thread for details, link below:
If you do not have access to an OS X machine, then purchase the OS X Snow Leopard DVD from the Apple Store ($19.99 US or £14.00 UK), you can use the DVD directly but it is easier to covert the DVD to an ISO.
In order for you to be able to use the .dmg created for Lion or Mountain Lion as install media in Workstation 10/11 or Player 6/7 (Workstation Pro 12, Workstation Player 12, Fusion 6, 7 and 8, ESXi 5.1, 5.1 U1, 5.5, 5.5 U1, 5.5 U2, 5.5 U3, ESXi 6.0 and ESXi 6.0 U1x support .dmg files directly) the .dmg will have to be converted to an .iso, this can be done at the command line in OS X see below or in Windows using a third party product like ImgBurn ( http://www.imgburn.com/ ) or dmg2img ( http://vu1tur.eu.org/tools/ ) .
In an OS X terminal session run the following:
hdiutil convert ./InstallESD.dmg -format UDTO -o InstallESD
mv InstallESD.cdr InstallESD.iso
The conversion process will take some time, so apply the cup of coffee rule. The conversion process actually creates a .cdr file which we rename with the mv command to a .iso which can be mounted as installable media in Workstation 10/11 and Player 6/7.
If the "Unlocker" has been installed correctly you should now be able to create a new Apple Mac OS X Guest operating system in Workstation 10/11, Workstation Pro 12, Player 6/7, Workstation Player 12 and ESXi 5/6 and select the InstallESD.iso you have created as the "Installer disc image file (iso):" and select Apple Mac OS X as the "Guest operating system" and Mac OS X Server 10.6 64-bit, Mac OS X 10.7 64-bit, Mac OS X 10.8 or Mac OS X 10.9 as the "Version". Mac OS X 10.9 can be installed from the .dmg created via the script detailed in the second post of this topic as can Mac OS X 10.10 which is only available as an option in Workstation 11, Workstation Pro 12, Player 7, Workstation Player 12 and ESXi 6, although Mac OS X 10.10 or 10.11 can be installed in Workstation 10, Player 6 or ESXi 5 using Mac OS X 10.9 as the "Version". Please note Mac OS X 10.11 is only available as an option in Workstation Pro 12, Workstation Player 12 and ESXi 6.
Next choose a name and location for the virtual machine, then specify the "Maximum disk size (GB):" (I always select a minimum of 64 GB) and whether to "Store virtual disk as a single file" or "Split virtual disk into multiple files", (I select single file for ease of portability, but in theory selecting multiple files should offer better performance), however the default settings work fine.
You now have the option to either Finish and create the Virtual Machine or "Customize Hardware...", I recommend adding a second core under the "Processors" section (although you may get the "Your Mac OS guest might run unreliably with more than one virtual core." warning message) and change the "Network Adapter" to "Bridged" (in theory these are optional, the default settings should will work fine, but experience would suggest using the recommended settings) select Close and Finish.
It has been noted, thanks to cb4, that sometimes you get no keyboard input in the virtual machine, setting the "Enhanced virtual keyboard" from "Off" to "Use if available (recommended)" resolves the issue, and does not appear to have any issues when set on for a working OS X Virtual Machine.
To disable the VMware multiple vCPU message add the following line to the VMX configuration for the guest whilst it is shutdown.
isolation.tools.bug328986.disable = "TRUE"
Please note this will only suppress the message, and the guest may be unstable with more than 1 vCPU, however since it is a very long time since real Mac's were not at least dual core, I always add at least two vCPU's, and have not experienced any instability. Thanks to Donk for the fix, for additional information head over to the website "Shine The Braille Toad".
If you create the virtual machine using version 11 or later hardware, the default for Workstation 11, the virtual machine will not start and either hang or create a CPU core dump error message. Donk has confirmed two options to work around the issue:
1. Change the virtual machine to version 10 hardware.
2. Edit the configuration file (.vmx) and add the following line: smc.version = "0"
Power on the virtual machine and the OS X installation should proceed to the Welcome screen, select your main language, and in the OS X Utilities screen select Disk Utility. In the left panel you should now see 68.72 GB VMware Virtual SATA Hard Drive Media, or similar depending on the hard drive size you set when creating the virtual machine, select the drive, and select the Partition tab. Next change the Partition Layout: from "Current" to "1 Partition", you can select more than one partition but I generally keep it simple, give the Partition a meaningful name, i.e. OS X, and click on the Apply button (the default format "Mac OS Extended (Journaled)" should be selected, and under the "Options..." button "GUID Partition Table" should be selected), you will next get an "Are you sure" warning, click on the Partition button to complete the creation of OS X formatted partition, you should see it appear below the VMware Virtual SATA Hard Drive Media.
The DIsk Utility presentation has changed in El Capitan, having selected the VMware Virtual SATA Hard Drive Media as detailed above select the Erase button, the Partition button will be grayed out, and give the drive a meaningful Name: i.e. OS X and select "OS X Extended (Journaled)" as the Format: finally select "GUID Partition Map" as the Scheme and press the blue Erase button to create the drive ready for use by OS X.
Now exit Disk Utility and select Reinstall or Install OS X, click Continue at the Install OS X screen and agree to the Apple EULA, at the Select the disk where you want to install OS X, you should have a nice yellow OS X disk in the middle of the screen, select the disk and click on Install, the installation should then proceed with a blue progress bar and "About 9 minutes remaining", following a re-boot OS X will continue "Installing OS X on the disk "OS X ..." again with a blue progress bar and "Time remaining: About 19 minutes", if the screen goes white with only a black mouse pointer, just click to bring back the install OS X screen.
Following a further re-boot you should finally be at the Welcome screen with a map of the world and the option to select your language, followed by "Select Your Keyboard" to get the best support for a PC keyboard select the Show All and choose the relevant "Your Language - PC" version, i.e. British - PC.
The next couple of screens may or may not appear depending on your network configuration, "How Do You Connect?", "Your Internet Connection" probably best to leave "TCP/IP Connection Type" Using DHCP initially.
The next screen to appear is the "Transfer Information to This Mac", "Sign in with Your Apple ID" if you choose Don't sign in, you will get a splash screen with the option to "Skip", "Terms and Conditions", "Create Your Computer Account", "Select You Time Zone", "Diagnostics & Usage" followed by a "Setting up your Mac" and a thank you from Apple you should then be at the OS X desktop.
This will create a vanilla install of OS X which can and should be patched via "Software Update..." as a matter of priority.
It appears that like ESXi 5.5 and ESXi 6, Workstation Pro/Player 12 using Unlocker 2.0.7 or later, Install VMware Tools... under the VM tab now appears to work, if not use the method as for previous versions of Workstation and Player detailed below:
To install "VMware Tools" go to the Virtual Machine Settings for the OS X VM and select CD/DVD (IDE) and under "Connection" click on "Use ISO image file:". It is quite likely your InstallESD.iso will still be selected, so just "Browse..." to the location of the darwin.iso and select "Open". The darwin.iso is not present in a standard install of either Workstation 10/11, Workstation Pro 12, Player 6/7 or Workstation Player 12, it is added as part of the "Unlocker" install and is located in the VMware Workstation folder under "Program Files (x86)\VMware" or "Program Files\VMware" for a 32-bit host in a default install, make sure the "Device status" has "Connected" ticked generally "Connect at power on" is ticked by default.
You may get an error when switching mounted .iso's in the Virtual Machine Settings "The guest operating system has locked the CD-ROM door..." it is OK to answer "Yes" to the question "Disconnect anyway and override the lock?".
In your OS X desktop a VMware Tools icon should appear in the upper right corner, double click and select "Install VMware Tools", it is recommended to install VMware Tools as it offers an enhanced user experience when working with the Virtual Machine. Please note VMware Tools cannot be installed from the menu in Workstation 10/11 or Player 6/7 to an OS X Virtual Machine (unlike Fusion, ESXi 5.x snd ESXi 6.x and now Workstation Pro 12 and Workstation Player 12 which support VMware Tools installation from the menu).
The latest VMware Tools from Fusion 7/8 now provide the best graphics support for OS X virtual machines, and Zenith432 has stopped development of the VMsvga2 graphics drivers. The latest darwin.iso is available to download at the VMware CDS Repository, see link below:
Once you are up and running with your virtual OS X machine you need to do a couple of housekeeping tasks to ensure your virtual machine does not freeze. Firstly go to "System Preferences" and select "Energy Saver" and set "Computer sleep:" and "Display sleep:" to Never and also un-tick "Put hard disks to sleep when possible", "Wake for Ethernet network access" and "Allow power button to put the computer to sleep". In "Desktop & Screen Saver" set "Screen Saver" to "Start after:" Never.
One final note, occasionally you can come across an unsupported CPU error, this should be rare if you have recently downloaded your OS X media from the App Store as the App Store version is updated when a new combo patch is released. Alternatively if your CPU is one of the very latest Intel processors not yet supported by OS X, Apple check CPUID's when installing OS X, and will not install on an unsupported processor platform.
(A prime example was when Intel introduced the Core i3, i5 and i7 and Xeon 5600 series processors, Snow Leopard 10.6.4 was the lowest version of OS X to support the new processors, unfortunately the highest available "vanilla" media was and still is 10.6.3, without the CPUID mask it was impossible to install OS X Snow Leopard. As stated earlier in this post, Apple starting with OS X 10.7 Lion deliver the latest slipstreamed media via the App Store.)
In either case add the following line to your OS X virtual machine configuration file (.vmx):
cpuid.1.eax = "0000:0000:0000:0001:0000:0110:1010:0101" or cpuid.1.eax = "----:----:----:0010:----:----:1010:0111"
The first was based on work by myself and Donk, the second a recommendation by jmattson of VMware, see links below: