I originally wrote this short "How To" in the following topic "Workstation 8/9, Player 4/5 and Fusion 4/5 Mac OS X Unlocker" started by Donk, see link below, since there are now some 500+ replies in the topic, I thought it was time to extract the post and start a new topic.
Although a lot of alternative methods are available, the following "How To" based on the work by Donk, Zenith432 and others on "InsanelyMac" is a proven method for installing a "Vanilla / Retail" version of OS X (10.6) Snow Leopard, (10.7) Lion, (10.8) Mountain Lion and (10.9) Mavericks under VMware Workstation 9.0.2, Workstation 10.0.2, Player 5.0.2 or Player 6.0.2, using physical media (Snow Leopard) or an email with a content code for the Mac App Store (Lion and Mountain Lion) purchased from the Apple Store or media created from the App Store download. If you have access to an existing OS X machine physical or virtual you can download OS X (10.9) Mavericks for free. It is important for the method detailed below to work, you must be running a relatively recent Intel processor with VT-x (Hardware Virtualisation) support which has been enabled in BIOS.
Install the "Unlocker" version 1.3.0 created by Donk based on work by Donk and Zenith432, see the link above, to enable support for OS X in either VMware Workstation 9/10 or Player 5/6. If you have access to an OS X machine, Redeem your purchase and download OS X from the Mac App Store under Purchases or use Updates to get OS X (10.9) Mavericks once downloaded on your OS X machine extract the InstallESD.dmg from your App Store download of "Install OS X Lion" or "Install OS X Mountain Lion" in the Applications folder, right click the icon and select "Show Package Contents" open folder "SharedSupport" and copy the InstallESD.dmg to your desktop.
Please note this method will NOT work for OS X 10.9 (Mavericks) a script extracted from Fusion 6 is required to create a bootable DMG file from the "Install OS X Mavericks" app downloaded from the Mac App Store, see the second post in this thread for details.
If you do not have access to an OS X machine, then purchase the OS X (10.6) Snow Leopard DVD from the Apple Store ($19.99 US or £14.00 UK), you can use the DVD directly but it is easier to covert the DVD to an ISO.
In order for you to be able to use this as install media in Workstation 9/10 or Player 5/6 (Fusion 5/6, ESXi 5.1, 5.1 U1, 5.5 and 5.5 U1 support .dmg files directly) the .dmg will have to be converted to an .iso, this can be done at the command line in OS X see below or in Windows using a third party product like ImgBurn ( http://www.imgburn.com/ ) or dmg2img ( http://vu1tur.eu.org/tools/ ) .
In an OS X terminal session run the following:
hdiutil convert ./InstallESD.dmg -format UDTO -o InstallESD
mv InstallESD.cdr InstallESD.iso
The conversion process will take some time, so apply the cup of coffee rule. The conversion process actually creates a .cdr file which we rename with the mv command to a .iso which we can mount as installable media in Workstation 9/10 and Player 5/6.
If the "Unlocker" has been installed correctly you should now be able to create a new Apple Mac OS X Guest operating system in Workstation 9/10 or Player 5/6 and select the InstallESD.iso you have created as the "Installer disc image file (iso):" and select Apple Mac OS X as the "Guest operating system" and Mac OS X Server 10.6 64-bit, Mac OS X 10.7 64-bit, Mac OS X 10.8 or Mac OS X 10.9 (only available in Workstation 10 and Player 6) as the "Version".
Next choose a name and location for the virtual machine, then specify the "Maximum disk size (GB):" (I always select a minimum of 64 GB) and whether to "Store virtual disk as a single file" or "Split virtual disk into multiple files", (I select single file for ease of portability, but in theory selecting multiple files should offer better performance), however the default settings work fine.
You now have the option to either Finish and create the Virtual Machine or "Customize Hardware...", I recommend adding a second core under the "Processors" section and change the "Network Adapter" to "Bridged" (in theory these are optional, the default settings should will work fine, but experience would suggest using the recommended settings) select Close and Finish.
Power on this virtual machine and the OS X installation should proceed to the Welcome screen, select your main language, and in the OS X Utilities screen select Disk Utility. In the left panel you should now see 68.72 GB VMware Virtual SATA Hard Drive Media, or similar depending on the hard drive size you set when creating the virtual machine, select the drive, and select the Partition tab. Next change the Partition Layout: from "Current" to "1 Partition", you can select more than one partition but I generally keep it simple, give the Partition a meaningful name, i.e. OS X, and click on the Apply button (the default format "Mac OS Extended (Journaled)" should be selected, and under the "Options..." button "GUID Partition Table" should be selected), you will next get an "Are you sure" warning, click on the Partition button to complete the creation of OS X formatted partition, you should see it appear below the VMware Virtual SATA Hard Drive Media.
Now exit Disk Utility and select Reinstall OS X, click Continue at the Install OS X screen and agree to the Apple EULA, at the Select the disk where you want to install OS X, you should have a nice yellow OS X disk in the middle of the screen, select the disk and click on Install, the installation should then proceed with a blue progress bar and "About 9 minutes remaining", following a re-boot OS X will continue "Installing OS X on the disk "OS X ..." again with a blue progress bar and "Time remaining: About 19 minutes", if the screen goes white with only a black mouse pointer, just click to bring back the install OS X screen. Following a further re-boot you should finally be at the Welcome screen with a map of the world and the option to select your language, followed by "Select Your Keyboard", "How Do You Connect?", "Your Internet Connection" probably best to leave "TCP/IP Connection Type" Using DHCP initially, "Transfer Information to This Mac", "Terms and Conditions", "Create Your Computer Account", "Select You Time Zone", following a Thank You from Apple you should be at the OS X desktop.
This will create a vanilla install of OS X which can and should be patched via "Software Update..." as a matter of priority.
To install "VMware Tools" go to the Virtual Machine Settings for the OS X VM and select CD/DVD (IDE) and under "Connection" click on "Use ISO image file:". It is quite likely your InstallESD.iso will still be selected, so just "Browse..." to the location of the darwin.iso and select "Open". The darwin.iso is not present in a standard install of either Workstation 9/10 or Player 5/6, it is added as part of the "Unlocker" install and is located in the VMware Workstation folder under "Program Files (x86)\VMware" or "Program Files\VMware" for a 32-bit host in a default install, make sure the "Device status" has "Connected" ticked generally "Connect at power on" is ticked by default.
You may get an error when switching mounted .iso's in the Virtual Machine Settings "The guest operating system has locked the CD-ROM door..." it is OK to answer "Yes" to the question "Disconnect anyway and override the lock?".
In your OS X desktop a VMware Tools icon should appear in the upper right corner, double click and select "Install VMware Tools", it is recommended to install VMware Tools as it offers an enhanced user experience when working with the Virtual Machine. Please note VMware Tools cannot be installed from the menu in Workstation 9/10 or Player 5/6 to an OS X Virtual Machine (unlike Fusion 5/6, ESXi 5.1, 5.1 U1 and 5.5 which support VMware Tools installation from the menu).
To get the best out of the graphics, bearing in mind QE/CI are not supported in an OS X VM so applications that use QE/CI are not going to work correctly, Zenith432 has provided an enhanced graphics driver (VMsvga2_v1.2.5_OS_10.6-10.8.pkg) for Snow Leopard, Lion and Mountain Lion, and (VMsvga2_v1.2.5_OS_10.9.pkg) for Mavericks, as well as a patch for guest autofit (guestd_patches.pkg), see links below:
Once you are up and running with your virtual OS X machine you need to do a couple of housekeeping tasks to ensure your virtual machine does not freeze. Firstly go to "System Preferences" and select "Energy Saver" and set "Computer sleep:" and "Display sleep:" to Never and also un-tick "Put hard disks to sleep when possible", "Wake for Ethernet network access" and "Allow power button to put the computer to sleep". In "Desktop & Screen Saver" set "Screen Saver" to "Start after:" Never.
One final note, occasionally you can come across an unsupported CPU error, this should be rare if you have recently downloaded your OS X media from the App Store as the App Store version is updated when a new combo patch is released. Alternatively if your CPU is one of the very latest Intel processors not yet supported by OS X, Apple check CPUID's when installing OS X, and will not install on an unsupported processor platform.
(A prime example was when Intel introduced the Core i3, i5 and i7 and Xeon 5600 series processors, Snow Leopard 10.6.4 was the lowest version of OS X to support the new processors, unfortunately the highest available "vanilla" media was and still is 10.6.3, without the CPUID mask it was impossible to install OS X Snow Leopard. As stated earlier in this post, Apple starting with OS X 10.7 Lion deliver the latest slipstreamed media via the App Store.)
In either case add the following line to your OS X virtual machine configuration file (.vmx):
cpuid.1.eax = "0000:0000:0000:0001:0000:0110:1010:0101" or cpuid.1.eax = "----:----:----:0010:----:----:1010:0111"
The first was based on work by myself and Donk, the second a recommendation by jmattson of VMware, see links below: