Jump to content

How to Install and run OS X in a nested instance of VMware ESXi 5.5 within Workstation 10

MSoK

By MSoK,
in Multi-booting and Virtualisation

 Share
18 posts in this topic

Recommended Posts

MSoK
MSoK
Posted
Posted (edited)

VMware Workstation 10 and ESXi 5.5 have now been released and thanks to the latest Unlocker (1.2.x) from Donk and Zenith432 supports OS X 10.6 (Snow Leopard), 10.7 (Lion), 10.8 (Mountain Lion) and 10.9 (Mavericks) on non Apple hardware.

 

Great news, however what is really interesting, is one of the "unsupported" features of VMware products is "Nested Virtualisation" which if your Intel CPU supports VT-x +EPT (or AMD's AMD-V +RVI, not relevant for Vanilla OS X, but useful for other OS's) will allow nested 64-bit VM's to run like Windows Server 2008 R2, 2012 R2, Hyper-V and OS X. This support was available in ESXi 5.1 but it is only with the release of Workstation 10 and ESXi 5.5 that I have followed up on the capabilities and got them to work for OS X.

 

Donk did some of the original documentation on how to run ESXi under Workstation with Jim Mattson of VMware, as VMware have used nested virtualisation for a long time, if you have ever been on one of their training courses you will more than likely have been using nested instances in the labs, and I know VMware use nested virtualisation in the development cycle. However the stumbling block has been the ability to pass-through the hardware VT-x to more than one layer of virtualisation (Nested), with the latest virtualHW version 10 products Workstation 10, ESXi 5.5 and Fusion 6, (Player 6 does not support ESXi virtualisation) we can now run OS X in a nested instance of ESXi in Workstation which might be useful for development purposes as well as a technical exercise?

 

The process is actually very straight forward, firstly your Intel processor must support VT-x +EPT (Extended Page Table, Intel's branded name for Second Level Address Translation) and both must be enabled in BIOS, below is a link to the "About Intel Virtualisation Technology" web site which shows which processors support both VT-x and EPT:

 

http://ark.intel.com/products/virtualizationtechnology

 

I have broken the "How to" in to three sections, firstly installing ESXi as a guest in Workstation 10, secondly installing the unlocker in the guest ESXi host and finally installing OS X in the nested ESXi host, sections two and three are equally valid for a physical ESXi host as well as a virtual (nested) ESXi host.

 

1. Installing ESXi as a Virtual guest in Workstation 10

 

In Workstation 10 "Create a New Virtual Machine" and choose "Typical" and "Browse..." to the location of your ESXi 5.5 ISO, you should get a little pop up below the selection windows saying "VMware ESXi 5 detected." If you get the pop up the wizard will skip straight to the "Name the Virtual Machine" screen, if not or you selected "I will install the operation system later." Next> select "VMware ESX" as the "Guest operating system" and "VMware ESXi 5" as the "Version". Next> Give your Virtual machine a name and a Location. Next> "Specify Disk Capacity".

 

Next> "Customise Hardware..." 

 

The "Memory" default is 4 GB, so if you are going to use your virtual ESXi host in anger then give it as much memory as you can spare.

In "Processors" the default is 2 with "Number of processors:" 2 and "Number of cores per processor:" 1, I always mirror the physical host, so if my host has one processor with two cores I amend the "Processors" settings to match.

The "Virtualization engine" should be left default "Preferred mode: Automatic" and "Virtualize Intel VT-x/EPT or AMD-V/RVI" ticked, this is the single most important setting in providing 64-bit and virtualisation pass-through (nesting) support.

In terms of  the "Network Adapter" default is NAT but I always change this to "Bridged: Connected directly to the physical network" for best performance and reliability.

The remaining settings can be left as default, the USB Controller can be changed to USB 3.0 if you have the relevant hardware. "Close" and "Finish".

 

OK, provided your CD/DVD (IDE) is pointing at your ESXi 5.5 ISO image, "Power on this virtual machine" and follow the onscreen VMware ESXi 5.5.0 Installer instructions.

 

Once the installation is complete do not forget to "Remove the installation disc before rebooting." i.e. right click on your new "VMware ESXi 5" VM select "Settings..." choose "CD/DVD (IDE)" and either untick both options in "Device status" or choose "Use physical drive:" and click on OK. When you reboot if you have selected "Use physical drive:" it will open the drive door, this is normal.

 

Once your virtual VMware ESXi 5.5.0 is running "Download tools (vSphere Client) to manage this host from:" http://<ip address>/ (DHCP). If you already have the vSphere 5.5 Client installed, skip this step but make a note of the IP Address so you can connect via the vSphere Client. Please note connecting to the hosts website switches to https so you will get a security certificate warning and choose "Continue to this website (not recommended)" or equivalent depending on your browser. At the VMware ESXi Welcome screen select "Download vSphere Client", the client is not stored locally but is a link to the VMware website so takes some time to download and install. I do not recommend selecting "Download VMware vCenter" as it is 3.3 GB download.

 

Install and then start the vSphere 5.5 Client, choosing the "IP address / Name:" of your new virtual ESXi host, "User name:" root, with the "Password:" created this during the ESXi installation. A Security Warning screen will pop up, check the tick box "Install this certificate..." and click "Ignore", next is a "VMware Evaluation Notice" which you can "Assign a license to the ESXi host" or click "OK" to continue with the 60 day evaluation period.

 

While we now have a functioning virtual ESXi host, we need to complete some additional steps to allow OS X to be installed. Firstly you may wish to give the ESXi host a static IP Address and change the Hostname. As the VMware ESXi 5 console screen in Workstation 10 press <F2> to "Customise System/View Logs" the "Authentication Required" screen will pop up requiring Login Name: [root], Password: [******   ] , at the next screen select "Configure Management Network" where you can select "IP Configuration" to "Set static IP address..." and "DNS Configuration" to "Use the following DNS server addresses and hostname:"

 

2. Installing the Unlocker on an ESXi host (Physical or Virtual)

 

In the ESXi host we need to set "Troubleshooting Options" to "Enable ESXi Shell" and "Enable SSH", this is required to allow SSH access to the ESXi host filesystem to install the unlocker. Next we need to copy the unlocker esxi folder to datastore1. This is acheived by selecting your ESXi host, probably by IP Address, in vSphere Client and selecting the "Configuration tab", and under "Hardware" select "Storage" and right click on datastore1 and select "Browse Datastore..." Next click on the fourth icon, a stack of disks with a green up arrow and select " "Upload Folder" browse to the location of your extracted unlocker files and select the esxi folder and click "OK", you will get an "Upload/Download Operation Warning", click "Yes", the selected directory will be uploaded to the ESXi host datastore1.

 

While we are in "Browse Datastore..." in order to speed up the OS X guest VM installation process, create a directory, i.e. iso on the virtual ESXi host datastore1 and copy the OS X DMG image to the newly created directory.

 

Having copied the esxi folder for the unlocker 1.2.x on to the virtual ESXi host datastore we are ready to install the unlocker. To install the unlocker we need to connect to the ESXi host filesystem using SSH with a program like PuTTY (http://www.putty.org/). Putty requires the IP Address, and by default will supply the correct port (22), a terminal or CMD windows will pop up and request a "logon as:" (root) and "Password:" (the same as for vSphere Client) this will come back to a ~ # prompt, the "ls" command will confirm you are now at the root of the ESXi filesystem, this is NOT the same as accessing the datastore in vSphere Client. The datastore is where virtual machines are created and run, the ESXi filesystem is the base operating system of ESXi, so care is required when working with PuTTY in the ESXi filesystem.

 

Next we need to change directory to the unlocker files we uploaded to datastore1 use the following command: cd /vmfs/volumes/datastore1/esxi

 

To confirm the location of the three unlocker files, install.sh, uninstall.sh and Unlocker.ESXi use the following command: ls

 

Next we need to make the files executable with the following command: chmod +x *.*

 

Finally to run the unlocker use the following command: ./install.sh

 

The virtual ESXi host must be re-booted for the unlocker to become functional. 

 

3. Installing OS X as a virtual guest on an ESXi host (Physical or Virtual)

 

To install OS X 10.8 or OS X 10.9 from DMG is relatively straight forward, using the vSphere Client, create a "New Virtual Machine..." Typical, Next> give the virtual machine a "Name:", Next> "Select a destination storage for the virtual machine files:", Next> "Guest Operating System:" Other "Version:" Apple Mac OS X 10.7 (64-bit), Next> "Create Network Connections" leave as default, Next> "Create a Disk" the defaults are fine, but I tend to give my OS X VMs a "Virtual disk size:" of at least 64 GB, I also select "Thin Provision" as space tends to be at a premium bearing in mind we are running a VM within a VM within Workstation 10. Next> Select Finish or I recommend checking the tick box "Edit the virtual machine settings before completion", and click on Continue, as I like to add a second CPU, by adding a second core, to better match current Mac hardware. Select "Video card (adding) and increase the "Total video memory:" to the maximum support 128 MB. Also remove the "New Floppy (adding)" how long is it since a machine ship with a floppy drive! Finally select "New CD/DVD (adding)" and "Browse..." to the datastore where the OS X DMG image was stored, "File type:" needs to be changed to "All files (*.*)", and check the tick box "Connect at power on", once happy with the settings click on "Finish".

 

Although in the Summary tab your OS X virtual machine will show "Guest OS: Apple Mac OS X 10.7 (64-bit)" and "VM Version: 8" OS X 10.8 or 10.9 will install and work as normal, to complete the OS X installation refer to the "How to Install OS X 10.8 (Mountain Lion) and OS X 10.9 (Mavericks) in VMware Workstation 9/10 or Player 5/6" for detailed instructions, see link below:

 

http://www.insanelymac.com/forum/topic/290949-how-to-install-os-x-108-mountain-lion-and-os-x-109-mavericks-in-vmware-workstation-910-or-player-56/

 

Alternatively create your OS X 10.8 or 10.9 VM as detailed above in vSphere Client, then in Workstation 10 "Connect to Server..." and use the same credentials as used in vSphere Client to connect to your nested ESXi 5.5 host. Select the newly created OS X VM and select “Upgrade this virtual machine”, Next> change "Hardware compatibility:" to "Workstation 10.0", Next> and "Finish", once the conversion is complete “Edit virtual machine settings” select “Options” and change “Version:" to "Apple Mac OS X 10.8 (64-bit)" or "Apple Mac OS X 10.9 (64-bit)". Next power on the VM in vSphere Client or Workstation 10.

 

IMPORTANT: When creating a new OS X virtual machine in ESXi (normal or nested) always use vSphere Client NOT Workstation 10, an ESXi hosted OS X virtual machine created using Workstation 10 will not boot! as the vmx configuration file created is not complete. Once created in vSphere Client the vmx can be modified in Workstation 10, and if "Hardware compatibility:" has been upgraded to "Workstation 10.0" vSphere Client can no longer be used to edit Virtual Machine Settings.

 

Thanks to Donk and Zenith432 for the unlocker and William Lam of VMware for relevant resources on his site "virtuallyGhetto" regarding Nested ESXi see links below:

 

http://www.insanelymac.com/forum/topic/267296-esxi-5-mac-os-x-unlocker/

 

http://www.virtuallyghetto.com/

Edited by MSoK
  • Like 1
Link to comment
Share on other sites
Donk
Donk
Posted
Posted

I love this stuff and have run many different hypervisor and hosted products in a nested environment. All the unlocker code tests are run this way, before running on real hardware. I certainly think it is worth while documenting this.

Link to comment
Share on other sites
  • 4 weeks later...
MSoK
MSoK
Posted
  • Author
Posted

Any news regarding the guide?

sebus,

 

Sorry for the delay, yes I will put up the "How to" over the weekend, I sent Donk the instructions for a sanity check before posting, just need to check it all worked OK, stay tuned, i will drop you a PM when the post is available.

  • Like 1
Link to comment
Share on other sites
Donk
Donk
Posted
Posted

sebus,

 

Sorry for the delay, yes I will put up the "How to" over the weekend, I sent Donk the instructions for a sanity check before posting, just need to check it all worked OK, stay tuned, i will drop you a PM when the post is available.

 

Finished replicating it yesterday and worked for me. So good to go for posting.

Link to comment
Share on other sites
MSoK
MSoK
Posted
  • Author
Posted

Finished replicating it yesterday and worked for me. So good to go for posting.

Donk,

 

Thanks for the confirmation, sorry to put you on the spot, should have emailed first.

Link to comment
Share on other sites
Donk
Donk
Posted
Posted

Donk,

 

Thanks for the confirmation, sorry to put you on the spot, should have emailed first.

No problem. Using it now to try some alternative ways to patch ESXi without using vmtar files. Thanks for the notes on how to do it.

Link to comment
Share on other sites
sebus
sebus
Posted
Posted

All works well, nested ESXi 5.5 in Fusion 6.0.2

NFS -ro share from host ML 10.8.4 to ESXi

 

vSphere Client on another Win7 VM

 

As per this post I see no way to actually install Maverick...

 

Probably me being silly... hopefully somebody has simple solution

Link to comment
Share on other sites
  • 2 months later...
MSoK
MSoK
Posted
  • Author
Posted

This is working for me. ESXi 5.5 and OSX 10.9.

Can screen share from my laptop.

Config > 4 cores - Raw Disk - 8 GB Ram - VMware Tools installed.

 

But I can't seem to get the gpu forwarding working.

Anyone get lci pass through to work?

sulphurlad,

 

This should be possible provided your processor supports "Intel® Virtualization Technology for Directed I/O" (VT-d), but appears to be very fussy regarding supported graphics cards, see the excellent work by sgreszcz, link below:

 

http://www.insanelymac.com/forum/topic/285733-graphics-acceleration-in-esxi-51-and-os-x-mountain-lion-vm/?p=1883583

Link to comment
Share on other sites
 Share
Go to topic listing
×
×
  • Create New...