I discovered today that VMWare vSphere Client Version 4.0.0 Build 162856 running on Windows 2003 32bit does not like the screen to be in portrait mode. I witnessed some very strange behavior where regardless of what IP I plugged in (even 127.0.0.1 and any made up credentials), the client would show “Connecting…” and the spinning icon would appear and the go away, username and password boxes would grey out, and it would proceed to hang. Any mouse input would cause windows to append (Not Responding) in the title bar. I finally did get it to connect to one of my servers, but some text labels were missing text. I tried rebooting, updating windows update, etc. The missing labels clued me into perhaps a problem with the video so I rotated my screen back to standard landscape mode and poof, problem solved. Very odd!
I am super excited you can now download a compiled version of OpenSolaris that includes the new ZFS Dedupe support! However when I went to try installing build 128a from GENUNIX, I found it kernel panic’d on both VMWare Workstation and ESXi 4.0.
A little googling and I found this bug and workaround:
6820576 Kernel panic when booting Nevada and OpenSolaris http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6820576 When booting build 121 on a VMware guest instance, the system may panic with the following function listed in the kernel stack trace pcplusmp`ioapic_read Work-around: Boot with the "pcieb" driver disabled by editing the GRUB "kernel$" entry. This can be done interactively by typing the character "e" when the GRUB menu appears and using the arrows key to navigate to the "kernel$" entry. Entering a second "e" will allow one to append to the end of the line the string " -B disable-pcieb=true". To complete the boot, enter a carriage return followed by the "b" character. To make this change persistent, edit the file /rpool/boot/grub/menu.lst and add the same string to the appropriate "kernel$" entries.
That fixed the problem right up thankfully!
Update: VMWare ESXi 4.1 detects the SATA controller just find. The separate SAS card is no longer necessary.
The novelty of having a 1U server in my small apartment has worn off. Even on the workstation setting, the tiny fans running at 10k RPM make my home office inhospitable for all but brief periods. I’ve contemplated getting another case or jury rigging up some large low rpm fans, but in the end I decided its best if I just build a new machine and sell the old server on craigslist.
Before I dive into detail here is my parts list. Just add a SATA drive or two and you’re good to go.
|Supermicro X8SIL-O Motherboard||$149.99|
|4 – Kingston KVR1333D3E9S/2G 2GB 1333MHZ DDR3 ECC||$199.96|
|Lian-Li PC-V351B Case||$109.88|
|Intel Xeon X3440 Lynnfield 2.53ghz||$239.99|
|Rosewill Green Series RG530-2 530W Continuous @40°C, 80 PLUS Certified,ATX12V v2.3 & EPS12V v2.91||$42.49|
|Used LSI SAS3442E-R PCIe from Ebay||$135.99|
|Tax and shipping||$82.55|
I decided the base of my new ESXi system would be the Supermicro X8SIL-O MicroATX motherboard.
The onboard SATA controller is not detected by ESXi 4.0. Thankfully I have a supported LSI PCI-E SAS card.
- MicroATX form factor means I can fit it in a smaller case.
- 4 – DDR3 slots and can be populated with up to 32gb of RAM.
- 2 Intel Gig-E NICs which support jumbo frames (Hello iSCSI!). (Most inexpensive boards use Realtek nics which can be flaky under load and are usually not supported by VMWare out of the box.)
- USB connection on the motherboard that allows you to install your OS to a thumb drive and leave it inside the case.
- Onboard video means one less thing to buy.
I mated it with the Intel Xeon X3440 CPU which is basically the server version of the i7. This is currently the least expensive quare core intel chip that supports hyperthreading, giving you 8 logical cores.
I opted for 4 sticks of Kingston 2GB DDR3 ECC RAM bringing the total ram to 8gb. It is a downgrade from the 16 I have in the 1U server, but I think it’s worth trading in for a little silence.
To hold this beast, I decided on a small form factor case by Lian-Li. The PC-V351B is a almost square. The fans and drives are mounted with rubber grommets which cut sound and vibration nicely. The quality is top notch, but it’s definitely not the case you want if you plan on swapping parts frequently. It has a motherboard tray which slides out when you need to install cards. If you want to pop the side panels off, get your screwdriver out. Each side is held in place by 6 tiny screws. Thanks to my cat knocking my loose parts tray over during assembly, I only need to worry about 4 per side now. In theory, the motherboard tray sliding out means you shouldn’t need to take the sides off. In practice, this is not always the case.
For the power supply, I selected a mid-range supply from Rosewill. It should be a nice stable supply with enough power.
I only had one minor hiccup assembling the pieces. The Lian-li lead for the Power LED has a 3 pin female connector (with the center pin being unused). The board uses 2 pins side by side for the Power LED. It was an easy fix to gently push out the wire from pin 3 and move it to the unused pin 2. Other than that I had plenty of reach with the cables and had sufficient places to tuck excess cables.
Installing ESXi to a thumb drive was super easy. I just followed these instructions. The thumb drive plugs into the connector inside the case and doesn’t get in the way of anything.
I’ve migrated all my VMs to it and so far so good. The best part is I can’t hear it! It’s so quiet I can hardly tell it’s on. Now, who wants to buy a 1U server?
I discovered VMWare has a white paper on running nested vms. This is really cool and something I want to try. If you read the document you’ll notice you are limited on what CPU you can use to accomplish this. My home server had Opteron processors, but they aren’t the right generation. I decided to search craigslist and see if maybe someone was off loading the processors I needed. Wow, first hit I got really lucky. Someone was selling quad core Opteron processors, Barcelona generation family 10H! I drove an hour out and payed a very reasonable price for said processors.
After diligently cleaning the processors and heat sink, I buttoned up the server and pushed the power button. ESXi goes about it’s usually boot sequence and I get a dreaded purple screen that reads:
VMWare ESXi 4.0.0 build 171294 The system has found a problem on your machine and cannot continue. No supported microcode level for this stepping of AMD Family 10h B2 Processer.
I thought maybe I had to reinstall ESXi with the processor change so I inserted the installation cd and tried that. Same error. A quick search for the above string yields a terrible news for me in a post from the VMWare Communities. The last poster on the thread has a link to a wikipedia article that details a bug which makes virtualization fail. My bios actually has a work around option for the B2 cpu’s, but again, same error.
I’m going to try an email AMD, but I’m guessing I am now the owner of two reasonably priced quad core paper weights.
Update: I am out of luck, but AMD did provide some information to help avoid processors with this bug. The TLB erratum affects all original Phenom and Opteron quad core revision BA and B2 processors. Most of these were pulled from the market and replaced with revision B3.
Update: The problem I describe below is because I’ve had my ESXi installed longer than 60 days it is licensed as an ESXi Single Server (free). If you install ESXi initially you can run it for 60 days in Evaluation Mode. You can use an ESXi server in Evaluation Mode fine in Lab Manager.
My home lab runs ESXi 4.0. In a home lab, the only thing I really miss about not having vCenter (Virtual Center) is templates. But I really can’t complain too much, I get an awful lot of functionality from ESXi without paying a dime.
The other day I decided I wanted to install a trial of Lab Manager. I used Lab Manager 2.5 in a previous position and at work we are soon going to have at least a 3.0 Lab Manager installed. It’s a really cool product and I wanted to try out driving some automation using Lab Manager APIs. Having the lab at home allows me do these kinds of things without bothering the admins in the office. When I do finally bother them, I’ll have a better picture to give them.
For the trial, I created two new Win2k3 VMs. 1 for vServer and 1 for Lab Manager. VMWare has made it incredibly easy to install these products. vServer comes with SQL Server Express and installs by default. In a real environment you would likely opt to install on a physical machine with a full fledged SQL Server instance, but for testing SQL Express is fine. With just a few clicks vServer was installed. No issues adding my ESXi to it. Next I installed Lab Manager into the other VM. Easy as pie… Almost.
Warning You cannot add a template because you do not have any enabled hosts available.
The above error message is what I got when I went to create a new template in Lab Manager.
Going to Resources -> Hosts I see a big red X in the Ready column and a message that said Error in Preparing.
Here is the detail from Error in Preparing:
vCenter Lab Manager does not support the license edition of this host "esxi.mydomain.com". Upgrade your host license through vCenter and try again.
After trying a few things I contacted someone at VMWare. My contact said I can’t use ESXi free to trial Lab Manager and I should install an ESX trial. Unfortunately my lab is populated and I’m not willing to migrate all my VMs for the trial. A shame such a smooth process got wrinkled by this tiny detail.
Update: If you start with a fresh ESXi server licensed in evaluation mode you will not encounter the error above. But if you have an existing ESXi server and it’s past the original evaluation you will not be able to use it for a new eval of Lab Manager.
I have now built a couple ESXi machines at home and it can be tough finding hardware that you know is going to work with ESXi. I thought I would contribute a couple working configurations.
Motherboard: Gigabyte GA-P965-S3 rev 1.0
When I built this, onboard SATA ports and NIC wouldn’t work with ESX 3. I couldn’t even get the IDE channel to work when I bought a SAS card to use. It would boot off the IDE cdrom, get to a certain part and die. I ended up having to buy a sata cd-rom. One of the reasons I bought this board is it had 4 pcie ports which would be helpful when none of the onboard items worked.
Storage Controller: LSI SAS3442E-R PCIe
I got a pretty good deal one one of these hunting ebay. It has an internal and external port. To use 4 internal SATA disks, you’ll need a sff-8484 to 4 sata sff-8448 cable.
Video: PCI Radeon 7000 card (Important since the LSI card takes up the 1 – 16x PCIe slot)
When you manage to put together a supported config, ESXi is a very simple install. If you don’t have supported hardware, it fails and tells you pretty quickly. I ran both ESX 3.0 and ESXI 4.0 on the above hardware.
Once I got it up and running, it’s been a good system. I recently caught the ZFS bug though and I needed a new system to allow me to continue running ESXi and another system to start using OpenSolaris.
After scouring craigslist, I found a Supermicro AS-1021M-T2+B system used. It has a H8DME-2 motherboard. I was a little concearned about whether or not I was going to have to jump through hoops to get this to work. I searched a lot about the NVidia MCP55 chipset. It seemed like I would have to do some work and maybe buy either a new NIC or storage card. Turns out ESXi 4.0 installs without a hitch. Both GigE nics are supported as well as the onboard SATA controller. I even did an informal IOMeter test and I got better iops on this than on my other machine with SAS card.
Now I’m repurposing the Gigabyte machine to be my OpenSolaris machine. As I’ve come to expect, that’s not going as smoothly as I hoped. But that’s another post.
A cool use of (the free program) VMWare converter* is to increase the drive size of the windows boot drive (C:). Increasing the size of a separate disk in vmware isn’t too hard. One simple way is to attach a new larger vdisk, mount it in windows, robocopy to the new disk, change drive letter assignments, power down, remove old disk. But boot drives are tricky. With VMWare converter, I just clone my VM and ask it to change the disk size. You can even shrink disks with it.
I came across an irksome problem trying to increase the size of one of my VMs. I was using it to convert one VMWare Workstation image to another with a larger C: drive. This resulted in this message:
FAILED: A file I/O error occurred while accessing ‘J:
This seems to be caused by having opted to selecte pre-allocate for the new disk. Performance is better with pre-allocated disks. Not opting for pre-allocation the problem went away. I wonder if part of the reason is my source image is not pre-allocated.
One thing Converter seems to have over VMWare Diskmanager GUI is it actually resizes the NTFS partition, not just the virtual disk. I did use this utility after converter to Convert my disk from dynamic to preallocated.
Also I’d like to mention I used the donation coder program Screenshot Capture for the image above. Normally I just print screen and fire up Paint.NET, crop, and use red pencil for highlight. But I think the blur and highlight feature of Screenshot Capture is pretty neat.*VMWare vCenter Converter Standalone Client Version 4.0.0 Build 146302