Home Lab Server and Storage Consolidation using ESXi 4.1 and Solaris 11

May 30th, 2011 No comments

It’s been too long since my last post, so without further ado…

Recently I was doing some performance testing of my storage server.  Last I wrote about it, I was using OpenSolaris, but I’ve since moved on to Solaris 11 Express.   I wish I had saved the benchmark info, but I believe over cifs I was getting less than 20 MB/s sequential write.   One reason I suspected performance was poor was I was using 1.5tb drives which use the new 4k sector size.  Apparently Solaris has a problem with this.  Without getting too far ahead of myself I confirmed this did contribute to a 18% performance drop.  To remedy the situation I had to use a modified zpool binary from here to set the ashift value to 12 instead of 9.  Unfortunately you have to use this at pool creation time.

One thing that got me fired up about revisiting my lab is I found this article about using VMWare ESXi 4.1 Passthrough.  Given the correct hardware, you can assign a VM direct hardware access.  Which in my case means I would run Solaris in a VM, and attach the SAS card to it for direct access.  Although I might lose some flexibility, the idea of consolidating another two machines into 1 sounded good to me.  I confirmed the current hardware I had could pass through my SAS card to a Solaris VM just fine with some temporary re-jiggering.

I figured while I was changing my configuration, I would upgrade my storage a bit.  And while I LOVE that Lian-Li case for how quiet and sleek it is, there is no getting around the fact it is not going to hold enough drives.  My desired configuration was 6 – 2tb drives for a raidz2, a drive or two for local vm storage, and maybe some room for an SSD for Zil and/or cache.  My current LSI card only had 4 internal ports (4 external additionally, but I didn’t want to deal with adapters).  So I found a Dell PERC 6/i card on craigslist.

New Configuration (new parts I needed in bold):

Part Price
Supermicro X8SIL-O Motherboard (Price actually went up since I bought it) $154.99
4 – Kingston KVR1333D3E9S/2G 2GB 1333MHZ DDR3 ECC $119.98
Antec Three Hundred Case (6 – 3.5″, 3 – 5.25″) $59.82
Intel Xeon X3440 Lynnfield 2.53ghz (Same price as when I bought it) $239.99
Rosewill Green Series RG530-2 530W Continuous @40°C, 80 PLUS Certified,ATX12V v2.3 & EPS12V v2.91 

(No longer available, YMMVfor pricing a different one)

Dell PERC 6/i from Craigslist $50.00
6 – Samsung Spinpoint F4EG 2tb 5400rpm HD $480
1 Molex to 2 SATA Power Cable $1.80
Cooler Master 4 in 3 HDD Module $24.52
Cooler Master 120mm Fan 4 in 1 Value Pack $14.21
2 – 32pin SFF-8484 to 4 Sata (ebay) $26.38
Western Digital 150gb VelociRaptor (local vm storage) $114.99
8gb Thumb Drive (ESXi installation) $14.00
Total $1000.38

I find it interesting that when I bought these parts for my last esxi build, the motherboard was slightly cheaper and the processor was exactly the same price.  RAM of course dropped quite a bit.  I grabbed current pricing from newegg, amazon, etc.  I did not include tax, depending on vendor and your location that may or may not apply.

I deliberated quite a bit on the case.  Should I go full blown rack mount server case with hot swap sleds?  I decided to go with a mid-tower case that used 120mm fans for cooling.  I opted to NOT get hot swap sleds.  Although I love the convenience, the fact is you need to push more air with (probably) smaller fans to deal with the added bulk of the hot swap trays.  You’ll notice in the setup I’ve purchased, all drives have 120mm fans in front of them which delivers excellent air flow with the noise of a desktop, not a helicopter server.  The Antec Three Hundred is not a high end Antec case, but it is still good quality.  They included thumb screws for the 6 – 3.5″ drives and cable routing is good.

I have now successfully combined my storage and esxi server.  So far it’s running quite well.  I even got a Kill-A-Watt because I was concerned the additional drives might be pushing the Power Supply.  With 5 VM’s running and mostly idle it draws 105 watts.  When I was doing heavy copies it hit around 140, but that’s no where near the 530 watts the power supply is rated for.



Categories: Uncategorized Tags: , , ,

VMWare vSphere Client does not like 90 degree portrait mode

December 18th, 2009 2 comments

I discovered today that VMWare vSphere Client Version 4.0.0 Build 162856 running on Windows 2003 32bit does not like the screen to be in portrait mode.  I witnessed some very strange behavior where regardless of what IP I plugged in (even and any made up credentials), the client would show “Connecting…” and the spinning icon would appear and the go away, username and password boxes would grey out, and it would proceed to hang.  Any mouse input would cause windows to append (Not Responding) in the title bar.  I finally did get it to connect to one of my servers, but some text labels were missing text.  I tried rebooting, updating windows update, etc.  The missing labels clued me into perhaps a problem with the video so I rotated my screen back to standard landscape mode and poof, problem solved.  Very odd!

Categories: Uncategorized Tags:

Virtual Dedupe Requires Workaround: OpenSolaris build 128a Kernel Panics on Boot in VMWare

December 4th, 2009 2 comments

I am super excited you can now download a compiled version of OpenSolaris that includes the new ZFS Dedupe support!  However when I went to try installing build 128a from GENUNIX, I found it kernel panic’d on both VMWare Workstation and ESXi 4.0.

A little googling and I found this bug and workaround:

6820576 Kernel panic when booting Nevada and OpenSolaris

 	When booting build 121 on a VMware guest instance, the system
 	may panic with the following function listed in the kernel
 	stack trace


 	Work-around: Boot with the "pcieb" driver disabled by editing
 	the GRUB "kernel$" entry.  This can be done interactively by
 	typing the character "e" when the GRUB menu appears and using
 	the arrows key to navigate to the "kernel$" entry.  Entering a
 	second "e" will allow one to append to the end of the line the
 	string " -B disable-pcieb=true".

 	To complete the boot, enter a carriage return followed by the
 	"b" character.

 	To make this change persistent, edit the file
 	/rpool/boot/grub/menu.lst and add the same string to the
 	appropriate "kernel$" entries.

That fixed the problem right up thankfully!

Categories: Uncategorized Tags: ,

Easily secure your iPhone’s connection

October 24th, 2009 No comments

When you’re out and about using your phone’s wifi connection to get net access, you should be concerned about your security. Just because the sites you visit are SSL doesn’t mean you’re safe. Just have a look at the tool SSLStrip. Even if you trust the hotspot, do you trust that no one is intercepting the traffic between you and the hot spot?

I discovered today it’s actually quite easy to setup a home VPN and have all your iPhone’s traffic run over it. All you need is a compatible router to run the dd-wrt firmware on.  dd-wrt actually does a lot of cool features.

  1. Install the dd-wrt vpn firmware to your router
  2. Get a dynamic hostname and setup software to update when your ip changes.  (If you don’t already have one)
  3. Configure dd-wrt to update your dynamic host name under Setup->DDNS
  4. Configure the PPTP server built into dd-wrt
    1. Services->VPN
    2. PPTP: Enable
    3. Broadcast Support: Enable
    4. Force MPPE Encryption: Enable
    5. Server IP:
    6. Client IP:
    7. CHAP-Secrets: YourMadeUpUserName * YourMadeUpPassword *
  5. Setup VPN in iPhone
    1. Under Settings->General->Network->VPN->Add VPN
    2. PPTP
    3. Description: Home
    4. Server: yourdynamichostname.dyndns.org
    5. Account: YourMadeUpUserName
    6. Password: YourMadeUpPassword
    7. Encryption Level: Maximum
    8. Send All Traffic: ON
    9. Proxy: Off
    10. Save
    11. Slide VPN to ON
  6. You should see a little icon in the upper left that says VPN when you’re connected.  You can verify the vpn is working by going to www.whatismyip.com which should display your IP from home

The above was done with dd-wrt v24-sp2.

New Super Quiet Supermicro X8SIL VMWare ESXi Server

October 22nd, 2009 7 comments

Update: VMWare ESXi 4.1 detects the SATA controller just find.  The separate SAS card is no longer necessary.

The novelty of having a 1U server in my small apartment has worn off. Even on the workstation setting, the tiny fans running at 10k RPM make my home office inhospitable for all but brief periods. I’ve contemplated getting another case or jury rigging up some large low rpm fans, but in the end I decided its best if I just build a new machine and sell the old server on craigslist.

Before I dive into detail here is my parts list. Just add a SATA drive or two and you’re good to go.

Part Price
Supermicro X8SIL-O Motherboard $149.99
4 – Kingston KVR1333D3E9S/2G 2GB 1333MHZ DDR3 ECC $199.96
Lian-Li PC-V351B Case $109.88
Intel Xeon X3440 Lynnfield 2.53ghz $239.99
Rosewill Green Series RG530-2 530W Continuous @40°C, 80 PLUS Certified,ATX12V v2.3 & EPS12V v2.91 $42.49
Used LSI SAS3442E-R PCIe from Ebay $135.99
Tax and shipping $82.55
Total $960.85

I decided the base of my new ESXi system would be the Supermicro X8SIL-O MicroATX motherboard.

Motherboard Negatives:

  • The onboard SATA controller is not detected by ESXi 4.0. Thankfully I have a supported LSI PCI-E SAS card.

Motherboard Positives:

  • MicroATX form factor means I can fit it in a smaller case.
  • 4 – DDR3 slots and can be populated with up to 32gb of RAM.
  • 2 Intel Gig-E NICs which support jumbo frames (Hello iSCSI!). (Most inexpensive boards use Realtek nics which can be flaky under load and are usually not supported by VMWare out of the box.)
  • USB connection on the motherboard that allows you to install your OS to a thumb drive and leave it inside the case.
  • Onboard video means one less thing to buy.

I mated it with the Intel Xeon X3440 CPU which is basically the server version of the i7. This is currently the least expensive quare core intel chip that supports hyperthreading, giving you 8 logical cores.

I opted for 4 sticks of Kingston 2GB DDR3 ECC RAM bringing the total ram to 8gb. It is a downgrade from the 16 I have in the 1U server, but I think it’s worth trading in for a little silence.

To hold this beast, I decided on a small form factor case by Lian-Li. The PC-V351B is a almost square. The fans and drives are mounted with rubber grommets which cut sound and vibration nicely. The quality is top notch, but it’s definitely not the case you want if you plan on swapping parts frequently. It has a motherboard tray which slides out when you need to install cards. If you want to pop the side panels off, get your screwdriver out. Each side is held in place by 6 tiny screws. Thanks to my cat knocking my loose parts tray over during assembly, I only need to worry about 4 per side now. In theory, the motherboard tray sliding out means you shouldn’t need to take the sides off. In practice, this is not always the case.

For the power supply, I selected a mid-range supply from Rosewill. It should be a nice stable supply with enough power.

I only had one minor hiccup assembling the pieces. The Lian-li lead for the Power LED has a 3 pin female connector (with the center pin being unused). The board uses 2 pins side by side for the Power LED. It was an easy fix to gently push out the wire from pin 3 and move it to the unused pin 2. Other than that I had plenty of reach with the cables and had sufficient places to tuck excess cables.

Installing ESXi to a thumb drive was super easy. I just followed these instructions. The thumb drive plugs into the connector inside the case and doesn’t get in the way of anything.

I’ve migrated all my VMs to it and so far so good. The best part is I can’t hear it! It’s so quiet I can hardly tell it’s on. Now, who wants to buy a 1U server?

Categories: Uncategorized Tags: ,

Visual Studio Deployment Projects do not like dashes in your path.

October 22nd, 2009 No comments

I was attempting to build one of our projects under a new path and was experiencing this incredibly frustrating problem of having my output msi files being significantly smaller than the official build. We build a number of solutions and some contain Visual Studio Deployment projects which create merge modules. If I built the solution under c:\projects\WidgetX it built fine. If I built it under c:\projects\WidgetX-mirror it was missing a number of third party dependencies. Even pulling dependencies from the wrong location. It turns out if I remove the dash and build in c:\projects-WidgetXmirror it correctly detects dependencies. Underscores also work. Very bizarre.

Categories: Uncategorized Tags: ,

Quad core AMD 8354 Barcelona 10h B2 and ESXi does not work thanks to a bug in the processor

September 27th, 2009 No comments

I discovered VMWare has a white paper on running nested vms. This is really cool and something I want to try.  If you read the document you’ll notice you are limited on what CPU you can use to accomplish this.  My home server had Opteron processors, but they aren’t the right generation.  I decided to search craigslist and see if maybe someone was off loading the processors I needed.  Wow, first hit I got really lucky.  Someone was selling quad core Opteron processors, Barcelona generation family 10H!  I drove an hour out and payed a very reasonable price for said processors.

After diligently cleaning the processors and heat sink, I buttoned up the server and pushed the power button.  ESXi goes about it’s usually boot sequence and I get a dreaded purple screen that reads:

VMWare ESXi 4.0.0 build 171294
The system has found a problem on your machine and cannot continue.
No supported microcode level for this stepping of
AMD Family 10h B2 Processer.

I thought maybe I had to reinstall ESXi with the processor change so I inserted the installation cd and tried that. Same error. A quick search for the above string yields a terrible news for me in a post from the VMWare Communities. The last poster on the thread has a link to a wikipedia article that details a bug which makes virtualization fail. My bios actually has a work around option for the B2 cpu’s, but again, same error.

I’m going to try an email AMD, but I’m guessing I am now the owner of two reasonably priced quad core paper weights.

Update: I am out of luck, but AMD did provide some information to help avoid processors with this bug. The TLB erratum affects all original Phenom and Opteron quad core revision BA and B2 processors. Most of these were pulled from the market and replaced with revision B3.

Categories: Uncategorized Tags: ,

VMWare Lab Manager trial and ESXi 4.0 free

September 21st, 2009 3 comments

Update: The problem I describe below is because I’ve had my ESXi installed longer than 60 days it is licensed as an ESXi Single Server (free).   If you install ESXi initially you can run it for 60 days in Evaluation Mode.  You can use an ESXi server in Evaluation Mode  fine in Lab Manager.

My home lab runs ESXi 4.0.   In a home lab, the only thing I really miss about not having vCenter (Virtual Center) is templates.   But I really can’t complain too much, I get an awful lot of functionality from ESXi without paying a dime.

The other day I decided I wanted to install a trial of Lab Manager. I used Lab Manager 2.5 in a previous position and at work we are soon going to have at least a 3.0 Lab Manager installed. It’s a really cool product and I wanted to try out driving some automation using Lab Manager APIs. Having the lab at home allows me do these kinds of things without bothering the admins in the office. When I do finally bother them, I’ll have a better picture to give them.

For the trial, I created two new Win2k3 VMs.  1 for vServer and 1 for Lab Manager.   VMWare has made it incredibly easy to install these products.  vServer comes with SQL Server Express and installs by default.  In a real environment you would likely opt to install on a physical machine with a full fledged SQL Server instance, but for testing SQL Express is fine.   With just a few clicks vServer was installed.  No issues adding my ESXi to it.   Next I installed Lab Manager into the other VM.  Easy as pie… Almost.

You cannot add a template because you do not have any enabled hosts available.

The above error message is what I got when I went to create a new template in Lab Manager.

Going to Resources -> Hosts I see a big red X in the Ready column and a message that said Error in Preparing.

Here is the detail from Error in Preparing:

vCenter Lab Manager does not support the license edition of this host "esxi.mydomain.com".
 Upgrade your host license through vCenter and try again.

After trying a few things I contacted someone at VMWare.  My contact said I can’t use ESXi free to trial Lab Manager and I should install an ESX trial.  Unfortunately my lab is populated and I’m not willing to migrate all my VMs for the trial.   A shame such a smooth process got wrinkled by this tiny detail.

Update: If you start with a fresh ESXi server licensed in evaluation mode you will not encounter the error above.  But if you have an existing ESXi server and it’s past the original evaluation you will not be able to use it for a new eval of Lab Manager.

Categories: Uncategorized Tags: , ,

Goodbye MediaPortal, Hello XBMC!

June 16th, 2009 1 comment

It’s been a few years since I first dedicated a computer to be hooked up to my TV. The ideal, a nice pretty interface to access my digital content from. I’ve used MythTV, Freevo, my XBox 360 via TVersity, and MediaPortal.  The last two what being the most recent.  I have a TiVo so I don’t need a device that works with a tuner of any sort.  I can’t comment on the recording abilities of any of these apps.

I abandoned Freevo and MythTV because the setup of MediaPortal can’t be beat.  Install Windows, Install MediaPortal, and install K-Lite Codec pack.  Done.   No linux drivers to fuss with.

I had heard about XBox Media Center (XBMC) and it sounded interesting, but I discounted it because I would need an original xbox to load it on.  But wait!  It turns out I was wrong!  You can install it on an original XBox yeah, but it also works on Windows, Linux, and MacOS X!  Awesome.

The screenshots on the website make a nice case for the nice pretty interface.  But the proof is in the pudding (and I love pudding).  MediaPortal has a separate app to configure all the settings.  It’s a standard windows app and requires getting up close to the TV to see all the options.  XBMC has all the settings right there in the main program.  So setup is little more than navigating the big beautiful menus and hitting enter or escape.  You can tell the intent was to make it easy to setup even if you just have an xbox controller.  It makes for a pleasant experience.

Setup aside, one of the most compelling features for me is how easily you can make it pull down info from IMDB and a few other sources.  It turns a sparse file listing into a rich library with movie posters, plot, and cast info at your fingertips.  In MediaPortal, initial batch downloading of this info was done in the config app.  In XBMC I could still browse my movies while a nice box in the upper right hand corner told me which movies it was processing.  Very nice.

I’ve only dipped my toe in, but I’m sold on XBMC.

Suddenly can’t login to OpenSolaris 2009.06 CIFS share

June 16th, 2009 4 comments

I woke up this morning and couldn’t get onto my CIFS share. A quick look at /var/adm/messages and I saw this problem:

Jun 15 23:10:10 zed idmap[346]: [ID 702911 auth.notice] GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Clock skew too great)

Ok so this is because the clock on this machine is not close enough to the clock on my domain controller. I’ll just do a ‘crontab -e’ and plug this in:

# Sync date/time with my domain controller
15 * * * * /usr/sbin/ntpdate your.domain.controller.com

Now it should stay synchronized. But wait, I still can’t access my shares.

# svcadm disable idmap
# svcadm disable smb/server
# svcadm enable -r idmap
# svcadm enable -r smb/server

That didn’t do it.

# smbadm list
[*] [mydomain.com]

…and proceeds to hang.

# smbadm join -w WORKGROUP

# smbadm join -u domainuser mydomain

/var/adm/messages shows: svc.startd[7]: [ID 122153 daemon.warning] svc:/network/smb/server:default: Method or service exit timed out. Killing contract 70.

Also noticed despite disabling the smb/server, the process still appears to be running. Kill -9 does nothing.

I had experienced a similar issue earlier during setup and I had written it off. It’s looking like the stability of CIFS isn’t so rocksolid. This post on the cifs-discuss list definitely shows I’m not the only one having issues.

I’m tempted to use VirtualBox and run virtual Win2k3 server on top of OpenSolaris. I would create an iSCSI target in my zpool and point the Win2k3 box at that. Let windows seamlessly share files which it is good at and OpenSolaris manage the storage which it is good at. It’s an interesting thought, but I’m going to see if the latest SXCE fixes my CIFS woes first.

Categories: Uncategorized Tags: