wolf++

Now that I have a new box to run ESXi, I’ve repurposing my GA-P965-S3 based system for OpenSolaris.  I had a lot of trouble getting this to work.  I was initially using OpenSolaris 2008.11.  I could get it installed.  Reboot, login screen comes up.  I plug in my credentials and as soon as the password entry box dissappeared, lockup.  Mouse stops responding, keyboard stops responding.  I tried every bios setting, disabling everything, etc.  Tried different drives, different video card.  Even tried my LSI SAS card instead of the onboard SATA.  Finally I recalled reading a post somewhere that someone was having issues with 4gb of RAM.  So I brought the system down to 2gb and BAM it worked.  Soon after all this, 2009.06 came out.  I installed that and it worked fine with 4gb of memory.  All 6 onboard SATA ports worked.

For drives, I have 2 – 750gb from my old Win2k3 based fileserver.   I also had the 4 – 1.5tb Seagate drives that came with my Opteron box.   I am allocating 2 – 50gb partitions on the 750gb drives for OS, and carving the rest out for a mirrored data partition.  The 4 – 1.5tb drives are going to be in a raidz.

The OS installer doesn’t allow you create a mirror to start with.  I followed Darkstar’s post on creating a bootable root mirror and it worked great.  You can only do this with slices, not entire disks.  The OpenSolaris installer gives you the option of creating slices or using the entire disk, so remember to use slices if you want to create a mirror.

Creating the raidz is very simple.  In one command I had 4tb of useable storage with all the awesomeness of ZFS and RAIDZ.  I ran some simple benchmarks on a single 500gb drive (non-mirrored) and my new 4tb RAIDZ using FileBench.  The results of the benchmark are below.  This confirms my RAIDZ is quite a bit faster than the single disk.

I started to offload data from my old Win2k3 fileserver onto the new RAIDZ.  I added OpenSolaris to the domain and created a CIFS (windows friendly) share.   Tim Thomas’ blog has a good post on how to do this.  I did find out of the box, it didn’t like my Win2k8 domain controller.  I decided to just remove that machine from my domain while I work out the initial setup.  I’ll probably revisit this later.  Permissions appear to be another tricky part of CIFS I’m going to come back to.

Unfortunately after a few hundred gigs of transfer, 1 of the 1.5 tb drives failed.  The RAIDZ kept on going, but soon after the first drive failure, the second drive started showing errors.   I used the Ultimate Boot CD and confirmed both drives are indeed failing.  1 of which is making click of death noises, the other appears to be on the way to failure.  I opted to go with Seagate’s Advanced Replacement and pay $20 per drive so I could get everything back up and running quickly.  There should be a discount for multiple drives.  Also, paying for this at all on a drive that is a few months old kind of stinks.

Here are the benchmark results:

Throughput breakdown (ops per second)

Bandwidth breakdown (MB/s)

wolf Uncategorized hardware, solaris, zfs

I have now built a couple ESXi machines at home and it can be tough finding hardware that you know is going to work with ESXi.  I thought I would contribute a couple working configurations.

Motherboard: Gigabyte GA-P965-S3 rev 1.0

When I built this, onboard SATA ports and NIC wouldn’t work with ESX 3.  I couldn’t even get the IDE channel to work when I bought a SAS card to use.  It would boot off the IDE cdrom, get to a certain part and die.  I ended up having to buy a sata cd-rom.  One of the reasons I bought this board is it had 4 pcie ports which would be helpful when none of the onboard items worked.

Storage Controller: LSI SAS3442E-R PCIe

I got a pretty good deal one one of these hunting ebay.  It has an internal and external port.  To use 4 internal SATA disks, you’ll need a sff-8484 to 4 sata sff-8448 cable.

Video: PCI Radeon 7000 card (Important since the LSI card takes up the 1 – 16x PCIe slot)

NIC: Intel Pro/1000 PT Desktop Adapter

When you manage to put together a supported config, ESXi is a very simple install.  If you don’t have supported hardware, it fails and tells you pretty quickly.  I ran both ESX 3.0 and ESXI 4.0 on the above hardware.

Once I got it up and running, it’s been a good system.  I recently caught the ZFS bug though and I needed a new system to allow me to continue running ESXi and another system to start using OpenSolaris.

After scouring craigslist, I found a Supermicro AS-1021M-T2+B system used.  It has a H8DME-2 motherboard.  I was a little concearned about whether or not I was going to have to jump through hoops to get this to work.  I searched a lot about the NVidia MCP55 chipset.  It seemed like I would have to do some work and maybe buy either a new NIC or storage card.  Turns out ESXi 4.0 installs without a hitch.  Both GigE nics are supported as well as the onboard SATA controller.  I even did an informal IOMeter test and I got better iops on this than on my other machine with SAS card.

Now I’m repurposing the Gigabyte machine to be my OpenSolaris machine.  As I’ve come to expect, that’s not going as smoothly as I hoped.  But that’s another post.

wolf Uncategorized hardware, solaris, vmware

A cool use of  (the free program) VMWare converter* is to increase the drive size of the windows boot drive (C:).  Increasing the size of a separate disk in vmware isn’t too hard. One simple way is to attach a new larger vdisk, mount it in windows, robocopy to the new disk, change drive letter assignments, power down, remove old disk. But boot drives are tricky. With VMWare converter, I just clone my VM and ask it to change the disk size. You can even shrink disks with it.

I came across an irksome problem trying to increase the size of one of my VMs. I was using it to convert one VMWare Workstation image to another with a larger C: drive. This resulted in this message:

FAILED: A file I/O error occurred while accessing ‘J:
\VM\UniversalBuildTemplate\W2K3UniBuildTemplate\W2K3UniBuildTemplate.vmdk’.

This seems to be caused by having opted to selecte pre-allocate for the new disk.  Performance is better with pre-allocated disks. Not opting for pre-allocation the problem went away.  I wonder if part of the reason is my source image is not pre-allocated.

One thing Converter seems to have over VMWare Diskmanager GUI is it actually resizes the NTFS partition, not just the virtual disk.   I did use this utility after converter to Convert my disk from dynamic to preallocated.

Also I’d like to mention I used the donation coder program Screenshot Capture for the image above.  Normally I just print screen and fire up Paint.NET, crop, and use red pencil for highlight.  But I think the blur and highlight feature of Screenshot Capture is pretty neat.

wolf Uncategorized vmware

So in my previous post I wrote about how I got around a problem with some standard error from Perforce. I want to take my work around function and pipe it into Tee-object so I can simultaneously display and log what’s going on with the execution of my consistency check. A consistency check could take a while, so being able to see it’s running output is important.

In my function, I use Invoke-Expression. Well, turns out it’s not so easy to just pipe that into Tee-Object. At least in PSH V1.

Here is my test PowerShell script:

write-host Running in PowerShell Version $host.version.major
write-host 'Executing: Invoke-Expression "print.exe /?" | tee-object -filepath test.log'
Invoke-Expression "print.exe /?" | tee-object -filepath test.log
write-host Now lets get the content of test.log
get-content test.log
write-host
write-host 'Executing: Invoke-Expression "print.exe /?" > test.log'
Invoke-Expression "print.exe /?" > test.log
write-host Now lets get the content of test.log 
get-content test.log

Run it inside PowerShell v1:

PS C:\temp> .\test.ps1
Running in PowerShell Version 1
Executing: Invoke-Expression "print.exe /?" | tee-object -filepath test.log
Prints a text file.
 
PRINT [/D:device] [[drive:][path]filename[...]]
 
   /D:device   Specifies a print device.
 
Now lets get the content of test.log
Get-Content : Cannot find path 'C:\temp\test.log' because it does not exist.
At C:\temp\test.ps1:5 char:12
+ get-content  <<<< test.log
 
Executing: Invoke-Expression "print.exe /?" > test.log
Now lets get the content of test.log
Prints a text file.
 
PRINT [/D:device] [[drive:][path]filename[...]]
 
   /D:device   Specifies a print device.

The pipe doesn’t work, but standard redirection does. I also tried piping it to out-file. Oddly, the output goes straight to the screen and nothing to the file.

Now let’s try it in v2 CTP3:

PS C:\temp> .\test.ps1
Running in PowerShell Version 2
Executing: Invoke-Expression "print.exe /?" | tee-object -filepath test.log
Prints a text file.
 
PRINT [/D:device] [[drive:][path]filename[...]]
 
   /D:device   Specifies a print device.
 
Now lets get the content of test.log
Prints a text file.
 
PRINT [/D:device] [[drive:][path]filename[...]]
 
   /D:device   Specifies a print device.
 
 
Executing: Invoke-Expression "print.exe /?" > test.log
Now lets get the content of test.log
Prints a text file.
 
PRINT [/D:device] [[drive:][path]filename[...]]
 
   /D:device   Specifies a print device.
 
PS C:\temp>

In v2, we get the expected result.

I was about to give up on this, but I found this post that describes basically the same issue. The suggested fix is to wrap the Invoke-Expression (or iex) in parenthesis.

PS C:\temp> (Invoke-Expression "print.exe /?") | tee-object -filepath test.log
Prints a text file.
 
PRINT [/D:device] [[drive:][path]filename[...]]
 
   /D:device   Specifies a print device.
 
PS C:\temp> get-content test.log
Prints a text file.
 
PRINT [/D:device] [[drive:][path]filename[...]]
 
   /D:device   Specifies a print device.
 
PS C:\temp>

Keep on truckin..

wolf Uncategorized powershell

Ok I’m baffled. I am working on the previously mentioned powershell perforce backup script. I intentionally corrupted one of my database files to make sure I’m correctly catching the error. This ran me into a brick wall with trying to trap some standard error output from Perforce. At least I think it’s standard error. At this point I could be convinced some new output stream has mysteriously been created that redirects output to some black hole.

Setting the script aside, let’s take a look at the basics. Here’s what happens in a standard dos command window:

F:\p4backup>p4d -r F:\P4ROOT -xv > test.log 2>&1
 
F:\p4backup>type test.log
Validating db.counters
Validating db.logger
Validating db.user
Validating db.group
Validating db.depot
Validating db.domain
Validating db.view
Validating db.review
Perforce server error:
        Database open error on db.have!
        BTree is corrupt!
 
F:\p4backup>

Now let’s try it in powershell:

PS F:\p4backup> p4d -r F:\P4ROOT -xv > test.log 2>&1
PS F:\p4backup> type test.log
Validating db.counters
Validating db.logger
Validating db.user
Validating db.group
Validating db.depot
Validating db.domain
Validating db.view
Validating db.review
PS F:\p4backup>

No error! It mysteriously disappeared. Ok this lead me to believe something funky was up with standard error redirection, so I wrote a little standard error test program:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
 
namespace StandardErrorTester
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("This is being written to standard output.");
            Console.Error.WriteLine("This is being written to standard error.");
        }
    }
}

Let’s run the test program in a dos command window:

F:\p4backup>StandardErrorTester.exe > error.log 2>&1
 
F:\p4backup>type error.log
This is being written to standard output.
This is being written to standard error.
 
F:\p4backup>

Works as expected. Now let’s try it in powershell:

PS F:\p4backup> .\StandardErrorTester.exe > error.log 2>&1
PS F:\p4backup> type error.log
This is being written to standard output.
StandardErrorTester.exe : This is being written to standard error.
At line:1 char:26
+ .\StandardErrorTester.exe  <<<< > error.log 2>&1
PS F:\p4backup>

The standard error does get wrapped with some strange powershell stuff, but all the output is present. I’m wondering if in the perforce case, the errors are getting interpreted and hidden somehow. Very strange.

Here is simple work around. It’s a function that takes what you want to execute, executes it via a batch script which makes the context the dos command shell. You can still check $LASTEXITCODE and filter the function through pipes or redirection. It’s ugly because it has to create a stub batch file, but it gets around the problem I’m having.

function RunCombineStdErrStdOut
{
    param([string]$commandLineToRun)
    # Get the path where the current script is so we can create
    # our batch file next to it
    $curScript = $script:myInvocation.MyCommand.Path
    $curPath = (Split-Path $curScript)
    # We're going to create a batch file right next to our ps1 script, wherever it lives.
    $runCombBatchFile = $curPath + "\RunCombineStdErrStdOut.cmd"
    if (test-Path -path $runCombBatchFile)
    {
        # Remove the batch file if it exists before we start 
        remove-Item $runCombBatchFile
    }
    # Create our tiny batch script to combine standard error and standard out
    write-Output '@echo off' | out-file $runCombBatchFile -enc ascii -append
    write-Output 'REM This script is auto-created by a PowerShell script' | out-file $runCombBatchFile -enc ascii -append
    write-Output 'REM Modifications will be lost. Do not edit directly' | out-file $runCombBatchFile -enc ascii -append
    write-Output ('REM Script Created by: ' + $curScript.ToString()) | out-file $runCombBatchFile -enc ascii -append
    # %* is a little trick that refers to all the paramters, not just a single one
    # Remove comment for debugging
    #write-Output 'echo Running %*' | out-file $runCombBatchFile -enc ascii -append
    write-Output '%* 2>&1' | out-file $runCombBatchFile -enc ascii -append
    # This will execute the batch file with the command passed to the function
    Invoke-Expression ($runCombBatchFile + " " + $commandLineToRun)
    # You can still check $LASTEXITCODE once this executes
    # it will reflect whatever command we executed
}
 
 
#  Just a simple example to get you going
RunCombineStdErrStdOut "ipconfig /all"

wolf Uncategorized powershell

Robocopy is a pretty good utility for copyin/mirroring directories in windows.  I use it a lot.  I just spent a bit of time thinking powershell hated robocopy because I was having a lot of trouble calling a robocopy against a directory that had a space in it.  Check out this command line.  Looks pretty innocent to me, but as you can see, robocopy blows its top!

PS H:\temp> robocopy 'h:\temp\test1' 'H:\temp\test space\' /L
 
-------------------------------------------------------------------------------
ROBOCOPY     ::     Robust File Copy for Windows     ::     Version XP010
-------------------------------------------------------------------------------
 
 Started : Wed Jan 28 09:43:27 2009
 
Source : h:\temp\test1\
Dest : H:\temp\test space" \L\
 
Files : *.*
 
Options : *.* /COPY:DAT /R:1000000 /W:30
 
------------------------------------------------------------------------------
 
2009/01/28 09:43:27 ERROR 123 (0x0000007B) Accessing Destination Directory H:\temp\test space" \L\
The filename, directory name, or volume label syntax is incorrect.
PS H:\temp&gt;</span>

Turns out the trailing slash on the path that was making it freak out.  Don’t end with a slash folks.  I’m not even sure why I was because I normally don’t!

wolf Uncategorized robocopy

I was working on a fancy backup script for Perforce written in Powershell and I’ve definitely come across a bug in PowerShell V1.  Take a look at this:

function Usage()
{
	write-host "Hey fool, you need to pass 1 paramter to this utility!"
	exit 1
}
 
if ($args.Count -ne 1)
{
	Usage
}
write-host "Doing stuff..."

So that little snippet, if run without a parameter, will tell you you’re a fool and never tell you it’s doing stuff. I use exit to abort the entire script.  Works as expected.  Now take a look at this:

function GetMyWidget()
{
	# Pretend I couldnt find my widget
	if ($foundWidget -ne $true)
	{
		write-host "Ooops, something went wrong and I want to exit this script!"
		exit 1
	}
	return $widget
}
$w = GetMyWidget
write-Host "Whatever you do, don't print this!"

I expect the above snippet to print out Ooops… and that’s it. Unfortunately this is what it prints:

PS H:\temp> H:\temp\test.ps1
Ooops, something went wrong and I want to exit this script!
The ‘=’ operator failed: System error..
At H:\temp\test.ps1:11 char:5
+ $w =  <<<< GetMyWidget
Whatever you do, don’t print this!
PS H:\temp>

Execute the same code in Powershell V2 CTP3, no error, and it NEVER prints “Whatever you do, don’t print this!”.   The difference between the snippets is I’m assigning a variable to the return value of my function that contains an exit.  Note both environments $errorActionPreference is continue.

That is pretty annoying.  The workaround is to put $script:errorActionPreference = “Stop” before you exit statement, or to make sure that is the preference somewhere.

wolf Uncategorized powershell

I just took ownership of a Dell M4400 refurbished laptop. I was excited to bust it out, but my heart sank when I cracked the lid and saw the bezel above the keyboard wasn’t seated. I pushed down and it clicked into place, all but one end. One of the plastic tabs that clicks into a receiving slot on the laptop had broken at some point prior to my taking ownership. I’m surprised Dell didn’t notice this when they were ‘refurbishing’ it. I hopped online with Dell tech support chat and I actually had a rather pleasant experience resolving the issue. It was only a couple moments before I had a live person and only a few more before they told me the part would be here tomorrow. No arm twisting to get them to understand I’m a nerd who is more than capable of installing a plastic bezel. Plus they sent me the OS install disk which I guess I forgot to opt for when I ordered the laptop, gratis. Sweet.

When I picked out the M4400, I specifically opted for one with the eye bleeding resolution of 1920×1080 for it’s 15.4″ screen. I was a tad concerned I maybe had gone too far this time, but it turns out I didn’t have to enable super xtra large blind fonts or anything. Such high resolutions are good for side by side viewing. I think the feature I was most excited about having in a laptop is the back lit keyboard. Even though I’m not a touch typist, I’ve found frequently in the dark I’ve wanted back lighting on my keys so I could find a function key.

The laptop came with Vista 32bit which certainly wouldn’t do. A while back I installed Vista on a machine and reverted back to XP after a few short days. For this machine, I opted to attempt to dual boot Windows Server 2008 64bit and Windows 7 Beta 64bit. I figure 2008 will give me the stripped down performance oriented laptop without the flashy stuff I always turn off. I wanted to also install Windows 7 Beta and see what all the fuss was about. Seems like every tech blog I read is just going ga-ga about it.

That was the plan. In actuality getting Windows Server 2008 to work on the M4400 may not fly in the end. My first stumbling block is a big one. Network connectivity. I can’t seem to get the ethernet connection to work nor the Intel 5300 Wifi. Searching around it looks like I’m not the only one. I found some instructions that claim people have gotten it working, but I am unable to replicate.

No connectivity in 2008 makes the laptop an expensive paper weight these days. I decided to switch gears and install Windows 7. With both Windows 7 and Server 2008, installation has finally become relatively painless. They’ve taken a page from MacOS X and only require a few clicks from the user to get the OS installed. What was really fantastic about Windows 7 is so far all my devices seem to work. Wireless, ethernet, etc. That is a nice treat after wrestling with 2008.

I plan to try to use Windows 7 as my primary OS on the machine. Especially since currently it’s the only OS with connectivity. The first thing that struck me is a new window management feature. The behavior reminds me of a freeware app I’ve used in XP called WinSplit Revolution. In Windows 7 you can drag one window to the left side of the screen and a ghosted window appears to show you how it’s going to snap the size of the window to the left quadrant of the screen if you let go. Same for the right. So it’s easy to get two windows maximized side by side. Hot keys winkey+left arrow, winkey+right arrow makes this a snap!

The other feature I’m on the fence about is the new taskbar. The first thing I do when I get a new XP machine is go into System->Advanced->Performance->Settings and change to adjust for best performance. This turns all the eye candy off and makes the tasbar nice and simple. I also right click on my taskbar, select properties and uncheck group similar icons. I am addicted to having a lot of windows open, and I find grouped windows make finding a particular window more difficult. This may have changed for me in Win7. When I mouse over a set of grouped windows in the taskbar now, I get a nice live preview thumbnail of what’s going on in each of the grouped windows. I can easily click on the large thumbnail of the window I’d like to switch to. I frequently have a number of file explorer windows open and this seems to be making it easy to find the one I want instead of lazily opening yet another explorer window.

Although I haven’t had time to explore this feature more than a cursory check, holly firewall batman. FINALLY a non-neutered firewall that feels first class. When you have a background of using a *nix firewall, like for me, ipfw in FreeBSD, you get quickly frustrated when you find windows barely has a thimble full of those features. Not to mention there isn’t a free firewall on the market I’ve found that will allow you to create a deny all rule and than explicitly allow only certain traffic. Yes I’ve used the ipfw port for windows, but really, why doesn’t something like ipfw exist on windows with a warm fuzzy gui with a price of $0. Well now it seems like it does! Good job Microsoft.

On the other hand, UAC is still there. And although reportedly not as annoying, it still gets in the way.

Guess we’ll see. I haven’t got a lot of experience with Vista, but from listening to the grapevine, Windows 7 might be the real XP upgrade path. Vista may be remembered like Windows ME, a bad memory.

wolf Uncategorized

I’ve been using FinalBuilder for a while now. It’s icon based scripting language for creating build scripts. It reminds me me of a multimedia language I used a long time ago called IconAuthor. Initially I had resisted the tool because I thought it might be too easy. But really when you think about creating build scripts, a good portion of it is creating thin wrappers around existing utilities. FinalBuilder does a good job of this and is flexible enough to create build script engines that have their own configuration and work with whatever process you come up with.

To create a ’subroutine’ or function in FinalBuilder, you have a few options. I like creating new stand alone FinalBuilder project files that are included by my main project. For instance, I’ve created a project that figures out if there have been any changes in the source control since the last build and creates a changelog. This is functionality I can use in any build script. Rather than cutting and pasting it in, I can just reference the project ’subroutine’ I created for it.

One limitation to this method is there is no such thing as function signature. If your subroutine project requires the parent project to define some variables, you need to add your own logic if you want to make sure some variable has been defined. If you don’t do this, you can get unexpected results and you’ll need to debug what went wrong. “Oh I forgot to define the BuildDirectory variable in my parent script!”

What I decided to do is create some pre-canned functionality to just check to make sure the specified required variables exist and have non-empty value. I actually initially wrote this as a separate FinalBuilder project. I thought about it for a while and decided this might be better as it’s own custom action. FinalBuilder supports developer created actions and even provides a nice IDE to help author them. You have the option of using VBScript, JavaScript, Powershell, and .NET. I went with JavaScript since this action didn’t need to be too complicated.

Along with the Custom Action I built a simple example and an attempt at unit testing the action. There is no built in way to do unit testing in FinalBuilder, but I took at a stab at creating a simple structure to support it. I have 8 tests which are included in a single FinalBuilder project file.

I’ve hosted the project and source at https://launchpad.net/wolf-fbcustomactions. The first release can be downloaded from http://www.wolfplusplus.com/projects/WolfFBCustomActions-0.1.7z.

wolf Uncategorized

Wow, this problem took way too long to track down. I was creating a build script for a legacy VS2003 Web App project. It is being migrated from Perforce and given an official build script. The command line build was failing. When I opened the solution in the VS2003 IDE, I was plagued with the following dialog:

—————————
Microsoft Development Environment
—————————
Refreshing the project failed. Unable to retrieve folder information from the server.
—————————
OK
—————————

I tried a number of different things. One thing I suspected was maybe IIS was having a problem with the NTFS permissions since my virtual directory was pointing to a different drive. It lead me to a cool post on expert’s exchange which gave instructions on how to use this cool open source utility SetACL to copy permissions from one folder to another. Using robocopy /copyall doesn’t copy inherited permissions, but SetACL can. It wasn’t a permissions problem, but I’m sure I’ll need SetACL at some point.

Googling the erorr message yields advice to delete your C:\Documents and Settings\(username)\VSWebCache folder. This isn’t bad advice, but didn’t solve my particular problem. It turns out VS2003 Web projects don’t like folders that begin with a dot. Subversion by default litters every source controlled directory with hidden .svn folders. Subversion has a FAQ topic about this problem. To make subversion use _svn folders intead of .svn folders, set an environment variable named SVN_ASP_DOT_NET_HACK to any value. You may also want to enable this option in TortoiseSVN.

Very annoying problem solved.

wolf Uncategorized