How to shoot yourself in the foot writing installer upgrades

I’m in the middle of a development cycle for one of our products, when QA logged an unexpected bug.  When you upgraded an existing version of the product to the new one, only some of the settings were being saved.  This was a new bug, something I did recently broke the installer.

A little background on the product.  It’s a set of data collection services that allow our Onscreen product to work with various 3rd party GPS vendors.  I have a generic collection service, plus a handful of services for vendors that can’t work with the generic service. Each collector is a service written with in C# for the .NET Framework.  Most of the settings for each collector are stored in the service’s app.config file.

I have written the installer so that it does a full install each time.  If it detects a previous version of the product, it caches the service status (running, disabled, etc) and the app.config file for each service.  It puts the cached files in a temp folder that will get purged at the end of the installation. Then it does a silent uninstall of the previous version and installs the new version.

After the files for the new version have been installed, the installer restores the service status for each collector service and retrieve the cached settings.  I don’t copy the old file over the new, I just retrieve a set of settings and update the default values in the new app.config with the cached values from the previous one.  You don’t want to blindly copy over the app.config, you would wipe out new or changed settings that are required for the new version.

I wrote a simple command line app that gets called by the installer.  it reads the new app.config, the cached copy, and a set of rules in XML format.  For each new file, it looks for the same file in the temp folder created by the installer.  It then does a XPath search and replace for each rule.  The rules are a little flexible.  It can be directed to replace the default with the cached copy or replace it with a new default value.

When I need to do some file manipulation, I usually write a small command line app instead of trying to have the installer code try to do it.  There’s a couple of reasons for this.  By using a separate app, I get to code and test the file manipulation outside of the installer environment.  Having an installer locate and enumerate a set of files, and then update them based on the contents of others would be a difficult to write and maintain.

Getting back to bug at hand, the installer installs five services by default.  Three of them were getting their settings persisted across upgrades, two were not.  That threw me, for this type of activity usually everything works or everything breaks.  I spent some time tracing through the code, which is always fun with an installer.  I ended up spending quality time with a VM of Server 2008 and had my command line app write detailed information to the Windows Event Log.  Crude, but effective.

As it turns out, I created the problem at the start of this development cycle.  I had renamed the executables as part of a rebranding exercise.  When the the code when to locate the cached copy of the service, the new name didn’t match the cached name.   Yes sir, I had shot myself in the foot with a device of my own making. 

I had to tweak my command line app a bit.  As it processed each app.config file, I added a some additional code so that it would make two attempts to load the cached copy.  First attempt would by the current name.  If that fails, it then uses the previous file name.  After that,we were back in business.

A work around for Delphi 2007/2009 with Windows 7 64 bit

I just installed Windows 7, 64 bit edition so that I could code and test against this new operating system.  Visual Studio 2008 installed and ran without any incident, but I hit a snag with Delphi 2007.  Delphi 2007 installed just fine, but it would die when you ended a debugging session.  An assert error would get thrown and after you cleared the dialog, Delphi would be terminated.
The error was thrown by bordbk105N.dll and had the following text:

Assertion failure: "(!"SetThreadContext failed")"
in ..\win32src\thread32.cpp at line 412
Continue execution?

That was followed by a “Yes/No” set of buttons.  It didn’t matter which choice you made, it was pretty much game over for Delphi at this point. After getting this happen 3 times in a row (even with Delphi launched with “Run As Administrator”), I decided to check the Internet and see if this was unique to me and if there was a fix.
I checked Google and found a few hits.  One of them led me to a blog written by Olaf Monien, a German developer with close ties to the Delphi R & D department.  His post, “Delphi 2009 / Windows 7 / 64 bit Debugger Crash Workaround”, described the issue in great detail.  Apparently an incorrect call is being made to the SetThreadContext API call.  The debugger dll has an assert call based on the return value SetThreadContext().  It’s odd that production code went out with asserts enable, that’s usually used during testing.  Some people traced through the debugger code and found that if a single byte is changed in the dll, you could get by this error.  This byte changes the logic so it ignores the return code from SetThreadContext() and never hits the assert code.  You can read about how they came up with that idea here.
An enterprising programmer named “LordByte” wrote a handy little utility that will patch the debugger dll for both Delphi 2007 and Delphi 2009.  Olaf is hosting the patch tool on his blog and This file found a permanent home at Embarcadero and you can download it from Delphi_2007_2009_WOW64_Debugger_Fix.zip.  This works but you now have the added risk of actual errors being returned from that call to SetThreadContext() being ignored.  Since the alternative is an unusable Delphi, it’s a risk I’m going to have to take.

[Edited April 22nd, 2013]
While I no longer use Delphi, I hate to see a broken link.  I updated the download link for the debugger patch to it's current home at Embarcadero.

Using the test form from remote connections for .NET web services

I’m working on a web service using C# and targeting the .NET 2.0 Framework.  Nothing terribly fancy, but it has some code to log the caller’s IP address (via HttpContext.Current.Request.UserHostAddress).  While testing the code, I like using the test form functionality that .NET provides with web services.  By design, it only works locally.  If you try to use the test form from a remote machine, you’ll get the error:

The test form is only available for requests from the local machine

That’s fine, when you expose a web service you don’t want to make it too easy for people to start poking at your web service methods with a pointy stick.  Still, I wanted to test it from another machine and it saves time being able to enter in different values without having to code up a test framework.  Plus, I wanted to make sure that my code was reliably picking up the caller’s IP address.

After just a tiny bit of searching in the series of tubes, I found how to easily enable the web service test form for remote connections.  Juan Ignacio Gelos posted what you need to add to the web service web.config file to enable the test form.

<configuration>
    <system.web>
        <webServices>
            <protocols>
                <add name="HttpGet"/>
                <add name="HttpPost"/>
            </protocols>
        </webServices>
    </system.web>
</configuration>

You are still limited to run methods that have simple data types as input, but it’s very handy for testing.  Since this setting lives in web.config, it’s easy to remove it when I run a build.  I’m using FinalBuilder on a dedicated build box and it’s easy to clean up .config files so that developer settings get scrubbed before the files get packages into an installer.

Handy tip for debugging a Windows Service

Debugging a Windows Service is always a pain. You can't run a service like a regular application, you have to run it from Windows Service Control Manager (SCM) and then have your debugger attach to the process while it's running. The problem is that it's difficult to debug problems with the service startup as the debugger can't attach to the service in time.

I came across a tip on the .NET Tip of The Day site, "How to debug Windows Service startup". Basically, you just add a line that calls Debugger.Launch() or Debugger.Break() in your startup code. When your code hits one of those lines, the Visual Studio Just-In-Time Debugger dialog will be invoked and you can select your debugger to handle the error. That will allow you to continue along in the code and debug until the cows come home.

That works better than a service debugging tip I posted a couple of years back, calling the Sleep API in your startup code to allow enough time to attach a debugger to the service. That was a hack, this is much cleaner.

All in all, I still prefer to separate the functional code from the service specific code. I can then run that code from a desktop app, making it much easier to debug. That works about 99.9% of time. Every now and then, I do need to run the actual service code and the Debugger.XXXX() calls will make that task much easier.