How to shoot yourself in the foot writing installer upgrades

I’m in the middle of a development cycle for one of our products, when QA logged an unexpected bug.  When you upgraded an existing version of the product to the new one, only some of the settings were being saved.  This was a new bug, something I did recently broke the installer.

A little background on the product.  It’s a set of data collection services that allow our Onscreen product to work with various 3rd party GPS vendors.  I have a generic collection service, plus a handful of services for vendors that can’t work with the generic service. Each collector is a service written with in C# for the .NET Framework.  Most of the settings for each collector are stored in the service’s app.config file.

I have written the installer so that it does a full install each time.  If it detects a previous version of the product, it caches the service status (running, disabled, etc) and the app.config file for each service.  It puts the cached files in a temp folder that will get purged at the end of the installation. Then it does a silent uninstall of the previous version and installs the new version.

After the files for the new version have been installed, the installer restores the service status for each collector service and retrieve the cached settings.  I don’t copy the old file over the new, I just retrieve a set of settings and update the default values in the new app.config with the cached values from the previous one.  You don’t want to blindly copy over the app.config, you would wipe out new or changed settings that are required for the new version.

I wrote a simple command line app that gets called by the installer.  it reads the new app.config, the cached copy, and a set of rules in XML format.  For each new file, it looks for the same file in the temp folder created by the installer.  It then does a XPath search and replace for each rule.  The rules are a little flexible.  It can be directed to replace the default with the cached copy or replace it with a new default value.

When I need to do some file manipulation, I usually write a small command line app instead of trying to have the installer code try to do it.  There’s a couple of reasons for this.  By using a separate app, I get to code and test the file manipulation outside of the installer environment.  Having an installer locate and enumerate a set of files, and then update them based on the contents of others would be a difficult to write and maintain.

Getting back to bug at hand, the installer installs five services by default.  Three of them were getting their settings persisted across upgrades, two were not.  That threw me, for this type of activity usually everything works or everything breaks.  I spent some time tracing through the code, which is always fun with an installer.  I ended up spending quality time with a VM of Server 2008 and had my command line app write detailed information to the Windows Event Log.  Crude, but effective.

As it turns out, I created the problem at the start of this development cycle.  I had renamed the executables as part of a rebranding exercise.  When the the code when to locate the cached copy of the service, the new name didn’t match the cached name.   Yes sir, I had shot myself in the foot with a device of my own making. 

I had to tweak my command line app a bit.  As it processed each app.config file, I added a some additional code so that it would make two attempts to load the cached copy.  First attempt would by the current name.  If that fails, it then uses the previous file name.  After that,we were back in business.

Installation is not configuration

Christopher Painter has a good post about the problems inherent with with having installers run SQL scripts.  Having an installer communicate with a database server just opens the door to all sorts of issues.  Just handling the connection to the server requires making sure that you have all of the required bits installed and that you can locate, and connect to the server.  None of that code is rocket science, we’ve been using it for years. I just don’t think having it in an installer is the right place for that type of code.

The problem is that you are running that code from inside a relatively fragile box, your installer.  Most installer authoring tools provide rudimentary support for running SQL scripts at install time.  They work just fine when all of your ducks are lined up in a row.   However your clients may be missing an odd duck or two, and you end up with fragile code wrapped inside a DLL that your installer will call.

I’ve always treated the installer package as just part of the actual installation process for the end user.  The main task of the installer is to get the bits of your application in place and handle any prerequisite runtime library your code may need.  When it comes to initializing or updating a database, I leave that to the main application and/or it’s support utilities.  if a SQL script is flawed and or doesn’t handle some edge condition that only one customer has, you can break the install.

By using a full blown application to handle the database task, your life (and your customer’s life) becomes much easier.  By running your own application, you have (or should have) a much richer environment for developing code.  You have complete control over the UI and you do not have to be concerned with trying match the UI style of the installer.   Also the testing and debugging of the database utility code becomes much easier as no longer need to account for the installer.

You also have the ability to run the application at any time after the install, without having to invoke the installer.  If it turns out the problem was a SQL script that didn’t work for that customer, you can immediately email or post online an updated SQL script, without forcing the user to run the installation process again.

We use a mixture of script based installers and Windows Installers and none of them make any attempt to run a SQL script.  I wrote a database utility application that gets launched after an install.  I wrote it when our company supported the MSDE, back in the day of SQL Server 7.  Back then, Microsoft provided absolutely zero for tools to manage the MSDE.  So this utility, by necessity needed to be able to attach and detach databases, back up and restore databases, manage the server and database logins, and perform schema updates. 

Over the years, this utility has matured and is very easy for the end user to use.  We provide all of schema updates in a single, compressed file.  For the use to apply an update the database schema, they just run my utility and it reads the update file and it knows which updates have already been applied and only runs the ones it needs to.

All very easy for the end use, but it would have been a nightmare to get that level of functionality running as part of the installer.  As soon as you add an outside dependency (in this case, the database server), you have added a point of failure that you will have absolutely no control over.

Saving application settings over installs

We have been using Windows Installer (WI) based setups for all of our newer applications.  I used to use Wise, but I have migrated our installers to InstallAware.  Why I made that change will be another blog post, but needless to say I had very good reasons to make the switch.  When we release a new version or upgrade for one of our applications, we do full installs.  We don't do patches.

WI technology supports the distribution of patch files.  If you are only updating one or two files, the patch file will only contain what has changed, greatly reducing the size of the installable bits needed to be distributed.  I did use patch files when I first started doing WI based setups, but I abandoned the practice early on.  Patches were hard to automate as part of the build process and they had to account for cumulative changes.  The other drawback was that they could add files, but not remove them.  Another drawback was that it yet another file extension, .msp instead of .msi, to explain to the users.

I decide to have the installers upgrade in place. When a new version was installed over a previous version, the previous version would be silently uninstalled, then the new version would be installed.  This basically gives you a clean slate per install, which is great because you only install the files that are needed and the dependant objects will be up to date.  The bad part is that the user settings will be removed as part of the previous versions uninstall.

What I needed to do was to persist the user defined settings.  Consider a web application, if all of the settings are in the web.config file, then you just need to keep that web.config file around.  The simple solution is to tell the installer not to delete that file on the uninstall.  There's a couple of problems with that approach.  The first one is that the new install is a full install and the user gets to pick the destination.  They may want to put the web application in a different folder or drive and I didn't want to take that option away just to make my job easier.

The other drawback was far more serious.  I could keep the web.config around between installs, but what if the new version added content to the web.config file?  You don't want to save the old settings and lose the new settings.  That could cause hard to diagnose bad things to happen.   What was needed was the ability to merge selected content from the old web.confile and into the new web.config.  You can do many things with WI technology and InstallAware provides powerful functions, but this is a task best left outside the installer code.

I decided to write a Win32 command line app that would be able to save the old web.config and merge selected values from that file into the new one.  The installer would be include that file and be able to run it without permanently installing it.  I could have used C# with .NET (and I even had most of the code already written), but I didn't want to deal with .NET security issues or make the .NET Framework a prerequisite for non .NET based installs.  Delphi 2007 provided everything that I needed.

From the installer, I call this app just before the previous version is uninstalled with a command line parameter to designate which app is being installed.  It will locate the current web.config file for that application and copy it to a folder in the user's temp directory.  I always have our installers write the location of the install folder to the registry, it makes tasks like this much simpler.  After the new version has been installed, I run the app one more time, with a "-restore" command line parameter.  It locates the cached copy of the old web.config and the new web.config and updates the new web.config file with the settings that need to be persisted across installs.  Once that task is complete, it removes that temporary folder.

The app knows what settings to use from rules encoded into the source code.  I could have used a file to store a list of settings to persist, but that would have made the installer a bit more complicated.  It's fairly easy to run an app from the installer without actually installing it, but if you want it to have it read an external file, then you need to make sure that file is in a place where it can be read from.  That comes one more part of the installer to be tested and the mechanism for handling that would be specific to installer tool used.  By making the app contain the rules it needed to follow, running it from the installer becomes a trivial task.

The other advantage of having a separate app to handle to the saving of the user settings is that the code is much easier to test and debug.   It becomes a modular piece of the install and can even be unit tested in a scriptable manner.  This type of code can be easily adapted to the types of application.  I use a similar type of program to persist settings and current run start for some of our Win32 service applications.