Jon Galloway had a posting about issues that he came across when he opened too many tabs in IE7. Apparently the Desktop ran out of heap space and Bad Things happened. Jeff Atwood came across the same problem. I remember reading Jeff's post back in October, but I didn't associate it with a problem that I was having.
I was finding my self rebooting my XP dev box a couple of times a day. The desktop was basically losing control and I couldn't do anything with the task bar or Start Menu. Part of the problem was due to wonky display drivers from Nvidia. I updated the drivers and the problem went mostly away. But not completely. I have a dual monitor setup, and I have a tendency to keep lots of things open on a desktop that spans both displays.
The longer I used Firefox, the less stable the system became. I just became used to restarting Firefox. One of the nicer things with Firefox is that you can restart it and come back to the same tabs loaded with the same pages. BTW, Jed Brown's Restart Firefox Add-on is very helpful when you want to bounce FireFox and still have the same 12 tabs open.
At any rate, I was seeing the same symptoms that Jon reported. There is a registry setting that you can edit that will allow you to specify the heap size reserved for the Desktop. The default size is 3MB, everybody seems to agree that 8MB is the way to go. The 3MB number was set for NT 3.51, back when we had a lot less memory in out machines. This is only applicable for 32-bit versions of Windows, the 64-bit version do not have these memory limitations.
Kevin Dente documented how to bump this value. To change this setting, you'll need to edit the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems and edit the "Windows" value. This value is (as Kevin put it) a "big honkin' string" and you only want to edit one portion of it. Look for the part that starts with "SharedSection". It will have three numbers, the middle one will probably be 3072. Change that to 8192. On my machine, that part changed from "SharedSection=1024,3072,512" to SharedSection=1024,8192,512. The YMMV rule is especially applicable here. Remember to reboot after making this change.
The first number is the shared heap size common to all desktops. The global handle table comes out of the shared heap. The second and third numbers in SharedSection represent the size of interactive and non-interactive desktop heaps (services that interact with the desktop). If terminal services are enabled, there may be a fourth number. You only want to change the second number.