More about the impact of clicking Send Error Report

I blogged last week about how important it is to us for you to click "Send Error Report" when the Windows Error Reporting dialog boxes appear. It's probably changed now, but much to my amusement I discovered that yesterday if I simply typed "Send Error" into MSN Search or Google that my blog entry was the first hit!

I've had a few comments that I wanted to respond to, and I have some more data to share that Jeff and Jiange just sent me today.

More detailed data on how have we resolved the error reports for Team Foundation so far
Jeff and Jiange did some in mining to correlate the number of "Watson hits" with the resolution of the corresponding bugs. Every time someone clicks "Send Error Report" counts as 1 Hit. If you have the same crash and send in a report in multiple times, or multiple users hit the same crash, these will all (usually) correspond to a single Watson entry in our database (and a single bug). Sometimes we discover that we have opened more than one bugs to track hits which end up being the same underlying issue, so these show up as resolved as "duplicate" instead of "fixed"

The important observation from the data we have so far - is that according to the table below we're actually fixing 99% of all Watson hits for Visual Studio Team Foundation. Obviously this isn't a guarantee that you won't run into an issue that we haven't found or fixed yet, but I'm really happy to see that we've spent our time fixing the right issues so far.

Please read my earlier post to see what I found out when I checked to see if we made the right calls when we didn't resolve a bug as 'fixed'.

Can I use Windows Error Reporting in my own apps?
Alfred Wallace asked if there is a way he could use the Windows Error Reporting technology in his own applications. I haven't really looked into this too deeply myself, but we do offer this service to 3rd party developers from the Windows Quality Online Services web site. This page describes how you can start using Windows Error Reporting logs for yourself. I think I've heard talk of making this even more accessible in future, but I don't have any idea what that might look like.

Why don't my bugs get fixed?
Peter Richie commented that he didn't feel that the error reports he sent over the years for Visual Studio 6.0 had been addressed at all. I can appreciate how frustrating this can be - I've experienced this exact same scenario with other Microsoft products in the past (and some fairly recent ones too, but I'll refrain from naming them here). It's definitely true that different teams seem to be able to get higher fix rates than others for these "Watson" issues.

I don't know if it will help you Peter, but I tried to think through some reasons for why you might have observed and I think there are several. The first thing that springs to mind is that Visual Studio 6.0 is an old product now - and, as you know, is no longer under active development. Once we've shipped any product, the cost for delivering fixes for any bugs increases astronomically and in the past we haven't really had a great vehicle for shipping fixes for Visual Studios issues that we might not call "recall class". I'm really proud of our fix rate for issues we found in the Beta program, but I know that this will drop for issues we don't uncover until after RTM.

Visual Studio 2005 is the first version that we'll ship a service pack for - in the past we haven't done this, so the bugs have gone unaddressed. It's quite possible that your bugs were addressed in subsequent versions of Visual Studio - if you're still seeing the same crashes in Visual Studio 2005 that you saw in Visual Studio 6.0 please let me know and I will chase down the issue for you.

Secondly, the bugs were most likely to be addressed during the earlier years of Visual Studio 6.0's life span - but at that time the tools for handling Watson reports were really limited. Our infrastructure for handling Watson reports has been significantly improved over the years (take a look this entry on Chris Pratley's blog for some history). Today there are some really sophisticated tools to help me gather data, debug in the dump reports that get sent in to us, and even refine the data we collect for specific crash reports over time. Unfortunately, this probably wasn't available to help debug your VS6 issues.

I know that this won't satisify your frustration over these older bugs but I hope it illustrates how we're taking steps to do much better in future.