Application Compatibility Testing -- Overview

  • Article

Delivering a new release of Windows includes a major effort to insure that applications continue to function as well on the new release as they have on the previous release. At the PDC we talked about some of the new areas of Windows Vista that reduced this level of compatibility, such as changes we made around the OS security model. With Windows 7 we renewed our engineering efforts to maintain compatibility. As with device testing, compatibility testing is an effort that spans the entire engineering organization, though we also have a group that is dedicated to this effort. This post was authored by a set of folks and coordinated by Grant George, the corporate vice president for testing in the Windows Experience team. --Steven

We have taken a very proactive approach to Application Compatibility and our process starts from the moment we first plan our product schedule and design and check in code for Windows and runs through all of our engineering processes and disciplines leading up to our final release (and beyond).

Application Compatibility Goals for Windows 7

Our main Application Compatibility goal for Windows 7 is to make sure that most all applications which work on Windows Vista will continue to work seamlessly on Windows 7. We do have to be careful about making this claim to be universal because there is a class of applications that are always updated in tandem with a new Windows release. These applications are primarily system utilities, diagnostics, and security software—the common thread is that they make assumptions about the underlying implementation of Windows internals and thus require updates. We carefully coordinate with a large set of ISVs who provide these applications. This was talked about earlier this month as we announced our Ecosystem Readiness Program, which we will discuss more below.

At the start of our product cycle we review our new features and changed designs to ensure that every element of Windows 7 has Application Compatibility in mind. Our engineering process includes automated quality checks to assure public APIs don’t change, and our test engineers have the right tools, engineering time and information that is used to find application issues as early as possible in our development cycle. Telemetry information is collected to assess and prioritize the breadth of applications our users depend on, paired with market data and install base information, across a wide variety of software categories to make sure they work as expected in our new OS version.

Below we expand on how we deliver on this goal.

Engineering Application Compatibility

Rich telemetry from external usage and the broad software marketplace helps us to provide a list of the most popular and critical applications, updated frequently throughout our development cycle. Our engineers then acquire these applications and build automated tests that verify each one works as expected on Windows 7.

Changes that could impact application compatibility are followed up closely, for example Legacy Code Removal which involves removing code that existed in the previous release of the product is strictly guarded and tightly managed and should not happen without proper documentation and engagement with the impacted parties. For example if we need to deprecate an API call, the documentation for this API will be updated and we will wait until the impact of the removal is minimal as indicated by telemetry data unless it is required sooner to fix a security issue.

Throughout the development process we are running tests in the background creating an ongoing validation of new code relative to application compatibility. As code is getting ready to be checked into the main build, if a compatibility failure is detected in an automated regression test the checkin is halted. At that point the code is scanned for known compatibility issues and if an issue is detected the developer is asked to fix the problem. Of course we also develop new tests throughout the course of developing Windows 7 in order to broaden our coverage of third party software.

Engineering & Testing Strategy

We have several teams dedicated to application compatibility as part of the Windows development effort. These teams provides guidance on how to build in application compatibility, provides data on application dependencies, and information on what applications may be impacted for any particular change we make in the Windows platform. These teams also reviews new feature designs, as well as other planned changes, to ensure that the engineering team has fully taken application compatibility into account so that we achieve the tenet of "keeping applications working on Windows 7".

In addition to working with the internal Windows engineering teams, we also reach out to 3rd party developers writing Windows applications to ensure these partners have all the information they need to make their solutions fully compatible with Windows Vista and Windows 7. Furthermore if we do uncover any issues that may need to be resolved by the 3rd party developers, we collect all the information, resources and guidance and engage in a conversation with those external developers to help them understand and fix these issues.

We recently announced the Windows 7 Ecosystem Readiness Program. This program provides partners with access to Windows 7 builds and tools they need to test solutions for Windows 7, Windows Vista, Windows Server 2008 and Windows Server 2008 R2. The Ecosystem Readiness Program also facilitates testing multiple components of the ecosystem together to improve the overall user experience. Rather than just focusing on getting a specific OEM product, software application, or hardware device certified, we will be bringing multiple components together to verify a rich user experience that delivers quality, reliability, and performance as well as innovation through new feature adoption.

As mentioned earlier we use telemetry data and market share information to choose the applications we directly test on as part of our compatibility efforts, below you can find the a sample of the Consumer Scenarios we focus our testing on:

  • Communications
  • Gaming
  • Fundamentals like Setup, Security and Performance
  • Memories
  • Music
  • Productivity
  • TV/Movies
  • Data Backup/Security
  • Mobility
  • Financial Management

To view a full list of Application Categories covered please take a look at the Appendix section at the end of this post. While we would love to provide the detailed list of products, several of the sources of this information are based on proprietary research from third parties.

Another very important set of technologies we test on are the middle tier technologies like Java, the .Net Framework, etc. to make sure the applications that use these technologies continue working as expected

In addition to 3rd party stand alone applications we test a subset of OEM pre-installed software and their inbox applications for compatibility. The software tested come from the engagements we have with our OEM partners and their submitted installation images. These images are tested on clean installations of Windows 7 and upgrades from Windows Vista on OPEM standard hardware. This level of coverage allows us to best replicate the initial experience with Windows 7 for many of our customers. Because many of these applications are closely aligned with the OS, hardware and drivers, it is not unusual for an OEM to provide updates to this software with a new OS release.

In addition to the above mentioned testing approaches, Microsoft IT maintains a software portfolio of approximately 1,500 applications. These applications must be tested prior to software deployments inside Microsoft.

Microsoft IT developed an application-tracking method that simplified the process of selecting applications for sample-based testing. By identifying groups of applications that have similar data processing, controls, underlying technology, and methods, Microsoft IT is able to test approximately 4 to 6 percent of the total applications and gain a reasonable assurance of compatibility for all.

For more information you can read the LOB Application Compatibility Technical White Paper.

Validation Strategy

Part of our testing process includes the creation of scenarios that we validate on 3rd party applications. We approach this by verifying the intended functionality of the application while focusing some of our attention to changes in the OS, new functionality and risky integration areas. Manual and automated test passes are scheduled to cover the identified scenarios and to verify the user experience. We cover the applications on different sets of hardware and make sure that that we cover a lot of different configurations, x86, x64, Intel, AMD, touch and multi -touch machines, etc.

We use specific categories when measuring the compatibility of the applications we test on:

  • Excellent - No compatibility problems
  • Poor - Minor functionality is not working
  • Failure (regression) - It used to work in Windows Vista now it is not working properly
  • Failures (non-regression) - The application does not work on Windows 7 but it did not work on Windows Vista either
  • Accepted Regression - We know of changes in the OS that may affect the applications and we have/are working with ISVs on mitigations for these issues. A simple example is version number. An application that is hard coded to only work on specific versions and it doesn’t know what our new operating system version is would fail.

We will cover these categories in more detail in subsequent blog posts about how we manage application compatibility.

As part of our testing we do find issues that may need to be resolved by the 3rd party developers at the companies that develop and sell these applications. We collect all of the relevant technical information, resources and guidance and engage in a conversation with those external partners to help them fix the issues. Of course we also engage them in our technical beta programs so they can test Windows 7 compatibility with their products at the same time we do.

Final Word

Application compatibility is very important to the Windows team. We are constantly working to improve your experience with applications as you move from one release of Windows to the next. We encourage you to try our Windows 7 beta release to experience the improvements in the application compatibility space and we want to hear your feedback.

It is worth mentioning the work we have done from an end-user perspective to assist in application compatibility. Many failures of application compatibility happen at install time and to assist with this, in Windows 7 we have improved the detection of failed installations and provided a step by step wizard which will help to get an application’s “compatibility mode” correctly set. We also provide real-time problem reports and solutions that can help you locate an updated version or patch. Many of you might have experienced this when trying to run Skype after an upgrade or install the current version, and in both cases you were automatically referred to the Beta download site.

Appendix

Microsoft Covered Consumer Scenarios & Application Categories

Table of Microsoft Covered Consumer Scenarios and Application Categories

Our list includes the top 50% best selling applications in the past 24 months, some of this data is collected and aggregated from several well-known third party research information providers.

Comments

  • Anonymous
    March 02, 2009
    this is my feedback and my forum frends Beta 7000  (Italian Feedback) http://tinyurl.com/bqz8u5  

  • Anonymous
    March 03, 2009
    All my WDM softsynths fail to work on Windows 7 but work flawlessly on Windows Vista. In Vista the UI to select the softsynth was gone but with community replacements like "MIDI Device Picker" (search a bit), it was fixed. Now after installing my favorite Roland Virtual Sound Canvas VSC-MP1 or WinGroove or Yamaha XG Softsynth on Windows 7, I find they no longer show up. My sound card's FM synthesizer also does not work. Tried the 32-bit PDC build, beta everything. Has this been fixed in RC1?

  • Anonymous
    March 03, 2009
    The comment has been removed

  • Anonymous
    March 03, 2009
    Won't removing Microsoft Agent break some apps? Moreover installing Agent from the 2000-ish redistributable succeeds however the Agent characters lose their transparency (looks very ugly). The Vista release of the Agent binaries doesn't have this problem but it isn't redistributable.

  • Anonymous
    March 03, 2009
    "A simple example is version number. An application that is hard coded to only work on specific versions and it doesn’t know what our new operating system version is would fail." I still find it very strange that Windows 7 is internally 6.1. I agree that it is indeed "6.1", but if marketing decides that the next version is called "Windows 7" the version number should reflect that. But you guys do a good job. This is one of the most difficult things to work on. And you're doing great. Is it getting easier with each new Windows release? Is Windows 7 an "easy" release for you since there aren't many fundamental changes?

  • Anonymous
    March 03, 2009
    The VPN client of my university by Cisco didn't work under Windows 7. I wasn't able to boot into Windows 7 because right before the login screen the VPN client caused a bluescreen. There wasn't anything I could do except using one of the recovery tools Windows provides to recover an earlier state of the OS.

  • Anonymous
    March 03, 2009
    The Vista version of Inkball is not compatible with Windows 7.  Inkball was excluded from the Windows 7 beta.  Inkball is a great game that should be playable on future versions of windows.  Would you reconsider including it in the final release.  Alot of people will be disappointed by it's omission. Perhaps you can release downloadable non-drm x86 and x64 versions.  Thanks.

  • Anonymous
    March 03, 2009
    Um - what about compatibility with apps that work in XP? A smooth upgrade from Vista won't be what MS has to sell us in the corporate world, but a hassle-free move from XP. MS has to convince us to move from XP. Merely releasing 7 as an upgrade to Vista isn't the deal.

  • Anonymous
    March 03, 2009
    The comment has been removed

  • Anonymous
    March 03, 2009
    Among apps, I haven't found anything that is compatible with Vista but not compatible with Windows 7. Inkball, talked about above, is the only thing I've heard about.

  • Anonymous
    March 03, 2009
    here comes the "Why doesn't my 12 year old 32-bit application not work on Windows 7" comments. Their are people still lamenting the death of 16-bit application support in Windows.

  • Anonymous
    March 03, 2009
    I was trying to install the latest version of ISOrecorder, but came across the "this product must be installed on Vista"-type error message.  How is Microsoft classifying errors such as these?  The product works under Win7, I'm sure, but won't install because it detects the wrong OS.

  • Anonymous
    March 03, 2009
    i'am sorry if this dont belongs here but i fond no other place @solaris:

  1.  Uninstall any previous version of Cisco VPN client that you may have installed
  2.  Reboot
  3.  Download the Citrix DNE Update and install it. or
  4.  Download the Cisco DNE update and install it.
  5.  Reboot
  6.  Take ownership and delete ndis.sys (in c:windowssystem32drivers).
  7.  Take ownership and delete ndis.sys.mui (in c:windowssystem32driversen-us).
  8.  Install Cisco VPN Client 5.0.04.0300.
  9.  Reboot
  10.  Windows 7 will repair itself (should take a few seconds) and automatically reboot.
  11.  Cisco VPN Client should work without any other tweaks. this worked for me...but no guarantee ;)

  • Anonymous
    March 03, 2009
    "someone" wrote:   "All my WDM softsynths fail to work on Windows 7 but work flawlessly on Windows Vista. In Vista the UI to select the softsynth was gone but with community replacements like "MIDI Device Picker" (search a bit), it was fixed. Now after installing my favorite Roland Virtual Sound Canvas VSC-MP1 or WinGroove or Yamaha XG Softsynth on Windows 7, I find they no longer show up. My sound card's FM synthesizer also does not work. Tried the 32-bit PDC build, beta everything. Has this been fixed in RC1? Tuesday, March 03, 2009 9:09 AM by someone" Hey "someone" -- we've been trying to find a repro for this issue that we first heard about in Larry Osterman's blog.  I'd love to find out more about your specific HW and config to see if we can get to the root cause?   -Steve Ball from the Windows Sound team.

  • Anonymous
    March 03, 2009
    @gormed Thanks for the note. Please use the Send Feedback button to submit your report with the details above.

  • Anonymous
    March 04, 2009
    Suggestion:

Integration WordPad with Notepad... Two editors built-in Windows, too many, too stupid... (After Integration rename application? use a all-new name?) (1) Integration WordPad with Notepad (2) complete support Multi-Language (ANSI Code). support read SJIS, Big5, GB, KOR, etc... (MacOSX can, Windows can't)

  • Anonymous
    March 04, 2009
    Suggestion:

Add Virtual Desktop http://www.windows7taskforce.com/view/1553 http://www.windows7taskforce.com/view/143 Like: http://www.youtube.com/watch?v=TM6HXoa0Lgk Unix/Liunx has Virtual Desktop MacOSX 10.4 Added Virtual Desktop Windows: None

  • Anonymous
    March 04, 2009
    Suggestion:

Add Toggle option: show thumbnails in 'Details', 'List', 'Small Icons' view mode in other system desktop environment, 'Details', 'List', 'Small Icons' view mode also show thumbnails. Includes all picture files, video files, text files, etc... but Windows can't. add custom option by user to toggle show thumbnails in 'Details', 'List', 'Small Icons' view mode look screenshot: http://uploads.bizhat.com/view/383070

keep Start Menu and Applaction Menu style consistency There is no consistency between the Start Menu and the Applaction Menu. To keep Start Menu and Applaction Menu style consistency, like Windows XP. All use white color for background, light blue color for selected item. to improve it. screenshot: http://uploads.bizhat.com/view/387119 http://www.badongo.com/file/13695750

  • Anonymous
    March 04, 2009
    @Domenico yeah, I know Longhorn (WinHEC 2003) and Longhorn concept: http://www.youtube.com/watch?v=b9ifQvQCO7Y but it's freeze by MS.

  • Anonymous
    March 04, 2009
    @Domenico yeah, I know Longhorn (WinHEC 2003) and Longhorn concept: http://www.youtube.com/watch?v=b9ifQvQCO7Y but it's freeze by MS.

  • Anonymous
    March 04, 2009
    The comment has been removed

  • Anonymous
    March 04, 2009
    @gormed: thanks for your suggestions. But I won't experiment with it and risk another total crash of the OS.

  • Anonymous
    March 05, 2009
    Sounds interesting and you make it sounds like I should be part of the team :)

  • Anonymous
    March 05, 2009
    Suggestion:

Improve Start Menu and Appliaction Menu color style (suggestion consistency) There is no consistency between the Start Menu and the Appliaction Menu. Suggestion to keep Start Menu and Appliaction Menu color style consistency, like Windows XP. Suggestion all use white color for background, light blue color for selected item. to improve Appliaction Menu color. (discriminated Appliaction Menu color very blurred) screenshot: http://uploads.bizhat.com/view/387397 http://www.badongo.com/file/13707468

  • Anonymous
    March 05, 2009
    Can you guys try to get Bitdefender free V10 to work with Windows 7 64 bit. I use the antivirus software and its 32 bit only. It would be nice if there was a way to get it to work in Windows 7 64 Bit.

  • Anonymous
    March 06, 2009
    Autotune (VST version) doesn't work in Win7

  • Anonymous
    March 07, 2009
    I thought Y2K problem was solved in 1999 (That's why We all still alive :). But Microsoft continues support for programs solving that problem. Look at last record in Fundamentals scenario in the picture above. Microsoft is very serious about Y2K problem, so Y2k Solutions presents in the 'Data Security And Backup' scenario too. That's what I call Compatibility :-)

  • Anonymous
    March 09, 2009
    > Their are people still lamenting the death > of 16-bit application support in Windows. Say the company sends invoices using a 16-bit application.  There is no business need to replace the application. Assuming they want to switch to Vista or W7, how would they go about it? Some people tried, and they got burned.  And that was in better economic times :)

  • Anonymous
    March 09, 2009
    I'm wondering if it's possible to successfully network a win98 machine to Windows 7? I'm looking at purchasing some PCs w/Windows 7 for a small business that presently has primarily XP Pro machines except for 2 PCs that run win98. The 98 boxes run proprietary software that controls production machines in the shop and they cannot be upgraded. They do however need access to files that would live on the Windows 7 machines. If anyone could provide feedback on: a) it is possible b) what extra steps would be required to get this working Thanks for any information you can provide!  

  • Anonymous
    March 19, 2009
    The comment has been removed

  • Anonymous
    April 06, 2009
    It would seem easier to help test if we had access to current builds. As of now, only build 7000 is available for download. It is rumored that build 7105 should be "leaked" soon. If we are testing, it seems that it would be more relevant if we were testing using the latest build. Is there a way to get access to these newer builds?

  • Anonymous
    May 06, 2009
    The comment has been removed

  • Anonymous
    May 23, 2009
    Big Hands to all for a lot of inspiration.

  • Anonymous
    May 25, 2009
    I have identified an alarming problem with Windows 7 release candidate… Up until now we have instructed our application’s users to select “Run as Administrator” and the application has worked well. Now that our application can be run as a Standard User some of our users are still setting it to “Run as Administrator” because that is what they are used to. This is the problem, they can never undo the “Run as Administrator” and go to back to normal. The application uses the All Users directory for the application’s common data and the path to that directory is no longer valid under this scenario. Uninstalling and reinstalling the application does not fix the problem.

  • Anonymous
    June 17, 2009
    The application uses the All Users directory for the application’s common data and the path to that directory is no longer valid under this scenario. Uninstalling and reinstalling the application does not fix the problem.

  • Anonymous
    June 17, 2009
    If we are testing, it seems that it would be more relevant if we were testing using the latest build. Is there a way to get access to these newer builds?

  • Anonymous
    June 17, 2009
    The 98 boxes run proprietary software that controls production machines in the shop and they cannot be upgraded. They do however need access to files that would live on the Windows 7 machines.

  • Anonymous
    June 17, 2009
    Microsoft is very serious about Y2K problem, so Y2k Solutions presents in the 'Data Security And Backup' scenario too.

  • Anonymous
    June 17, 2009
    There is no consistency between the Start Menu and the Applaction Menu.

  • Anonymous
    August 24, 2009
    Awesome stuff, i'm just concerned that compatibility won't be as 'good' as microsoft claims. I guess only time will tell.

  • Anonymous
    August 22, 2010
    halo 2 for windows vista is not compatible for my windows 7

  • Anonymous
    February 17, 2011
    The comment has been removed

  • Anonymous
    February 17, 2011
    The comment has been removed

  • Anonymous
    March 07, 2011
    The comment has been removed

  • Anonymous
    March 09, 2011
    When I open this game on windows 7, the screen goes black and returns to the home page twice. There is a little icon that is at the bottom of the screen that says that the running program isn't compatible with certain visual elements of windows. What is wrong and how do I fix the problem?

  • Anonymous
    March 11, 2011
    I've downloaded the Itouns program & now I get a message saying that it needs the saport area along with the help area. How do I get it so I can use the Itunes program. Can anyone help me if you can PLEASE email me at galen612@yahoo.com

  • Anonymous
    January 06, 2015
    attempting to install Print Shop version 3.5  will not complete instllation.

  • Anonymous
    January 26, 2015
    EVER SINCE UPDATE, MALWAYBYTES WILL NOT WORK??? HELP