Windows Store app test cases

  • [アーティクル]

[ This article is for Windows 8.x and Windows Phone 8.x developers writing Windows Runtime apps. If you’re developing for Windows 10, see the latest documentation ]

Here's a set of test cases that you can use to test your app before you submit it to the Windows Store. Use these cases along with the guidance in Debugging and testing either to supplement your existing test process or to develop a new one.

Note  These test cases support app functionality that meets the latest Windows and Windows Phone Store Policies. Make sure to align any of your own test cases to those requirements whenever possible.

 

Use the links in the panel to the right to jump to the feature area that's relevant to your app. Identify the test cases that best apply to your app based on how it uses specific features, and then use the provided examples and resources to develop your own test routines.

A quick note on additional UX design testing

A complete functional review to check for critical bugs is an important step in completing the development of your app; you can't close the book on all user-experience (UX) concerns during the design phase. You have to implement some elements of your app design before you can see how they contribute to the user experience.

For examples of what to test for, see App UX test cases, a companion to the UX design checklist.

Store certification

Test case Details

Your app meets Windows Store app certification requirements.

Review the Windows and Windows Phone Store Policies and see if your app meets all applicable criteria prior to its final submission to the Windows Store.

Your app passes self-testing using the Windows App Certification Kit .

First, download the Windows App Certification Kit for the appropriate platform version (Windows 8 or Windows RT).

Next, test your app using the kit as described in Using the Windows App Certification Kit.

Your app starts within five seconds.

Your app can display a splash image while loading at Start, but the first app screen should still render within five seconds to support a responsive user experience.

Example test

  1. Start your app.
  2. Verify that the first screen is rendered within the five seconds after start.

 

Functionality and core scenarios

Test case Details

Your app fully implements all advertised features and functionality.

Example test

  1. Start your app.
  2. Test all primary features and functionality.
  3. Observed behavior should match expectations initially set for your users.

Your app supports primary orientation scenarios.

The behavior of your app should be consistent when responding to orientation changes.

Your app should not display a blank screen, blank tiles, or empty white rectangles.

Your app should maintain context. For example, your app should be in same state as it was in when the screen was locked and device orientation was changed.

If the user was watching a video when the orientation changed, playback should continue after the change.

Example test

  1. Start your app in landscape orientation.
  2. Lock the screen.
  3. Change the orientation of the device.
  4. Unlock the screen.
  5. Repeat for portrait orientation.

For more info, see Guidelines for layouts.

Your app uses the correct navigation UI element to move from any page back to the app homepage.

The primary method of navigation back to the app home page should be accomplished using the provided app bar button.

Example test

  1. Start your app
  2. Go to any second-level page.
  3. Navigate back to the app home page using the Home button on the app bar.

For more info, see Navigation patterns.

 

User input

Test case Details

Your app does not hide user input controls when the on-screen keyboard is in use, or as a result of changes to page dimensions after switching between portrait and landscape orientation.

User input, such as buttons or other text boxes, should not be obscured by the on-screen keyboard or as a result of orientation changes.

Example test

  1. Start your app.
  2. Place a focus within a text box.
  3. Observe the placement of UI elements with the on-screen keyboard active.
  4. Observe the placement of UI elements when your app transitions between portrait and landscape orientation with and without the on-screen keyboard.
  5. Where applicable, repeat the previous steps for other app pages.

For more info, see Guidelines for layouts.

Your app is fully functional using all supported input types.

The user experience should be consistent in quality across primary control methods supported by Windows 8.

Example test

  1. Start your app.
  2. Test app features with touch.
  3. Test app features with mouse and keyboard.
  4. Test content containers (for example, lists and grids) to see if the mouse wheel allows content scrolling.
  5. Test for basic keystroke navigation.

Responding to user interaction (JavaScript and HTML or C#/VB/C++ and XAML)

Your app disables a user-input element until the last interaction with that element has been processed.

After the user interacts with a UI element (for example, a button), that element should be disabled if the operation that it initiates requires completion before another call is made (for instance, a sign-in to a remote service).

Example test

  1. Start your app.
  2. Observe the app's response to actions like tapping or clicking a button several times in succession. Depending on the complexity of the operation that the UI element represents, these conditions can result in your app crashing.

Responding to user interaction (JavaScript and HTML or C#/VB/C++ and XAML)

Your app implements the provided app bars correctly.

If commands on the app bar are vital to the screen or further navigation, they should appear on the canvas.

Example test

  1. Start your app.
  2. Verify that both the top and bottom app bars are transient.
  3. Observe whether swiping up or down from the bottom or top of a touch display, right-clicks on the canvas, or selects items for which there are associated actions in the app bar results in the app bar appearing.
  4. Ensure that the app bars are not blank.
  5. Verify that taps or clicks made anywhere on the canvas dismiss the app bars.
  6. Repeat these tests for each app page.
Note  

Your app should not display the app bar programmatically.

 
Note  

Your app should not display app bar if there is nothing to show in it.

 

For more info, see Guidelines and checklist for app bars (JavaScript and HTML or C#/VB/C++ and XAML)

 

Connectivity

Test case Details

Your app behaves normally in environments where network connectivity isn't reliable.

Your app provides a relevant connectivity error message to the user when a connection is lost or reestablished.

Example test

  1. Confirm that the device has a working connection.
  2. Start your app.
  3. While your app is running, close the network connection.
  4. Continue using your app. Make sure to test functionality that requires connectivity.
Note  

One option for scenarios where connections aren't available is for your app to operate in an offline mode without alerting the user.

 

Your app behaves normally in airplane mode.

Example test

  1. Turn airplane mode on.
  2. Start your app.
  3. Perform an action that will cause a connection attempt.
  4. Test when on-connected on first run of the app and on subsequent app sessions.

 

Content interaction

Test case Details

Your app uses appropriate list-box interactions.

Items in a list-box element should change text color, but should not be active if there are no actions currently associated with the selection.

Example test

  1. Start your app.
  2. Find a list of objects that the user can tap to transition to a different page.
  3. Select one of the objects on a page by tapping.
  4. Select one of the objects on a page by swiping down or using right-click.

Your app does not trigger an action while a scrolling control is in use.

Ensure that items within a grid or scroll viewer are not triggered accidentally by events on their container controls.

Example test

  1. Start your app.
  2. Go to a page that uses a grid layout.
  3. Pan the grid in quick swipes on the area over a child control.
  4. If the grid reacts by scrolling, the child control should not be triggered.
  5. Repeat the previous steps on any app page that displays content with a scroll bar.

For more info, see Guidelines for panning.

Your app does not trigger an action if the user moves across a scrolling control grid populated with thumbnails.

When using touch to swipe left and right (to browse through thumbnails presented on a grid), the content moves left or right without activating a thumbnail.

Example test

  1. Place a finger on one of the thumbnails on a grid and drag the content sideways to swipe.
  2. After the swipe, lift the finger.
  3. Observe to see if the thumbnail is activated.
  4. Repeat the previous steps any app page that displays a grid populated with thumbnails.

For more info, see Guidelines for panning.

Your app supports hover feedback (a mouse scenario).

When the mouse cursor is placed over an item in a list or grid, the item shows a highlight.

Example test

  1. Start your app.
  2. Hover over various content items with the mouse cursor.
  3. Repeat the previous step on any app page that displays content with a scroll bar.

Your app items are not selected automatically.

Go through all pages in your app and verify that each item, especially the first item on a page, is not selected by default when the page loads or when page contents are refreshed by clicking the Refresh button on the app bar.

Example test

  1. Start your app
  2. Go through each page and observe app behavior.

 

Performance and stability

Test case Details

Your app successfully resumes from a suspended state.

The app should resume from the suspended state successfully. Your app must not crash on resume, and user must be able to continue using your app.

We recommend that your app resume at the same page it was on before the app was suspended, and with its UI in a similar state. For example, text entered by a user before app suspension is still present after the app resumes.

Example test

  1. Start your app.
  2. Take steps to get your app into a state in which data is being presented.
  3. Suspend the app (for example, press the power button on the device).
  4. Wait a few seconds for the device to go into a low power state.
  5. Press the power button to turn the device back on.
  6. Bring up your app and confirm that its state was persisted.
Note  

If your run the app from Microsoft Visual Studio, app suspend and app termination can be achieved via buttons on the app bar.

The suspend the application command from Visual Studio can also be used with any Windows 8 app by attaching to the process from Visual Studio via the Debug->Attach to process menu command.

 

For more info, see Guidelines for app suspend and resume.

Your app demonstrates smooth and fluid page transitions.

Verify that animations and transitions are smooth and fluid. This includes things like page animations or object animations. The overall quality of this experience is most important in highly visible areas like the app home page and any other page where a user is most likely spend a lot of time using the app.

For more info, see Animating your UI with the Animation Library and Stand out with styling and animation in your Windows Store app.

Your app provides clear and concise process feedback to the user.

Your app must appear responsive at all times. If content is slow to load, the app should indicate to the user that the action is in progress by displaying progress notifications via

UI elements.

Example test

  1. Start your app
  2. Carry out operations or request content that meet the conditions required to trigger any progress indicators (for example, a progress bar) that you have implemented to inform the user.
Note  

Some pages may rely on online content that takes time to load. The best practice is to navigate quickly to the page, then display a progress control as the content loads.

 

Your app does not crash.

Record in detail any crashes you encounter during app testing and resolve them before you submit the app to the Windows Store.

Example test

  1. Carefully document steps so that a crash can be reproduced later.
  2. Track a count of distinct crashes as your testing process continues. (For example, the app crashed three times during start-up and five times while the user was buying an item.)
  3. Develop processes for debugging these crashes, using your records.

Your app manages memory consumption effectively.

If memory usage continues to increase as your app is being used, and doesn't show indications of a decrease, there is a chance that there is a memory leak.

Example test

  1. Start your app.
  2. Start Task Manager to observe app memory usage.
  3. Complete one or more usage scenarios to verify memory consumption (for example, go to and back from several app pages).

Your app uses an appropriate amount of CPU resources while idle.

When idle your app shouldn't consume an inappropriate amount of CPU resources. You can use Task Manager (preferably tm.exe to taskmgr.exe) to test CPU usage.

Example test

  1. Start your app and allow it time to load the content.
  2. Go to a few different pages and let the app settle down.
  3. Verify that your app does not consume an unexpectedly high amount of CPU resources while idle. (For example, start Task Manager and observe your app's resource usage.)
Note  

Ideally, when an app is idle, it consumes 0% of CPU resources.

 

Your app uses appropriate methods of data caching.

Your app should use app data caching appropriately to avoid delays in loading content or excessive use of a network connection.

 

Games

Test case Details

Your game app can save the current game state if the app is suspended.

Game-state info should persist through app suspension. This includes info like the screen state, the score, level progress, and number of lives.

Example test

  1. Start your game and start playing.
  2. Play the game for a while and observe the current game state (such as score, screen state, items, and level progress.)
  3. Suspend the game and return to it. (To do this, press the power button on the device and wait 10 seconds, and then press the power button again to wake the device.
  4. Start the game.
  5. Observe whether game state has been restored.

For more info, see Application lifecycle.

Your game makes global settings available through the Settings charm.

Games often have interactive controls for global settings on the canvas and this is acceptable as long as the same commands are also in the Settings charm.

Example test

  1. Start your game.
  2. Use the Settings charm to display settings.
  3. Confirm that global app settings are provided.

For more info, see Guidelines for app settings.

Your game app's audio effects and music don't play while the app is in the background.

The game's sound effects and media aren't playing while the game is in the background.

Example test

  1. Start your game
  2. Switch to an app that doesn't use sound or music and leave the game in the background.
  3. Verify that the game's audio cannot be heard.
  4. Return to the game and verify that the sound effects and music resume.

For more info, see Playing and previewing audio and video.

Your game app allows music from an app in the background to play in place of the game music.

You can play the game and listen to music from an app in the background. The GameEffects sounds are heard, but the GameMedia sounds are not.

Example test

  1. Start playing music in an app that's capable of playing music while in the background.
  2. Start a game with GameEffects and GameMedia sounds.
Note  

Reinforce this behavior by adding the correct msAudioCategory tag to your audio element.

 

For more info, see Playing and previewing audio and video.

 

Contracts and extensions

Test case Details

Your app uses the Share contract correctly.

If your app is a Share source, the content that it shares should be both meaningful and relevant. If your app is a Share target, the content preview should be rich and accurate.

Example test

  1. Start your app.
  2. Verify that your app supports sharing as a source.
  3. If your app is a Share source, attempt to share content from your app.
  4. If app is a Share target, Start another app and share content to your app.

For more info, see App contracts and extensions.

Your app uses the Search contract correctly.

If your app offers search, it should provide accurate search results that closely match results expected by the user for the initial query. If necessary, your app should provide additional methods of result filtering. If no results are returned, the Search panel should display a clear and concise message.

Example test

  1. Verify that your app supports searching content within the app.
  2. Access the Search charm. (On the keyboard, press Windows Logo key + Q)
  3. Type a search query that's context relevant to what your app provides (for example, if you're testing a news app, type in a news topic).
  4. Select your app on the search panel.
  5. Confirm that your app opens and displays search results.
  6. Verify that search results are displayed and that:
    • You can view and browse results.
    • You can open specific results.
    • You app indicates why a result matches the query.
    • You can navigate back to the previous page after looking at the details for a result.
    • Your app provides additional search behavior recommendations.
  7. Verify that white space in query strings is handled properly by entering a query string that begins with white space.

For more info, see App contracts and extensions.

Contracts and extensions

Your app should provide contract functionality such as Search and Share only where appropriate.

Example test

  1. Review your app's AppxManifest file to determine which contracts or extensions are implemented. Do this in Visual Studio by using the Store->Edit App Manifest command.
  2. Next, refer to App contracts and extensions when creating basic test scenarios for verifying the quality of each experience.
Note  

Avoid declaring contracts and extensions that enable functionality that your app doesn't actually implement.

 

For more info, see App contracts and extensions.

Your app uses the Settings charm correctly.

 Your app should provide the capability to adjust settings specific to the app using the Settings charm.

Example test

  1. Start your app.
  2. Verify that Settings flyout disappears when the user touches anywhere on the screen outside of the surface.
  3. Verify that your app reflects changes to a setting immediately.
  4. Verify that a new setting value is applied as soon as the user stops interacting with a particular setting control.
  5. Verify that the Settings flyout doesn't include a button to commit changes.
  6. Verify that recommended Settings flyouts are in use. A Settings flyout should be narrow (346 pixels) or wide (646 pixels), and should have a header that includes a back button, the name of the entry point that opened the flyout, and the app icon.
Note  

Handle the beforeshow and afterhide events to do control initialization and state serialization.

 

For more info, see Guidelines for app settings.

Your app provides users with control over the behavior of web-based file transfers created using the background transfer feature.

Confirm that control of a background transfer operation and indication of its progress are being relayed to the user.

Example test

  1. Start your app
  2. Confirm that the user is provided with controls required to construct and initiate background transfers.
  3. Confirm that a transfer begins when the control to begin a transfer is used.
  4. Confirm that there is a UI element that displays transfer progress info.
  5. Confirm that there is a way to cancel a transfer that's in progress.
  6. Confirm that the a transfer is canceled when the user indicates that it should be.
  7. Confirm that canceled transfers no longer appear in any UI elements used to track in-progress transfers.

Your app uses push notifications correctly to keep the user informed.

Your app should use push notifications where appropriate. A push notification demonstrates itself to user via a toast notification or a tile change (in text, a number, or a picture) in the Windows 8 Start experience.

Example test

  1. Start your app.
  2. Begin app activities that are known to elicit push notifications.
  3. Verify that notifications can be tapped or clicked for additional info.
  4. Review notifications for meaning and usefulness.
  5. Verify that notifications dismiss themselves after a period of time, if they're not interacted with.
  6. Verify that your app supports notifications for both small and wide tiles (where a wide tile is supported).

For more info, see Push notifications.

Your app provides a way to manage scheduled notifications.

Example test

  1. Start your app.
  2. Look for a way to check any future scheduled notifications.
  3. Look for a way to delete future scheduled notifications.

Note  

This test applies only to apps that use Scheduled Notifications. (for example, alarms or reminders).

 

For more info, see Scheduled Notifications.

 

Language and localization

Test case Details

Your app displays accurate info for all regional setting requirements.

Regional info displayed by your app should align to the regional formatting selected by the current user.

Example test

  1. Change the device region and language settings to a configuration supported by the app.
  2. Restart the device.
  3. Start your app.
  4. View as many UI elements as possible.
  5. Repeat for each supported region and language configuration.

Your app provides error messages that are contextual, user friendly, and not generic.

Error messages should very clearly define what the error was and what next steps are required.

Make sure that your app does not display empty pages. In situations where content is not available to display on a page, provide appropriate error or help text to the user.

Example test

  1. Start your app.
  2. Conduct usage exercises that result in an error message.
  3. Review the content of each message.
Note  

Error messages must not contain any code details or stack traces.

 

Your app provides accurate platform and app descriptors where applicable.

Any references to the platform should correctly reference Windows 8, and the app should reference itself accurately.

Example test

  1. Start your app.
  2. Access the Settings charm.
  3. Read any disclaimers, EULAs, or About text.
  4. Check links to external websites that contain app info.

Your app is fully localized to all languages it claims to support.

Verify that all strings have changed (are localized) and no hard-coded strings are present in the app. Confirm that all strings are completely visible and are not truncated.

Example test

  1. Change the region and language settings on the test device to a supported configuration for the app (for example, countries/regions where the app will be available).
  2. Restart the device.
  3. Start your app.
  4. View as many UI elements as possible.
  5. Check links to external websites, which may contain legal info that has not been localized.

Your app UI uses appropriate font types and sizes.

To make your app readable and appear consistent with broader Windows 8 design principles, confirm the use of the following font types and sizes:

  • Use Segoe UI (the primary Windows typeface) for UI elements such as buttons and date pickers.

  • Use Calibri for text that the user both reads and writes, such as email and chat.

  • Use Cambria for larger blocks of text, such as for a magazine or RSS reader.

For more info, see Guidelines for fonts.

 

Audio and video

Test case Details

Your app continues music playback while in the background.

If you develop a music app, it should enable users to listen to music while they work in other apps on the device.

Example test

  1. Start playing music in your app.
  2. Transition your app to the background.
  3. Start an app that doesn't use sound or music, leaving the app you're testing in the background.
  4. Observe background playback capabilities.

For more info, see Playing and previewing audio and video.

Your app correctly adjusts music playback volume for the duration of a voice call.

If your music app is capable of playback while in the background, it should decrease playback volume, or pause music playback, for the duration of a voice call started in another app.

Example test

  1. Start music playback in your app.
  2. Transition your app to the background.
  3. Start any VoIP-enabled app.
  4. Initiate or receive a voice call.
  5. Observe audio levels of the music playback.
Note  

Reinforce this behavior by adding the correct msAudioCategory tag to your audio element.

 

For more info, see Playing and previewing audio and video.

Your Voice over IP (VoIP)-enabled app maintains normal volume levels for the duration of a voice call.

This test case example is specific to the playback volume of a voice call when an app in the background is playing music.

While the app in the background needs to lower the music playback volume, your Voice over IP (VoIP)-enabled app keeps audio levels normal during voice calls.

Example test

  1. Start any music app and start playing music.
  2. Transition the music app to the background.
  3. Start your app.
  4. Start a voice call and observe volume levels.
Note  

Reinforce this behavior by adding the correct msAudioCategory tag to your audio element.

 

For more info, see Playing and previewing audio and video.

Your app enables the user to adjust audio playback volume using the volume control.

Your app should provide audio controls that allow the user to affect playback progress and volume levels. However, if the app doesn't support this functionality, the buttons should be disabled.

Example test

  1. Start your app.
  2. Begin audio playback.
  3. Adjust the volume control.
  4. Use next and previous track controls.

For more info, see Playing and previewing audio and video.