XR-003: Title Quality for Submission

  • Article

Version 1.0, 3/1/2020

Titles must be functionally complete, testable and meet Xbox console quality standards when they are submitted for certification.

Functionally Complete and Testable: The title submission must include all client code, submission artifacts, and downloadable content. Partner services must be available and properly configured for testing and certification. Titles must package cleanly with no failures using the current version of Submission Validator when submitted to Microsoft, and they must include Submission Validator logs with their submission.

Xbox Quality Standards: Games on Xbox consoles must function correctly across game modes in a variety of scenarios to meet player needs.

  • Title Integrity: Users must be able to complete all game modes and the title must provide a consistently playable experience for players. A playable experience varies per title, but generally means no severe drops in frame rate, no freezes, impasses, bugs causing major progression hindrances, or graphical corruptions.
  • Options and Settings: Game Settings, such as options, controls, and languages are applied correctly when updated in game and after terminating and launching the title again. Default console settings are respected where appropriate.
  • Navigation and Content Availability: There are no dead ends in content, the game menus can be navigated and all content is available (where appropriate).
  • Supports players with low and high friend counts: experiences remain stable and function when a player has no friends or a large number of friends. Typically, a large number of friends can cause issues in areas with stats posting/retrieval
  • Gameplay functions with the allowed number of local or online multiplayer players: Using supported modes, the gameplay modes function using the supported mix of local or online players
  • Save Game Compatibility: Save games and/or player progress continue to function following content updates. Additionally, permanent save data loss must not occur when loading an updated save with the base disc version.
  • Suspend and Resume: Title progress is preserved when resuming from connected standby or suspension.

More Information

Inaccurate and incomplete submissions will be rejected. Submission artifacts include items such as Help content, metadata, and so on. The most current version of Submission Validator can be found on the Certification Tools page.

Submission Validator is run automatically as part of calling 'makepkg.exe pack'.

Submission Validator will check for:

  • Debug files—Compiler or linker generated source or debugger support files. These files are not allowed in compiled packages. File types not allowed: .PDB, .OBJ, .CPP, .C, .CS, .LIB, .H, .SYM, .MAP, debug_licensing_overrides.xml, and Thumb.db. Additionally, any EXE or DLL that is not an AMD64 native code binary will be flagged as part of this check.
  • Xbox background music services—Although Xbox One supports background audio, titles are not allowed to use it at this time.
  • Launch Link handlers—Titles are not permitted to register new Launch Link handlers (windows.protocol registration) on Xbox One. The Xbox One system registers a default protocol association on behalf of the app.
  • Single app per package—Titles are only allowed to have one application element per application manifest.
  • Title capabilities—Titles must register the capabilities required for their apps and must not exceed the minimum capabilities to deliver their advertised experience. Submission Validator will display the capabilities listed in the title against known capabilities but will not validate that the correct capabilities are used in relation to title code. For more information about capabilities, see the white paper App Manifest - Solid Information on Metadata (Developer Education Materials > All NDA Whitepapers).
  • Persistent local storage usage—exclusive app game titles must indicate in their manifest the amount of persistent local storage to be used. Titles cannot exceed 4 GB of storage. Intent: Ensure that titles are ready to launch when they are submitted for certification.

Additional Resources

Test Cases

003-02 Title Integrity

Test Steps

  1. Sign into an Xbox profile.
  2. Launch the title.
  3. Navigate through all menus and sub-menus, review all types of media and/or features, or enter and complete all game modes.
    • Games: Navigate, interact and complete all menu UI, extra content, single player modes, and multiplayer modes, including any additional features. Test in offline, online, split-screen, and system link modes, if applicable. Test multiplayer game modes with the maximum number of players whenever possible.
    • Apps: Navigate and interact with all different types of media content, including any additional features.

Expected Result
All titles must provide users with a reliable, fair, consistent, and complete Xbox entertainment experience.

Pass Examples

  1. The title can be completed in all modes.

Fail Examples

  1. The title crashes at the end of a given level.
  2. A particular mode cannot be completed if the user has already completed a different mode.

003-03 Options

Test Steps

  1. Sign in to an Xbox profile.
  2. Launch the title.
  3. Set a spread of various options across the test team such that every available value is tested for each option setting.
    • This must cover options both within the title and on any system settings that can be observed while the title is running.
    • Check that all settings and options within the title's menus are correctly reflected in the end-user experience (for example, brightness, contrast, audio volume, difficulty, and so on).
  4. Terminate the title and then ensure that all settings and options persist after re-launching the title.
  5. Have the test team change their option settings in all areas of the title, and verify that these are accurately reflected throughout the title.

Expected Result
All option settings within a title must work as expected and options and settings values must persist after re-launching the title.

Pass Examples

  1. With the in-game language option set to Dutch, the title uses only Dutch subtitles.

Fail Examples

  1. If the user inverts the horizontal or vertical camera controls with the pause menu, the camera controls do not change in-game.
  2. Any options values set during gameplay are reset to default after terminating and re-launching the title.

003-04 Language Support

Test Steps

  1. Sign in to an Xbox profile.
  2. Launch the title.
  3. Navigate to the console's language settings and set the language to match that of the console locale being used.
  4. Taking each in turn, ensure that the title boots correctly and exhibits the correct localization for each supported language in the title.
  5. Ensure that all game-critical text that is essential for progression within the title is legible.

Expected Result
Titles must boot correctly in all language settings, and they must exhibit the correct localization for all supported languages in-game. In areas where text for supported languages is displayed, the text must remain legible regardless of the display setup.

Pass Examples

  1. All localization displays correctly in all languages.
  2. Launching the title with the console set to an unsupported language results in only English text being displayed.

Fail Examples

  1. German localization is shown as French.
  2. Significant localization issues are found when the console is set to German.
  3. Launching the title with the console set to an unsupported language results in no text or null characters being displayed.

003-05 Navigation

Test Steps

  1. Sign in to an Xbox profile.
  2. Launch the title.
  3. Navigate through all menu and option screens (front-end, in-game, and other areas) with the controller, ensuring that all the navigation buttons function as expected. If button labels or command prompts are displayed within any menu UI, verify that each command can be issued correctly.
  4. Repeat Step [3] with natural user interface (NUI), if supported and the sensor is connected and enabled.
  5. Repeat Step [3] with voice user interface (VUI), if supported and the sensor is connected and enabled. If additional VUI commands other than the default commands are available for use, verify that all VUI commands are accurate in all supported languages.

Expected Result
Navigation commands must function as expected.

Pass Examples

  1. All areas of the title can be navigated as expected.

Fail Examples

  1. When entering certain menus, the title displays imagery associated with a button on the controller that has no action when pressed.

003-07 Leaderboards

Test Steps

  1. Sign in to an Xbox profile.
  2. Launch the title.
  3. Console A and B: Sign in to profiles that are friends with each other.
  4. Console A: Navigate to an area of the title where leaderboard posting is possible (only if leaderboards are supported by the title).
    • Testers must check both in-game leaderboards and leaderboards displayed in the system.
  5. Meeting the minimum requirements necessary, perform all steps in order to post to a leaderboard.
  6. Console A: Once the scores have been posted, verify that the statistics are posted to the leaderboards correctly and that the statistics can be sorted using the filters provided.
  7. Console B: Navigate to the leaderboards menu and verify that the statistics for Console A are correctly posted to the leaderboards and that the statistics can be sorted using the filters provided.
  8. Repeat Steps 2 - 7 for all game modes.
  9. Consoles A and B: Where competitive co-op gameplay across Xbox Live is possible, repeat Steps 2 - 7 for each game mode.
  10. Where single-console multiplayer split-screen gameplay across more than one Xbox Live profile is possible, ensure that one of the following occurs:
  • Users are able to post to each leaderboard.
  • At least the lead profile is able to post a score to each leaderboard.
  • Where the title indicates that both users are able to post a score to the leaderboards, scores are correctly registered for each of the participants.

Expected Result
Titles must allow a user to post statistics to all available leaderboards and sort those leaderboards using the filters provided.

Pass Examples

  1. All leaderboards can be posted to successfully.

Fail Examples

  1. The title stops responding while navigating from leaderboard A to leaderboard B.
  2. Users cannot post to specific leaderboards.

003-08 Low or High Friend Count

Test Steps

  1. Sign in to an Xbox profile.
  2. Sign in using a profile that has 100 friends.
  3. Launch the title.
  4. Perform the following actions where applicable:
    a. View all available leaderboards and use all filter options. b. View a friend's gamercard. c. View a friend's rich-presence information. d. Invite a friend to a game session and play until completion.
  5. Repeat Step 4 using an account that has zero friends (if supported, use a "Recent Players" list to verify Steps 4.b - 4.d).
  6. Repeat Step 4 using a profile that has more than 100 friends.
  7. Report any instance of negative behavior.

Expected Result
Title functionality should not be adversely affected by the number of friends a profile contains in its friends list.

Pass Examples

  1. Both minimum and maximum friend counts have no negative impact on the title.

Fail Examples

  1. When using an Xbox Live-enabled profile that has zero friends, if the user enters the leaderboards and views the "Friends Only" filter, a message similar to "Error. Not connected to Xbox Live." is incorrectly displayed.
  2. Any attempt to view a friend's leaderboard results in a blank page. Toggling the filter away from "Friends" then displays an error message similar to: "Failed to read Leaderboard stats".
  3. When using a gamer profile with over 100 friends and creating a private match, the title does not provide access to the user's friends list when entering a multiplayer mode.

003-14 Local Multiple Players

Test Steps

  1. Establish the maximum number of users per console that the title supports.
  2. Consoles A and B: Connect the relevant number of controllers for the maximum number of users per console.
  3. Sign in to Xbox profiles.
  4. Launch the title.
  5. Console A: Create an Xbox Live multiplayer game session with enough slots to allow the maximum number of users on both consoles to join.
  6. Console B: Find the multiplayer game session hosted by console A.
  7. Verify that all users are able to join the multiplayer game session.
  8. Console A: Launch the game session and verify the following:
    • All users can enter gameplay.
    • There is no slowdown.
    • There are no graphical issues.
    • There are no sound issues.
    • There is no negative gameplay experience.
    • All users are able to complete the game session.

Expected Result
Title must allow the maximum number of users per console to play in all Xbox Live-supported game modes without error.

Pass Examples

  1. With two players on a single console, there is no negative impact to the frame rate.

Fail Examples

  1. The Xbox Live game session displays graphical and audio slowdown when any of the participating consoles reaches the maximum number of players.

003-16 Save-Game Compatibility

Test Steps

  1. Sign in to an Xbox profile.
  2. Launch the base title without a connection to Xbox Live.
  3. Enter gameplay and proceed far enough to save game progress in each mode (such as game progress, options and settings, avatar creation, and so on).
  4. Reboot the title and verify that you can load and resume the game-save progress from Step [3].
  5. Connect to Xbox Live and download the content update for the base title.
  6. With the content update applied, verify that you can still load and resume the game-save progress made in Step [3].
  7. Reboot the title, start a new game, and proceed far enough to save game progress again.
  8. Exit and uninstall the title.
  9. Launch the base title without a connection to Xbox Live.
  10. Verify that one of the following occurs:
  • You can load and resume the game-save progress without error.
  • The title displays a message indicating that the game-save progress cannot be used without a content update from Xbox Live.
  • The title does not display the game save made in [6].
  1. Reboot the title and connect to Xbox Live to receive the content update.
  2. With the content update applied, verify that you can still load and resume the game-save progress.

Expected Result
A content-updated version of a title must be able to successfully load a game save created using the non-content-updated version of the title.

Pass Examples

  1. All saves can be loaded successfully by a content-updated version of a title.
  2. When launching the base version to load an updated save, the user is notified of missing content and given a reason why the save file could not be loaded, or the saves made in the content-updated version are not visible in the base version.

Fail Examples

  1. The content-updated version of the title is unable to load a game save that was made during Step [3].
  2. The base version of the title crashes when loading a save made with the content-updated version of the title.

003-17 Headset State Change
Test Steps

  1. Enable Windows Sonic for Headphones as your Headset format under Xbox Audio Settings.
  2. Attach a pair of headphones to the controller or attach a pair of headphones wirelessly to the Xbox.
  3. Boot the title and progress into gameplay.
  4. Verify audio is heard through TV and headphones.
  5. Unplug headphones, verify audio returns after a short amount of time.
  6. Re-plug headphones, verify audio is heard through TV and Headphones.
  7. Repeat steps [5]-[6] throughout the title.

Expected Result
Audio continues to be heard without issue.

Pass Examples

  1. Audio returns without issue whenever the headset is removed.
  2. Audio returns without issue when headset is connected.

Fail Examples

  1. Audio is no longer heard after the headset state changes.
  2. Audio is distorted or corrupt after the headset state changes.

For reference: Comparison of Xbox One Software Development Kit and Microsoft Game Development Kit Audio API


003-18 Headset State Change after Suspend Test Steps

  1. Enable Windows Sonic for Headphones as your Headset format under Xbox Audio Settings.
  2. Attach a pair of headphones to the controller or attach a pair of headphones wirelessly to the Xbox.
  3. Boot the title and progress into gameplay.
  4. Verify audio is heard through TV and headphones.
  5. Suspend the title naturally.
  6. Unplug the headphones.
  7. Resume the title and verify audio is heard through TV.
  8. Re-plug the headphones and verify audio is heard through TV and Headphones.
  9. Repeat steps [5]-[8] throughout the title.

Expected Result
Audio continues to be heard without issue.

Pass Examples

  1. Audio returns without issue whenever the headset is removed.
  2. Audio returns without issue when the headset is connected.

Fail Examples

  1. Audio is no longer heard after the headset state changes.
  2. Audio is distorted or corrupt after the headset state changes.

For reference: Comparison of Xbox One Software Development Kit and Microsoft Game Development Kit Audio API


003-19 Headset State Change after Connected Standby Configuration: One console with the following configuration:

  • Auto sign-in enabled in Settings -> Sign-in, security & passkey -> Instant sign-in.
  • Instant-on set in Settings -> Power & Start-up -> Instant-on.

Test Steps

  1. Enable the Instant-on setting and the Instant sign-in settings on the console (see Configuration).
  2. Sign into an Xbox profile with the console set as a user's home console and with a license to the title.
  3. Enable Windows Sonic for Headphones as your Headset format under Xbox Audio Settings.
  4. Attach a pair of headphones to the controller or attach a pair of headphones wirelessly to the Xbox.
  5. Boot the title and progress into gameplay.
  6. Verify audio is heard through TV and headphones.
  7. Put the console into Connected Standby.
  8. Unplug headphones.
  9. Turn the console back on and resume the game.
  10. Verify audio is heard through TV.
  11. Re-plug headphones, verify audio is heard through TV and Headphones.
  12. Repeat steps [4-11] throughout the title.

Expected Result Audio continues to be heard without issue.

Pass Examples

  1. Audio returns without issue whenever the headset is removed.
  2. Audio returns without issue when the headset is connected.

Fail Examples

  1. Audio is no longer heard after the headset state changes.
  2. Audio is distorted or corrupt after the headset state changes.
  • There is a known issue in the August QFE6 GDK or earlier that will cause titles to miss certain system notifications after a Quick Resume from Connected Standby. For audio, this means that critical audio client notifications will be missed. Guidance is to update to the June QFE9 GDK or later OR the August QFE7 GDK or later.

For reference: Comparison of Xbox One Software Development Kit and Microsoft Game Development Kit Audio API