Troubleshooting silent testing

  • Article

The Microsoft eCDN Silent Testing framework allows running simulations on multiple devices easily to emulate and examine how a given network behaves under the load of a live video event. All findings are logically presented in Microsoft eCDN's analytics dashboards.

Potential problems:

  • Unable to create a silent test
  • No runners (or peers) participate in a silent test
  • Unable to validate the instantiation of a silent runner
  • Lack of analytics
  • Low peering efficiency
  • Large discrepancy between concurrent viewers and assigned devices

Solution for inability to create a silent test

If missing the plus (+) button for creating a silent test, ensure your Microsoft Entra user has an appropriate role assigned which grants Silent Test Modify permissions. For more, see the applicable roles in the Manage access documentation.

Small screenshot of Silent Test page header with create button highlighted in red outline.

Solution for lack of runner participation in silent tests

Review the silent testing process overview to ensure compliance. Continue with this troubleshooting guide for more.

Solution for inability to validate the instantiation of a silent runner

The silent runner is designed to be inconspicuous to the user so validating the instantiation of one isn't a straight forward endeavor. Here's what to look for when seeking to validate whether a silent runner is active or not.

  • The Connected Clients count in your Silent Testing dashboard.
  • The presence of log files as "$env:TEMP\p5_log_" + $TestID + ".txt" where $env:TEMP is a system-set path and the $TestID value is set in the script. For example "C:\Users\MYUSERNAME\AppData\Local\Temp\p5_log_123.txt"
  • The presence each of these background processes.
    • powershell or pwsh - A hidden PowerShell instance that runs the silent runner script; the choice of which is typically controlled by your endpoint management system.
    • msedge or chrome - Hidden Chromium browser instance(s), which take on the role of your simulated viewer.
    • cmd - A hidden command prompt watchdog process that ends the PowerShell and chromium processes after the timeout elapses.

Note

When using some endpoint management tools such as Microsoft Intune to deploy the silent test script, note that some tools determine the script to have failed due to the fact that it did not return an exit code within a pre-determined amount of time. This is expected due to the long-running nature of the script. We suggest using different success criteria, such as some of the above mentioned points, to determine the script's success.

Solution for lack of analytics

If missing analytics, it's presumed that there's a blockage between individual clients and Microsoft eCDN's backend. Begin troubleshooting by opening the Management Console then navigating to Advanced > Silent Tester. At the top-right of that page, you'll find a question mark (?) link that takes you to the helper page where additional information on steps 1 and 3 can be found. Alternatively, you can also refer to How To Perform Silent Testing.

  1. Open a direct runner URL taking care to use your tenant ID instead of the TENANT_ID_HERE placeholder. Construct it from the template URL provided here or you can use the prebuilt URL found in your helper page. The direct runner page simulates a single viewer, with which we'll investigate the state of the required connections.

    https://st-sdk.ecdn.teams.microsoft.com/?customerId=TENANT_ID_HERE&adapterId=Direct

    Image of Test Runner webpage; includes Customer ID, Client ID, and Adapter ID, the values of which are blacked-out.

  2. Reveal the browser's DevTools. On Microsoft Edge, right-click on the page, then select "Inspect."

    Important

    The DevTools must be opened before starting the silent test in the following step, or crucial session start-up information will not be captured for your analysis.

  3. Back on the Silent Tester page, start a Scheduled (Silent) Test, leaving all settings on their defaults aside from Video Quality. Set video quality to a static value, such as 480p - 1Mbps to remove variability. Once started, a gray DIV bar appears on the direct runner page with the text "Ongoing Test" to confirm the stream has begun, as seen in this example figure.

    Image of the same Test Runner page. The page now displays more information underneath in a gray, horizontally long box, which contains session details such as 'time left' and 'stream url.'

    You can also confirm the session is ongoing by observing video chunks downloading in the DevTool's Network tab.

    DevTools window showing list of downloading video chunks.

  4. Inspect the DevTool's Network and Console tabs for errors. Pay particular attention to the beginning of the session.

    Note any errors in red, excluding the "font" errors, which are known and unimpactful. Any errors that may indicate something blocking Microsoft eCDN is a fair lead in the troubleshooting effort.

    For example, in the Console tab, you may witness a connection rejection error similar to the following figure.

    Small screen capture of console error text.

    This may be indicating that our domain ecdn.teams.microsoft.com hasn't been added to your allowlist under Websites Allow List in your Third Party Platforms page.

    Image of 'Websites Allow List' UI, containing a textbox and purple 'Add' button.

    Or alternatively, your IP may not be included in the End-user IP Allow List of your Security page.

    Image of 'End-user IP Allow List' UI, containing a textbox and purple 'Add' button.

    Note

    Neither filter requires entries for Microsoft eCDN to function. That is, leaving either of these filters unpopulated disables the filter.

    If you’re not getting HTTP video data chunks, it could mean that something is blocking you from connecting to our backend; be it a firewall, proxy, etc. If you can identify the URL or protocol that is being blocked, check with your networking and/or security team to see if connections of that sort are being allowed. Review the following documents to ensure proper exceptions have been created: Network Requirements, Cloud and Security doc.

Tip

You can also use our Tester page to identify potential network issues such as your firewall blocking websocket connections. If any of the items under the Networking section are marked with a red X, download the report and send it to your Microsoft Customer Success Account Manager for review and support.

Solution for low peering efficiency

Low peering efficiency typically occurs when you're conducting a silent test with too few clients (under 20). By increasing the number of clients, you're enlarging the peering groups thus increasing the peering efficiency. See low efficiency troubleshooting for a list of other potential causes and how to troubleshoot each.

Solution for a large discrepancy between concurrent viewers and assigned devices

This situation typically happens when:

  • local or corporate security software is blocking Microsoft eCDN
  • the silent test is being conducted during transition periods (employees leaving the office and shutting down their workstations)

Here's how to troubleshoot:

  • Avoid starting the silent test in transition periods (where employees are leaving the office) since then that might skew the number of participants.

  • Allow between 1 - 3 minutes for real-time analytics to reflect in the Microsoft eCDN Analytics dashboard.

  • Ensure the runner URL isn't being blocked. To see more, refer to the steps in the Solution for lack of analytics section.