Issues with Visual Studio Team Services - 08/04 - Mitigated

  • [アーティクル]

Final Update: Friday, August 4th 2017 16:01 UTC

We’ve confirmed that all systems are back to normal as of 12:34 UTC. Our logs show the incident started on 4th August 2017 10:30 UTC and that during the 2 hours and 4 minutes that it took to resolve the issue 100% of customers experienced 500 Errors while connecting to VSCOM. Sorry for any inconvenience this may have caused.

  • Root Cause: VSTS traffic was getting routed to recently upgraded DNS servers which were unhealthy. The Azure DNS team has taken the faulty servers out of rotation and have also made sure that any automated upgrades will not add these servers back into rotation accidentally.
  • Chance of Re-occurrence: Low
  • Lessons Learned: We are working with Azure DNS to get full RCA details and permanent fix..
  • Incident Timeline: 2 hours & 4 minutes – 4th August 10:30 UTC through 4th August 12:34 UTC

Sincerely,
Thomas

Initial Update: Friday, August 4th 2017 12:26 UTC

A potentially customer impacting alert is being investigated. Triage is in progress and we will provide an update with more information.

  • Next Update: Before Friday, August 4th 2017 13:30 UTC

Sincerely,
Thomas