Update on Microsoft’s Symbol Server

  • Article

Update on Microsoft’s Symbol Server

"Can you please make available the symbols for the latest public non-beta Creators Update ntkrnlmp on the MS Symbol server? Having a real hard time debugging issues w/out them."

"We specifically require the symbols for ntoskrnl.exe (ntkrnlmp.pdb) in order to debug a critical deadlock in the file system drivers we are using. Would it be possible to get these new symbols ASAP?"

We've received a lot of emails like this in the past 6 months or so. Developers trying to do the right thing and develop on the latest Windows previews were having problems because our symbol pipeline couldn't keep up with how quickly Windows was releasing.

When the Windows Insider program started releasing builds on a cadence quicker than once a month, we realized we really need to re-work how our symbol processing works in order to make sure that developers on the fast ring would be able to properly debug their drivers and applications. We started down this long road many months back, and as of today, we're moving the back-end behind https://msdl.microsoft.com/download/symbols/ to an Azure-based symbol store that is backed by a more robust publishing pipeline than the one we've had in the past.

Here's a bit about the journey it took to get here and some of the inner workings of our engineering system. Note that most of this is Windows focused, but relevant to all Microsoft published symbols (Windows is just the vast majority of what we get feedback around).

Where are we coming from?

The symbol server that's in use today has been around for 10+ years, back when it was brought online, Microsoft shipped all its products every few years in cardboard or plastic boxes, and the uploading of symbols was something that someone had to do a few times a year at most. As time has gone on, the actual hardware powering the symbol server has of course changed and grown with the scale of Microsoft and Windows, and we've automated a lot of the actual publishing of symbols, but the services that process the files and make them available publicly have largely stayed the same.

As the release model for big releases and patches for Windows evolved from XP all the way to 8.1, there weren't any major issues with symbols. A patch would come out on a Tuesday, symbols would be available that Thursday or so, and everything would be fine until the next month. Big releases were typically years apart with longer runways to get public symbols in the hands of customers. Back when this was all being setup, the processes that were put into place to publish symbols took somewhere between one and three days and no one really noticed. The actual time it took for symbols to be available depended a lot on when during the day the request was made, how many other requests had been made recently, and normal maintenance tasks for all the servers and services involved.

With the monthly patch cadence, symbols tended to go through that delay before most people even had the update installed, let alone started debugging crashes on it. Then Windows 10 and the Windows Insider Program came around…

What changed?

If you go look through the Windows Insider Blog for late 2016 or early 2017, you'll notice a pretty clear trend. The Insider Program really likes releasing Windows builds whenever possible. With fast ring releases going out every few days, there were builds that were out of date before symbols were even available. Developers, rightfully so, started clamoring about how difficult it was to develop and debug on Insider builds due to missing symbols.

A short delay on symbols being available relative to a monthly release was minor, a short delay when builds are coming out weekly is a workflow blocking issue.

So is it fixed?

Not entirely but we’ve hit a point where we wanted to start talking about the progress more publicly. We've removed the largest impacting factor on the delay. We took measurements with some recent releases and found that the new back-end has symbols available right along-side the build they're related to. There are still some areas in our pipeline that can cause delays which we are already working on, and we'll be closely monitoring what we have completed and any feedback we get from developers as the system evolves.

When is the change happening?

We're incrementally moving traffic over from the old back-end to the new one starting today. We don't anticipate any issues with the transition, we're doing it in a way that you really wouldn't notice unless you read this blog post, but if you hit any problems that you think could be related, feel free to comment here or email WinDbgFB@Microsoft.com

-Andy

@aluhrs13

Comments

  • Anonymous
    October 19, 2017
    for me the ntkrnlmp.pdb for 16299.19 (x64) is missing. Saw this today during analyzing a ETL file with WPA.exe
  • Anonymous
    November 01, 2017
    Hi Andy,There seems to be an issue with symbols right now, seehttps://social.technet.microsoft.com/Forums/en-US/a47feb44-cb50-4240-8983-c44f6b1d5c0a/windbg-1001629915-thread-process-command-errors?forum=win10itprohardwareI'm seeing the same thing with dumps coming in now - I have one here 14393.1770.amd64fre.rs1_release.170917-1700and !thread fffff800ddfc9940: Unable to get thread contents and !process **** NT ACTIVE PROCESS DUMP **** Unable to read _LIST_ENTRY @ fffff800ddf0c3f0 just don't work. Had the same thing with another dump from the same build yesterday and sent an email but no reply yet - people busy I guess.Dumps from other builds are fine using the exact same windbg executable - can't see it's anything other than a symbols issue.So if that could be fixed it would be great...
  • Anonymous
    February 17, 2018
    4 months on and symbols still not available for 1709. Why would you apply the move to Azure when you haven't tested and moved everything over first. What is this, 2 sprints per year?
    • Anonymous
      February 18, 2018
      Do you have the exact version number and binary that's missing symbols? The latest Windows Updates from this Tuesday are still processing. One of the legacy pieces I mention in the post that we're planning on replacing in the near future is currently backlogged.
  • Anonymous
    February 20, 2018
    I seriously think of abandoning Insider Preview, because it's impossible to debug the driver code without symbols. Today is the end of February, build 17101, no symbols. Why not to make a separate server for Insider Preview?
    • Anonymous
      February 20, 2018
      Sorry this is impacting you so much. The current delays we're hitting are due to the last piece of our legacy pipeline that we're on track to replace in the next month. Standing up a separate endpoint to handle the scale that would be needed would unfortunately take more time than that, and lead to confusion around which endpoint is the "right" one depending on what you're debugging.
  • Anonymous
    April 17, 2018
    Morning, Could comone please post/provide a solid, current link for the process and resources necessary to install/update the symbol libraries on Windows Server 2016/Windows 10 workstations that have NO INTERNET ACCESS. Our environment does not permit Windows Server 2016 to have internet access at all, and a percentage of our Windows 10 workstations are the same way. We use a WSUS server along with System Center Config Mgr to patch our environment so nothing needs to go to Microsoft. The Get-WindowsUpdateLogs process is useless for us as all the logs contain only: "Unknown GUID-----No Format Information Found"..Thanks you in advance for your time and assistance!!TonyNCDOT, Raleigh, NC.