What does a performance developer do at Microsoft?
Someone was just asking me this question the other day so I figured that I would answer it here for people that are interested in this role inside and outside the company.
I have been a performance developer at Microsoft for a little more than 4 years. During these years I have done performance investigations for Visual Studio (Workflow Designer), .Net Framework (Windows Communication Foundation), Azure Service Bus, Azure Event Hub and Azure Api Management. And still some friends at Microsoft are not sure what I do in my day to day work.
Typically Microsoft used to have 3 clearly differentiated roles (This has been changing lately): Program Managers, Software Developer Engineers and Software Developer Engineer in Test. Some SDETs specialize in UI testing or Stress testing. The PM role has a couple specializations: Release PM or Feature PM. The Developer has a few specializations as well: one of these specializations is the Performance Developer.
We are developers that focus on finding, analyzing and fixing performance problems (throughput, latency or scalability). We also work on adding performance features and since performance and scalability are so closely related we are also pulled-in to work on issues that might affect how a system behaves at scale. So part of our day can be spent looking at a profile or we can be looking at crash dumps to figure out why a given product is crashing when it’s being hit by thousands of requests.
I personally work in a group called "Azure Application Fundamentals". This team owns the Stress team, the Performance team and the Servicing team for the Azure Application group. What this means is that any team that is part of the Azure Application can ask us to help them with performance investigations.
This is a list of some things we do as part of our daily work:
Measure scenarios: Work with the feature team to identify main scenarios that we want to measure, establish goals, test authoring, ensure the tests are getting executed regularly and that the results are reported somewhere.
Regression Prevention: Make sure that the performance of the main scenarios improves as time goes by.
Performance Analysis: Jump in and analyze the performance of any feature important to the team and figure out why it’s not meeting goals.
Optimization: Fix the performance problems or implement performance features to improve the performance of the product.
Education: Teach the feature team what things are not good for performance and pretty much make sure they can do any of the previous steps themselves. Also share what you learn about the different tools used to measure and analyze performance problems.
There is a lot of case by case investigation but something important is that we try to make sure that the teams that are under our purview are measuring and reporting all their performance scenarios to a centralized Reporting system. This makes it easier to understand how the team is related to performance. It makes it very clear to understand when things are getting better or worse.
I find a couple of things interesting about this role. It lets even junior developers learn about performance, investigate profiles and memory dumps and basically grow until they become more knowledgeable about performance. In my experience not a lot of developers know what to do when something is slow. Having a specialized role encourages sharing of this information.
Also it's not very common for a team to be interested in investing in performance during the early parts of the development cycle. They want to have something to show as soon as possible. However what I've seen is that if you leave the performance as the last thing you do you will not be able to fix much. You will find that your design has certain performance characteristics that are simply too risky to change at the end of the lifecycle of the product. You might also find that everything is slow: there is no clear "smoking bullet" to change. This can be because you have used pervasive slow techniques across your entire product. This causes in my opinion a lot of teams to just ship something even though they know it is slow. Having someone from outside focusing on performance helps to avoid this.
So for me it’s been a very cool experience and I highly recommend it!
If you are interested in performance here are a few blogs of other Performance Engineers at Microsoft:
Sajay Antony- His blog has not been updated in a while but he has some neat projects in his Github
Bruce Dawson - Ex Msft – has fantastic posts about xperf