Automate Your Tests for Agile
The prevailing wisdom a few years back was that performance testing doesn’t fit into an Agile environment. I disagree with that idea wholeheartedly, and recognize that performance testing done correctly can work even with Agile teams. I have seen members of my team add performance tests into the CI Pipeline and perform continuous testing.
In order to achieve the velocity needed to be successful, automate as many of the steps of your test as possible. As a performance test engineer, you are not just automating the HTTP protocol requests anymore, you need to be able to automate the test execution, set-up, and results as well. I like to use PowerShell for the execution and set-up tasks and Excel for the reporting, when working with Visual Studio Load Tests.
Test Set-up
Automating your test set-up may include activities such as: restoring databases, recycling IIS, restarting services or machines, checking disk space on servers, setting up test agents and controllers, and maybe even running a smoke test. This should be a click and go script, meaning you can run it anytime and free yourself up to work on more important work, like new automation, test planning, results analysis. For example, if you need to reset IIS on servers prior to your test you could have a PS (PowerShell) script that looks like this.
## restart iis on servers
function resetIIS($server){
## $server could be a single machine or an array of machines
## usage:
## $server = @("web01","app01","app02")
## resetIIS $server
## powershell command to reset iis
Invoke-Command -scriptblock {iisreset} -ComputerName $server -AsJob
## using Get-Job allows multiple servers to restart iis in parallel
Get-Job | Wait-Job | Format-Table
}
Something important to note with the above script, it is a function. Create all your activities as functions, that way you can reuse them quickly as needed. Highly cohesive and loosely coupled functions are the goal when deciding what to add to each function.
My recommendation is to fist create a checklist. Jot down all the things you manually do before a load test, and then start automating them one at a time. You may need help, ask your DBA’s to write you a SQL script to restore the Database, talk to your operations team about getting access to the servers to perform IIS Reset.
$databases = @(“DB01”,”DB02”)
$IISservers = @("web01","app01","app02")
restoreDB($Databases)
resetIIS($IISservers)
executeSmokeTest()
Test Execution
If this is not your first time running a load test, don’t sit and watch it. Automate the execution and look at the results when it is finished. You may choose to automate the ability to know if the smoke test passed or you might want to add a manual verification step before the execution. Visual Studio exposes an executable that can be used to launch tests from the command line.
It can be found at C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\mstest.exe
switch ($VSversion) {
2013 {$mstest = "C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\mstest.exe"}
2015 {$mstest = "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\mstest.exe"}
2017 {$mstest = "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\mstest.exe"}
}
$mstestArguments = @($testFile, $testSetting)
& $mstest $mstestArguments >> $testoutLog
Now that you have everything scripted; how you choose to launch it is up to you. You could take it a step further and run from the windows task scheduler, or add it to a CI/CD pipeline. Even just using the script to launch from PowerShell will help you speed up delivery.
Results
I will be adding more articles about results analysis with specifics around the queries that I use and how to pivot the data in Excel. Let me know in the comments if this is something you are interested in.
Power Query in Excel is amazing, use it. The goal is to have preformatted tables and charts that you can use to quickly determine if the test met the acceptance criteria and how your changes to code or SQL affected the Key Performance Indicators (KPIs).
The next time a test runs you should just have to hit refresh on the data tab and your results will be there. Over time you will see trends and characteristics in the KPIs that will point you to where you need deeper analysis. After deep diving the data you will notice things that are missing from your automated results and then you add them in. Eventually, you will have a great automated results solution.
Comments
- Anonymous
October 09, 2017
Very good, good that I learned to make reusable tests!- Anonymous
October 11, 2017
Thanks Luiz. You are correct in that a major missing piece of the post is speeding up scripting by using reusable tests. I'll try to post something about creating reusable tests and other ways to speed up scripting in the future.
- Anonymous