When creating a build pipeline one area that seems to get a lot of attention is running UI tests. We want to get feedback as quickly as possible about our code changes.
Test frameworks tend to offer a way of running tests in parallel which is great but what if I want to split tests up over multiple machines and run them in parallel? This isn’t a new problem and there have been many solutions to this, for this article I am going to focus on achieving this in Azure DevOps Pipelines with some .NET code and just using YAML.
I have a project in Visual Studio that contains a basic ASP.NET Core website project and a UI test project using Selenium and NUnit. I deployed the website to Azure ready for the tests to be ran against it.
Azure Pipelines provides a Parallel Strategy, combining that with the VSTest Task will automatically divide up the tests up based on the number of parallel agents.
For this example I am going to use two jobs, one to build the tests and create the output as artifacts and one to download the tests and perform the test run. The second job is configured to run on 5 agents.
You might notice that in the VSTest task ‘runInParallel’ is set to false, this is because UI tests can cause issues running in parallel on the same box.
After the test has been ran you can see the results in the Azure DevOps UI. There is the Build UI Tests job and then each Run UI Tests is given a number for each parallel run on a different agent.
NOTE: the number of parallel jobs depend on the amount of agents you can run in your parallel in your Azure DevOps configuration and the number of available agents when the job runs.
Using the VSTest task and parallel strategy is a very simple way to get UI tests to run on multiple agents with a .NET project.
I found my example didn’t really offer much gain in terms of time but I only had 14 UI tests and just using one agent performed better than multiple agents. I expect the gains would be seen with a large test suite, I would suggest trying the test runs with different numbers of agents to see what works best for your tests. You might also gain benefits running the UI tests on your own agents.
Well that doesn’t work, the parameter values are not picked up by the FileTransform Task and so no substitution happens. So, what if we define the variables again and assign the values from the parameters.
Note: The Azure Pipelines editor shows the loop underlined and reports an error “The first property must be task”.
But when running validate.
The result is OK. Hopefully this issue will get resolved in the future and not show as an error in the UI.
So the loop worked and provided all of the variables to be substituted. We could leave it there as it works, but we could also use some of the other parameter properties such as displayName and values to provide a nicer configuration.