When creating a build pipeline one area that seems to get a lot of attention is running UI tests. We want to get feedback as quickly as possible about our code changes.
Test frameworks tend to offer a way of running tests in parallel which is great but what if I want to split tests up over multiple machines and run them in parallel? This isn’t a new problem and there have been many solutions to this, for this article I am going to focus on achieving this in Azure DevOps Pipelines with some .NET code and just using YAML.
I have a project in Visual Studio that contains a basic ASP.NET Core website project and a UI test project using Selenium and NUnit. I deployed the website to Azure ready for the tests to be ran against it.
For this example I am going to use two jobs, one to build the tests and create the output as artifacts and one to download the tests and perform the test run. The second job is configured to run on 5 agents.
The YAML for the pipeline looks like this:
trigger: - master variables: buildConfiguration: Release uiTestFolder: 'uitests' jobs: - job: BuildTests displayName: Build UI Tests pool: vmImage: windows-latest steps: - task: DotNetCoreCLI@2 displayName: Restore Packages inputs: command: 'restore' projects: 'mytests/*.csproj' - task: DotNetCoreCLI@2 displayName: Build Tests inputs: command: 'build' projects: '**/mytests.csproj' arguments: '--configuration $(buildConfiguration) -o $(Build.ArtifactStagingDirectory)/$(uiTestFolder)' - task: PublishPipelineArtifact@1 displayName: Upload Library inputs: targetPath: '$(Build.ArtifactStagingDirectory)/$(uiTestFolder)' artifactName: $(uiTestFolder) - job: RunTests displayName: Run UI Tests dependsOn: BuildTests pool: vmImage: windows-latest strategy: parallel: 5 variables: siteName: mytest-app baseSiteUrl: 'https://$(siteName).azurewebsites.net/' steps: - checkout: none - task: DownloadPipelineArtifact@2 displayName: Download Tests inputs: buildType: 'current' artifactName: '$(uiTestFolder)' targetPath: '$(Pipeline.Workspace)/$(uiTestFolder)' - task: FileTransform@2 displayName: Configure Test Run inputs: folderPath: '$(Pipeline.Workspace)' xmlTransformationRules: '' jsonTargetFiles: '**/*settings.json' - task: PowerShell@2 displayName: Check File Substitution inputs: targetType: 'inline' script: 'Get-Content -Path $(Pipeline.Workspace)/$(uiTestFolder)/testsettings.json' pwsh: true - task: VSTest@2 displayName: Run UI Tests inputs: testSelector: 'testAssemblies' testAssemblyVer2: | **\*tests.dll !**\*TestAdapter.dll !**\obj\** searchFolder: '$(Pipeline.Workspace)/$(uiTestFolder)' uiTests: true runInParallel: false testRunTitle: 'Basic UI Tests'
NOTE: At the time of writing the VSTest Task can only run on the windows agents with either Visual Studio Installed or by using the Visual Studio Test Platform Installer Task.
You might notice that in the VSTest task ‘runInParallel’ is set to false, this is because UI tests can cause issues running in parallel on the same box.
After the test has been ran you can see the results in the Azure DevOps UI. There is the Build UI Tests job and then each Run UI Tests is given a number for each parallel run on a different agent.
NOTE: the number of parallel jobs depend on the amount of agents you can run in your parallel in your Azure DevOps configuration and the number of available agents when the job runs.
Using the VSTest task and parallel strategy is a very simple way to get UI tests to run on multiple agents with a .NET project.
I found my example didn’t really offer much gain in terms of time but I only had 14 UI tests and just using one agent performed better than multiple agents. I expect the gains would be seen with a large test suite, I would suggest trying the test runs with different numbers of agents to see what works best for your tests. You might also gain benefits running the UI tests on your own agents.