When creating a build pipeline one area that seems to get a lot of attention is running UI tests. We want to get feedback as quickly as possible about our code changes.

Test frameworks tend to offer a way of running tests in parallel which is great but what if I want to split tests up over multiple machines and run them in parallel? This isn’t a new problem and there have been many solutions to this, for this article I am going to focus on achieving this in Azure DevOps Pipelines with some .NET code and just using YAML.

UI Tests

I have a project in Visual Studio that contains a basic ASP.NET Core website project and a UI test project using Selenium and NUnit. I deployed the website to Azure ready for the tests to be ran against it.

Azure Pipelines provides a Parallel Strategy, combining that with the VSTest Task will automatically divide up the tests up based on the number of parallel agents.

For this example I am going to use two jobs, one to build the tests and create the output as artifacts and one to download the tests and perform the test run. The second job is configured to run on 5 agents.

The YAML for the pipeline looks like this:

trigger: 
  - master

variables:
  buildConfiguration: Release
  uiTestFolder: 'uitests'

jobs:
- job: BuildTests
  displayName: Build UI Tests
  pool:
    vmImage: windows-latest
  steps:
  - task: DotNetCoreCLI@2
    displayName: Restore Packages
    inputs:
      command: 'restore'
      projects: 'mytests/*.csproj'
  - task: DotNetCoreCLI@2
    displayName: Build Tests
    inputs:
      command: 'build'
      projects: '**/mytests.csproj'
      arguments: '--configuration $(buildConfiguration) -o $(Build.ArtifactStagingDirectory)/$(uiTestFolder)'
  - task: PublishPipelineArtifact@1
    displayName: Upload Library
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)/$(uiTestFolder)'
      artifactName: $(uiTestFolder)  
- job: RunTests
  displayName: Run UI Tests
  dependsOn: BuildTests
  pool:
    vmImage: windows-latest
  strategy:
    parallel: 5
  variables:
      siteName: mytest-app
      baseSiteUrl: 'https://$(siteName).azurewebsites.net/'
  steps:
  - checkout: none
  - task: DownloadPipelineArtifact@2
    displayName: Download Tests
    inputs:
      buildType: 'current'
      artifactName: '$(uiTestFolder)'
      targetPath: '$(Pipeline.Workspace)/$(uiTestFolder)'
  - task: FileTransform@2
    displayName: Configure Test Run
    inputs:
      folderPath: '$(Pipeline.Workspace)'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/*settings.json'
  - task: PowerShell@2
    displayName: Check File Substitution
    inputs:
      targetType: 'inline'
      script: 'Get-Content -Path $(Pipeline.Workspace)/$(uiTestFolder)/testsettings.json'
      pwsh: true
  - task: VSTest@2
    displayName: Run UI Tests
    inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2: |
        **\*tests.dll
        !**\*TestAdapter.dll
        !**\obj\**
      searchFolder: '$(Pipeline.Workspace)/$(uiTestFolder)'
      uiTests: true
      runInParallel: false
      testRunTitle: 'Basic UI Tests'

NOTE: At the time of writing the VSTest Task can only run on the windows agents with either Visual Studio Installed or by using the Visual Studio Test Platform Installer Task.

You might notice that in the VSTest task ‘runInParallel’ is set to false, this is because UI tests can cause issues running in parallel on the same box.

After the test has been ran you can see the results in the Azure DevOps UI. There is the Build UI Tests job and then each Run UI Tests is given a number for each parallel run on a different agent.

NOTE: the number of parallel jobs depend on the amount of agents you can run in your parallel in your Azure DevOps configuration and the number of available agents when the job runs.

Conclusion

Using the VSTest task and parallel strategy is a very simple way to get UI tests to run on multiple agents with a .NET project.

I found my example didn’t really offer much gain in terms of time but I only had 14 UI tests and just using one agent performed better than multiple agents. I expect the gains would be seen with a large test suite, I would suggest trying the test runs with different numbers of agents to see what works best for your tests. You might also gain benefits running the UI tests on your own agents.

Additional Information

• Run Tests in Parallel
• VSTest Task
• Visual Studio Test Platform Installer Task