Azure Pipelines

Azure Pipelines – Running UI Tests on Multiple Agents Continued

In my previous article Azure Pipelines – Running UI Tests on Multiple Agents I focused on running UI tests using the VSTest task on windows agents. In this article I am going to look at splitting up the tests by using a PowerShell script and then running over multiple ubuntu agents.

As with the previous article the code is a  basic ASP.NET Core website project and a UI test project using Selenium and NUnit. Being built using .NET Core the tests can be ran on ubuntu agents instead of windows.

PowerShell Script

As I mentioned the split of the tests for this configuration is going to be done by a PowerShell script, so what does the script need to do:

  1. Get a list of tests
  2. Split the tests based on the number of agents
  3. Provide a way to get the list for a given agent
  4. Set a variable for the Azure Pipelines to use
  1. To get a list of tests I have used the dotnet test option –list-tests
$dotnetExe = Get-Command 'dotnet' -ErrorAction Stop
$Configuration = 'Release'

$testList = & $dotnetExe test --configuration $Configuration --no-build --list-tests | Select-String -Pattern Given

When running dotnet test a Microsoft header is added to the output e.g.

Microsoft (R) Test Execution Command Line Tool Version 16.7.0
Copyright (c) Microsoft Corporation.  All rights reserved.

The following Tests are available:

By using Select-String the header can be omitted. In my case all the test names start with Given so that is pattern I have added so the header is not added to the list of tests.

2. This part uses the modulus of count and number of agents to add the test to the list based on the output. Test one goes in list 0, Test two goes in list 1 and so on. I am also going to use dotnet test to run the code and so a filter is created using the filter property and the test name to be used later.

$testFilters = @{}
$count = 0

0..($agents-1) | ForEach-Object {
  $testFilters[$_] = New-Object System.Collections.ArrayList
}
   
$tests | ForEach-Object {      
  $item=$_.ToString().Trim()
  $filter = "$filterProperty=$item"
  [void]$testFilters[$count % $agents].Add($filter);
  $count++
}

3. The code above results in multiple lists in a hashmap so it can be accessed by index and then each filter can be joined with a pipe separator required by dotnet test filter option.

$filter = $testFilters[$agentNumber-1] -join "|"

4. As per the documentation for Azure Pipelines variables can be defined easily.

echo "##vso[task.setvariable variable=agentTestFilter]$filter"

The whole PowerShell script with parameters ends up like this:

param (    
    [Parameter(Mandatory=$true)][int]$agentNumber,
    [Parameter(Mandatory=$true)][int]$agents,
    [Parameter(Mandatory=$true)][string]$filterProperty = "Name"
)

function splitTests($tests, [int]$agents, $filterProperty) {

    if($null -eq $tests) {

        throw "There are no tests to split"
    }

    $testFilters = @{}
    $count = 0

    0..($agents-1) | ForEach-Object {
        $testFilters[$_] = New-Object System.Collections.ArrayList
    }
   
    $tests | ForEach-Object {      
        $item=$_.ToString().Trim()
        $filter = "$filterProperty=$item"
        [void]$testFilters[$count % $agents].Add($filter);
        $count++
    }
    
    return $testFilters
}

$dotnetExe = Get-Command 'dotnet' -ErrorAction Stop
$Configuration = 'Release'

$testList = & $dotnetExe test --configuration $Configuration --no-build --list-tests | Select-String -Pattern Given
$testFilters = splitTests -tests $testList -agents $agents -filterProperty $filterProperty
$filter = $testFilters[$agentNumber-1] -join "|"
echo "##vso[task.setvariable variable=agentTestFilter]$filter"

Azure Pipeline YAML

In the previous article I used multiple jobs but for this version I am going to use a single job that builds the tests, runs the PowerShell script defined above, I’ve called the script ‘split-tests.ps1’, and then execute the filtered tests on the given agent.

trigger:
  - master

variables:
  buildConfiguration: Release
  uiTestFolder: 'uitests'

jobs:
- job: RunTests
  displayName: Build and Run UI Tests
  pool:
    vmImage: ubuntu-latest
  strategy:
    parallel: 5
  variables:
      siteName: mytest-app
      baseSiteUrl: 'https://$(siteName).azurewebsites.net/'
  steps:
  - task: DotNetCoreCLI@2
    displayName: Restore Packages
    inputs:
      command: 'restore'
      projects: 'mytests/*.csproj'
  - task: DotNetCoreCLI@2
    displayName: Build Tests
    inputs:
      command: 'build'
      projects: '**/mytests.csproj'
      arguments: '--configuration $(buildConfiguration)'
  - task: FileTransform@2
    displayName: Configure Test Run
    inputs:
      folderPath: '$(Build.SourcesDirectory)'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/*settings.json'
  - task: PowerShell@2
    displayName: Check File Substitution
    inputs:
      targetType: 'inline'
      script: 'Get-Content -Path $(Build.SourcesDirectory)/**/testsettings.json'
      pwsh: true
  - task:  PowerShell@2
    displayName: Split UI Tests
    inputs:
     filePath: 'split-tests.ps1'
     workingDirectory: $(Build.SourcesDirectory)
     arguments: '-agentNumber $(System.JobPositionInPhase) -agents 5 -filterProperty "Name"'
     pwsh: true
  - task: DotNetCoreCLI@2
    displayName: Run UI Tests
    inputs:
      command: 'test'
      projects: '**/*tests.csproj'
      arguments: '--configuration $(buildConfiguration) --no-build --filter $(agentTestFilter)'

So the results in the Azure DevOps UI show each of the jobs and seemed to run quite quickly.

Conclusion

The tests running on ubuntu and split by PowerShell seemed to be significantly faster than using the VSTest task on Windows even though there are very few tests.

This has been a fun to workout how to run C# UI tests using Azure Pipelines across multiple agents and trying different techniques. I hope that this is useful for others writing Azure Pipelines for their UI tests.

Azure Pipelines

Azure Pipelines – Running UI Tests on Multiple Agents

When creating a build pipeline one area that seems to get a lot of attention is running UI tests. We want to get feedback as quickly as possible about our code changes.

Test frameworks tend to offer a way of running tests in parallel which is great but what if I want to split tests up over multiple machines and run them in parallel? This isn’t a new problem and there have been many solutions to this, for this article I am going to focus on achieving this in Azure DevOps Pipelines with some .NET code and just using YAML.

UI Tests

I have a project in Visual Studio that contains a basic ASP.NET Core website project and a UI test project using Selenium and NUnit. I deployed the website to Azure ready for the tests to be ran against it.

Azure Pipelines provides a Parallel Strategy, combining that with the VSTest Task will automatically divide up the tests up based on the number of parallel agents.

For this example I am going to use two jobs, one to build the tests and create the output as artifacts and one to download the tests and perform the test run. The second job is configured to run on 5 agents.

The YAML for the pipeline looks like this:

trigger: 
  - master

variables:
  buildConfiguration: Release
  uiTestFolder: 'uitests'

jobs:
- job: BuildTests
  displayName: Build UI Tests
  pool:
    vmImage: windows-latest
  steps:
  - task: DotNetCoreCLI@2
    displayName: Restore Packages
    inputs:
      command: 'restore'
      projects: 'mytests/*.csproj'
  - task: DotNetCoreCLI@2
    displayName: Build Tests
    inputs:
      command: 'build'
      projects: '**/mytests.csproj'
      arguments: '--configuration $(buildConfiguration) -o $(Build.ArtifactStagingDirectory)/$(uiTestFolder)'
  - task: PublishPipelineArtifact@1
    displayName: Upload Library
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)/$(uiTestFolder)'
      artifactName: $(uiTestFolder)  
- job: RunTests
  displayName: Run UI Tests
  dependsOn: BuildTests
  pool:
    vmImage: windows-latest
  strategy:
    parallel: 5
  variables:
      siteName: mytest-app
      baseSiteUrl: 'https://$(siteName).azurewebsites.net/'
  steps:
  - checkout: none
  - task: DownloadPipelineArtifact@2
    displayName: Download Tests
    inputs:
      buildType: 'current'
      artifactName: '$(uiTestFolder)'
      targetPath: '$(Pipeline.Workspace)/$(uiTestFolder)'
  - task: FileTransform@2
    displayName: Configure Test Run
    inputs:
      folderPath: '$(Pipeline.Workspace)'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/*settings.json'
  - task: PowerShell@2
    displayName: Check File Substitution
    inputs:
      targetType: 'inline'
      script: 'Get-Content -Path $(Pipeline.Workspace)/$(uiTestFolder)/testsettings.json'
      pwsh: true
  - task: VSTest@2
    displayName: Run UI Tests
    inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2: |
        **\*tests.dll
        !**\*TestAdapter.dll
        !**\obj\**
      searchFolder: '$(Pipeline.Workspace)/$(uiTestFolder)'
      uiTests: true
      runInParallel: false
      testRunTitle: 'Basic UI Tests'

NOTE: At the time of writing the VSTest Task can only run on the windows agents with either Visual Studio Installed or by using the Visual Studio Test Platform Installer Task.

You might notice that in the VSTest task ‘runInParallel’ is set to false, this is because UI tests can cause issues running in parallel on the same box.

After the test has been ran you can see the results in the Azure DevOps UI. There is the Build UI Tests job and then each Run UI Tests is given a number for each parallel run on a different agent.

NOTE: the number of parallel jobs depend on the amount of agents you can run in your parallel in your Azure DevOps configuration and the number of available agents when the job runs.

Conclusion

Using the VSTest task and parallel strategy is a very simple way to get UI tests to run on multiple agents with a .NET project.

I found my example didn’t really offer much gain in terms of time but I only had 14 UI tests and just using one agent performed better than multiple agents. I expect the gains would be seen with a large test suite, I would suggest trying the test runs with different numbers of agents to see what works best for your tests. You might also gain benefits running the UI tests on your own agents.

Additional Information