Azure Pipelines, DevOps, Testing

Azure Pipelines – Running UI Tests on Multiple Agents Continued

Posted on by Mark Johnson in Azure Pipelines, DevOps, Testing

In my previous article Azure Pipelines – Running UI Tests on Multiple Agents I focused on running UI tests using the VSTest task on windows agents. In this article I am going to look at splitting up the tests by using a PowerShell script and then running over multiple ubuntu agents.

As with the previous article the code is a  basic ASP.NET Core website project and a UI test project using Selenium and NUnit. Being built using .NET Core the tests can be ran on ubuntu agents instead of windows.

PowerShell Script

As I mentioned the split of the tests for this configuration is going to be done by a PowerShell script, so what does the script need to do:

  1. Get a list of tests
  2. Split the tests based on the number of agents
  3. Provide a way to get the list for a given agent
  4. Set a variable for the Azure Pipelines to use
  1. To get a list of tests I have used the dotnet test option –list-tests
$dotnetExe = Get-Command 'dotnet' -ErrorAction Stop
$Configuration = 'Release'

$testList = & $dotnetExe test --configuration $Configuration --no-build --list-tests | Select-String -Pattern Given

When running dotnet test a Microsoft header is added to the output e.g.

Microsoft (R) Test Execution Command Line Tool Version 16.7.0
Copyright (c) Microsoft Corporation.  All rights reserved.

The following Tests are available:

By using Select-String the header can be omitted. In my case all the test names start with Given so that is pattern I have added so the header is not added to the list of tests.

2. This part uses the modulus of count and number of agents to add the test to the list based on the output. Test one goes in list 0, Test two goes in list 1 and so on. I am also going to use dotnet test to run the code and so a filter is created using the filter property and the test name to be used later.

$testFilters = @{}
$count = 0

0..($agents-1) | ForEach-Object {
  $testFilters[$_] = New-Object System.Collections.ArrayList
}
   
$tests | ForEach-Object {      
  $item=$_.ToString().Trim()
  $filter = "$filterProperty=$item"
  [void]$testFilters[$count % $agents].Add($filter);
  $count++
}

3. The code above results in multiple lists in a hashmap so it can be accessed by index and then each filter can be joined with a pipe separator required by dotnet test filter option.

$filter = $testFilters[$agentNumber-1] -join "|"

4. As per the documentation for Azure Pipelines variables can be defined easily.

echo "##vso[task.setvariable variable=agentTestFilter]$filter"

The whole PowerShell script with parameters ends up like this:

param (    
    [Parameter(Mandatory=$true)][int]$agentNumber,
    [Parameter(Mandatory=$true)][int]$agents,
    [Parameter(Mandatory=$true)][string]$filterProperty = "Name"
)

function splitTests($tests, [int]$agents, $filterProperty) {

    if($null -eq $tests) {

        throw "There are no tests to split"
    }

    $testFilters = @{}
    $count = 0

    0..($agents-1) | ForEach-Object {
        $testFilters[$_] = New-Object System.Collections.ArrayList
    }
   
    $tests | ForEach-Object {      
        $item=$_.ToString().Trim()
        $filter = "$filterProperty=$item"
        [void]$testFilters[$count % $agents].Add($filter);
        $count++
    }
    
    return $testFilters
}

$dotnetExe = Get-Command 'dotnet' -ErrorAction Stop
$Configuration = 'Release'

$testList = & $dotnetExe test --configuration $Configuration --no-build --list-tests | Select-String -Pattern Given
$testFilters = splitTests -tests $testList -agents $agents -filterProperty $filterProperty
$filter = $testFilters[$agentNumber-1] -join "|"
echo "##vso[task.setvariable variable=agentTestFilter]$filter"

Azure Pipeline YAML

In the previous article I used multiple jobs but for this version I am going to use a single job that builds the tests, runs the PowerShell script defined above, I’ve called the script ‘split-tests.ps1’, and then execute the filtered tests on the given agent.

trigger:
  - master

variables:
  buildConfiguration: Release
  uiTestFolder: 'uitests'

jobs:
- job: RunTests
  displayName: Build and Run UI Tests
  pool:
    vmImage: ubuntu-latest
  strategy:
    parallel: 5
  variables:
      siteName: mytest-app
      baseSiteUrl: 'https://$(siteName).azurewebsites.net/'
  steps:
  - task: DotNetCoreCLI@2
    displayName: Restore Packages
    inputs:
      command: 'restore'
      projects: 'mytests/*.csproj'
  - task: DotNetCoreCLI@2
    displayName: Build Tests
    inputs:
      command: 'build'
      projects: '**/mytests.csproj'
      arguments: '--configuration $(buildConfiguration)'
  - task: FileTransform@2
    displayName: Configure Test Run
    inputs:
      folderPath: '$(Build.SourcesDirectory)'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/*settings.json'
  - task: PowerShell@2
    displayName: Check File Substitution
    inputs:
      targetType: 'inline'
      script: 'Get-Content -Path $(Build.SourcesDirectory)/**/testsettings.json'
      pwsh: true
  - task:  PowerShell@2
    displayName: Split UI Tests
    inputs:
     filePath: 'split-tests.ps1'
     workingDirectory: $(Build.SourcesDirectory)
     arguments: '-agentNumber $(System.JobPositionInPhase) -agents 5 -filterProperty "Name"'
     pwsh: true
  - task: DotNetCoreCLI@2
    displayName: Run UI Tests
    inputs:
      command: 'test'
      projects: '**/*tests.csproj'
      arguments: '--configuration $(buildConfiguration) --no-build --filter $(agentTestFilter)'

So the results in the Azure DevOps UI show each of the jobs and seemed to run quite quickly.

Conclusion

The tests running on ubuntu and split by PowerShell seemed to be significantly faster than using the VSTest task on Windows even though there are very few tests.

This has been a fun to workout how to run C# UI tests using Azure Pipelines across multiple agents and trying different techniques. I hope that this is useful for others writing Azure Pipelines for their UI tests.