Azure DevOps | Coding With Taz

When creating a build pipeline one area that seems to get a lot of attention is running UI tests. We want to get feedback as quickly as possible about our code changes.

Test frameworks tend to offer a way of running tests in parallel which is great but what if I want to split tests up over multiple machines and run them in parallel? This isn’t a new problem and there have been many solutions to this, for this article I am going to focus on achieving this in Azure DevOps Pipelines with some .NET code and just using YAML.

UI Tests

I have a project in Visual Studio that contains a basic ASP.NET Core website project and a UI test project using Selenium and NUnit. I deployed the website to Azure ready for the tests to be ran against it.

Azure Pipelines provides a Parallel Strategy, combining that with the VSTest Task will automatically divide up the tests up based on the number of parallel agents.

For this example I am going to use two jobs, one to build the tests and create the output as artifacts and one to download the tests and perform the test run. The second job is configured to run on 5 agents.

The YAML for the pipeline looks like this:

trigger: 
  - master

variables:
  buildConfiguration: Release
  uiTestFolder: 'uitests'

jobs:
- job: BuildTests
  displayName: Build UI Tests
  pool:
    vmImage: windows-latest
  steps:
  - task: DotNetCoreCLI@2
    displayName: Restore Packages
    inputs:
      command: 'restore'
      projects: 'mytests/*.csproj'
  - task: DotNetCoreCLI@2
    displayName: Build Tests
    inputs:
      command: 'build'
      projects: '**/mytests.csproj'
      arguments: '--configuration $(buildConfiguration) -o $(Build.ArtifactStagingDirectory)/$(uiTestFolder)'
  - task: PublishPipelineArtifact@1
    displayName: Upload Library
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)/$(uiTestFolder)'
      artifactName: $(uiTestFolder)  
- job: RunTests
  displayName: Run UI Tests
  dependsOn: BuildTests
  pool:
    vmImage: windows-latest
  strategy:
    parallel: 5
  variables:
      siteName: mytest-app
      baseSiteUrl: 'https://$(siteName).azurewebsites.net/'
  steps:
  - checkout: none
  - task: DownloadPipelineArtifact@2
    displayName: Download Tests
    inputs:
      buildType: 'current'
      artifactName: '$(uiTestFolder)'
      targetPath: '$(Pipeline.Workspace)/$(uiTestFolder)'
  - task: FileTransform@2
    displayName: Configure Test Run
    inputs:
      folderPath: '$(Pipeline.Workspace)'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/*settings.json'
  - task: PowerShell@2
    displayName: Check File Substitution
    inputs:
      targetType: 'inline'
      script: 'Get-Content -Path $(Pipeline.Workspace)/$(uiTestFolder)/testsettings.json'
      pwsh: true
  - task: VSTest@2
    displayName: Run UI Tests
    inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2: |
        **\*tests.dll
        !**\*TestAdapter.dll
        !**\obj\**
      searchFolder: '$(Pipeline.Workspace)/$(uiTestFolder)'
      uiTests: true
      runInParallel: false
      testRunTitle: 'Basic UI Tests'

NOTE: At the time of writing the VSTest Task can only run on the windows agents with either Visual Studio Installed or by using the Visual Studio Test Platform Installer Task.

You might notice that in the VSTest task ‘runInParallel’ is set to false, this is because UI tests can cause issues running in parallel on the same box.

After the test has been ran you can see the results in the Azure DevOps UI. There is the Build UI Tests job and then each Run UI Tests is given a number for each parallel run on a different agent.

NOTE: the number of parallel jobs depend on the amount of agents you can run in your parallel in your Azure DevOps configuration and the number of available agents when the job runs.

Conclusion

Using the VSTest task and parallel strategy is a very simple way to get UI tests to run on multiple agents with a .NET project.

I found my example didn’t really offer much gain in terms of time but I only had 14 UI tests and just using one agent performed better than multiple agents. I expect the gains would be seen with a large test suite, I would suggest trying the test runs with different numbers of agents to see what works best for your tests. You might also gain benefits running the UI tests on your own agents.

Additional Information

Azure Pipelines provides a FileTransform Task for variable substitution in configuration files, so given an appsettings file like this:

{
    "Logging": {
        "IncludeScopes": false,
        "LogLevel": {
            "Default": "Debug",
            "System": "Information",
            "Microsoft": "Information"
        }
    },
    "WeatherSettings": {
        "DefaultTemperatureType": "Celsius",
        "DefaultWindType": "MPH",
        "DefaultTemp": 30,
        "ShowTemp": true
    }
}

We could create pipeline variables that allow changes to the nested values e.g.

variables:
  WeatherSettings.DefaultTemperatureType: 'Fahrenheit'
  WeatherSettings.DefaultWindType: 'KMH'
  WeatherSettings.DefaultTemp: 12
  WeatherSettings.ShowTemp: false

So we could create a basic pipeline for the substitution :

trigger: none

pool:
  vmImage: 'windows-latest'

variables:
  appsettingsfile: appsettings.json
  WeatherSettings.DefaultTemperatureType: 'Fahrenheit'
  WeatherSettings.DefaultWindType: 'KMH'
  WeatherSettings.DefaultTemp: 12
  WeatherSettings.ShowTemp: false
  
steps:
  - task: FileTransform@2
    displayName: "Transform Json"
    inputs:
      folderPath: '$(System.DefaultWorkingDirectory)/**/'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/$(appsettingsfile)'
  - bash: |
      cat $(appsettingsfile)
    displayName: "Show Json substitution"

Now, what if we wanted to add parameters to set the values when running the pipeline?

We can turn all of those variables into parameters right? Seems a reasonable idea.

trigger: none

pool:
  vmImage: 'windows-latest'

parameters:
 - name: WeatherSettings.DefaultTemperatureType
   type: string
   default: 'Fahrenheit'
 - name: WeatherSettings.DefaultWindType
   type: string
   default: 'KMH'
 - name: WeatherSettings.DefaultTemp
   type: number
   default: 12
 - name: WeatherSettings.ShowTemp
   type: boolean
   default: false

variables:
  appsettingsfile: appsettings.json
  
steps:
  - task: FileTransform@2
    displayName: "Transform Json"
    inputs:
      folderPath: '$(System.DefaultWorkingDirectory)/**/'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/$(appsettingsfile)'
  - bash: |
      cat $(appsettingsfile)
    displayName: "Show Json substitution"

Well that doesn’t work, the parameter values are not picked up by the FileTransform Task and so no substitution happens. So, what if we define the variables again and assign the values from the parameters.

trigger: none

pool:
  vmImage: 'windows-latest'

parameters:
 - name: WeatherSettings.DefaultTemperatureType
   type: string
   default: 'Fahrenheit'
 - name: WeatherSettings.DefaultWindType
   type: string
   default: 'KMH'
 - name: WeatherSettings.DefaultTemp
   type: number
   default: 12
 - name: WeatherSettings.ShowTemp
   type: boolean
   default: false

variables:
  appsettingsfile: appsettings.json
  WeatherSettings.DefaultTemperatureType: ${{ parameters.WeatherSettings.DefaultTemperatureType }}
  WeatherSettings.DefaultWindType: ${{ parameters.WeatherSettings.DefaultWindType }}
  WeatherSettings.DefaultTemp: ${{ parameters.WeatherSettings.DefaultTemp }}
  WeatherSettings.ShowTemp: ${{ parameters.WeatherSettings.ShowTemp }}
steps:
  - task: FileTransform@2
    displayName: "Transform Json"
    inputs:
      folderPath: '$(System.DefaultWorkingDirectory)/**/'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/$(appsettingsfile)'
  - bash: |
      cat $(appsettingsfile)
    displayName: "Show Json substitution"

That doesn’t work either, using nested parameter names threw an error “Key not found”. OK so what if we change the parameters to not use nested names.

trigger: none

pool:
  vmImage: 'windows-latest'

parameters:
 - name: DefaultTemperatureType
   type: string
   default: 'Fahrenheit'
 - name: DefaultWindType
   type: string
   default: 'KMH'
 - name: DefaultTemp
   type: number
   default: 12
 - name: ShowTemp
   type: boolean
   default: false

variables:
  appsettingsfile: appsettings.json
  WeatherSettings.DefaultTemperatureType: ${{ parameters.DefaultTemperatureType }}
  WeatherSettings.DefaultWindType: ${{ parameters.DefaultWindType }}
  WeatherSettings.DefaultTemp: ${{ parameters.DefaultTemp }}
  WeatherSettings.ShowTemp: ${{ parameters.ShowTemp }}
steps:
  - task: FileTransform@2
    displayName: "Transform Json"
    inputs:
      folderPath: '$(System.DefaultWorkingDirectory)/**/'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/$(appsettingsfile)'
  - bash: |
      cat $(appsettingsfile)
    displayName: "Show Json substitution"

Great!! this now works and provides the substitution. Ah! but the boolean values become a string and the casing is invalid JSON.

"WeatherSettings": {
        "DefaultTemperatureType": "Fahrenheit",
        "DefaultWindType": "KMH",
        "DefaultTemp": 12,
        "ShowTemp": "False"
    }

At the time of writing this is a known issue with the FileTransform Task but there is a work around. Simply change the boolean variables to be a string.

parameters:
 - name: DefaultTemperatureType
   type: string
   default: 'Fahrenheit'
 - name: DefaultWindType
   type: string
   default: 'KMH'
 - name: DefaultTemp
   type: number
   default: 12
 - name: ShowTemp
   type: string
   default: 'false'

This now has the correct output and adds the boolean value in the JSON.

 "WeatherSettings": {
        "DefaultTemperatureType": "Fahrenheit",
        "DefaultWindType": "KMH",
        "DefaultTemp": 12,
        "ShowTemp": false
    }

It’s good this is now working but it seems a bit excessive to add duplicate variable for each parameter in order to successfully get this to work. What could we do to improve that?

Well one thing we can do is to use a loop to turn all of the parameters into variables at runtime.

trigger: none

pool:
  vmImage: 'windows-latest'

parameters:
 - name: WeatherSettings.DefaultTemperatureType
   type: string
   default: 'Fahrenheit'
 - name: WeatherSettings.DefaultWindType
   type: string
   default: 'KMH'
 - name: WeatherSettings.DefaultTemp
   type: number
   default: 12
 - name: WeatherSettings.ShowTemp
   type: string
   default: 'false'

variables:
  appsettingsfile: appsettings.json
steps:
  - ${{ each item in parameters }}:
     - bash: |
           echo "##vso[task.setvariable variable=${{ item.key }}]${{ item.value }}"
       displayName: "Create Variable ${{ item.key }}"
  - task: FileTransform@2
    displayName: "Transform Json"
    inputs:
      folderPath: '$(System.DefaultWorkingDirectory)/**/'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/$(appsettingsfile)'
  - bash: |
      cat $(appsettingsfile)
    displayName: "Show Json substitution"

Note: The Azure Pipelines editor shows the loop underlined and reports an error “The first property must be task”.

But when running validate.

The result is OK. Hopefully this issue will get resolved in the future and not show as an error in the UI.

So the loop worked and provided all of the variables to be substituted. We could leave it there as it works, but we could also use some of the other parameter properties such as displayName and values to provide a nicer configuration.

trigger: none

pool:
  vmImage: 'windows-latest'

parameters:
 - name: WeatherSettings.DefaultTemperatureType
   displayName: Temperature Type
   type: string
   default: 'Celsius'
   values:
     - 'Celsius'
     - 'Fahrenheit'
 - name: WeatherSettings.DefaultWindType
   displayName: Wind Type
   type: string
   default: 'MPH'
   values:
     - 'MPH'
     - 'KMH'
 - name: WeatherSettings.DefaultTemp
   displayName: Temperature
   type: number
   default: 30
 - name: WeatherSettings.ShowTemp
   displayName: Show Temperature
   type: string
   default: 'true'
   values:
     - 'true'
     - 'false'

variables:
  appsettingsfile: appsettings.json
  
steps:
  - ${{ each item in parameters }}:
    - bash: |
        echo "##vso[task.setvariable variable=${{ item.key }}]${{ item.value }}"
      displayName: "Create Variable ${{ item.key }}"
  - task: FileTransform@2
    displayName: "Transform Json"
    inputs:
      folderPath: '$(System.DefaultWorkingDirectory)/**/'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/$(appsettingsfile)'
  - bash: |
      cat $(appsettingsfile)
    displayName: "Show Json substitution"

In the Azure DevOps UI we can see the parameters have nice names instead of the nested ones and we can choose expected values.

I found this technique works really well and have already used it in a pipeline. You can also use a condition to only create a variable for parameters that starts with or contains etc. e.g.

  - ${{ each item in parameters }}:
      - bash: |
            echo "##vso[task.setvariable variable=${{ item.key }}]${{ item.value }}"
        displayName: "Create Variable ${{ item.key }}"
        condition: startswith('${{ item.key }}', 'WeatherSettings')

Coding With Taz

Coding With Taz

Sharing the joy of coding

Tag: Azure DevOps

Azure Pipelines – Running UI Tests on Multiple Agents

UI Tests

Conclusion

Additional Information

Azure Pipelines – Parameters + JSON File Substitution