Azure, Azure Pipelines

Azure Pipelines – Parameters + JSON File Substitution

Posted on by Mark Johnson in Azure, Azure Pipelines

Azure Pipelines provides a FileTransform Task for variable substitution in configuration files, so given an appsettings file like this:

{
    "Logging": {
        "IncludeScopes": false,
        "LogLevel": {
            "Default": "Debug",
            "System": "Information",
            "Microsoft": "Information"
        }
    },
    "WeatherSettings": {
        "DefaultTemperatureType": "Celsius",
        "DefaultWindType": "MPH",
        "DefaultTemp": 30,
        "ShowTemp": true
    }
}

We could create pipeline variables that allow changes to the nested values e.g.

variables:
  WeatherSettings.DefaultTemperatureType: 'Fahrenheit'
  WeatherSettings.DefaultWindType: 'KMH'
  WeatherSettings.DefaultTemp: 12
  WeatherSettings.ShowTemp: false

So we could create a basic pipeline for the substitution :

trigger: none

pool:
  vmImage: 'windows-latest'

variables:
  appsettingsfile: appsettings.json
  WeatherSettings.DefaultTemperatureType: 'Fahrenheit'
  WeatherSettings.DefaultWindType: 'KMH'
  WeatherSettings.DefaultTemp: 12
  WeatherSettings.ShowTemp: false
  
steps:
  - task: FileTransform@2
    displayName: "Transform Json"
    inputs:
      folderPath: '$(System.DefaultWorkingDirectory)/**/'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/$(appsettingsfile)'
  - bash: |
      cat $(appsettingsfile)
    displayName: "Show Json substitution"

Now, what if we wanted to add parameters to set the values when running the pipeline?

We can turn all of those variables into parameters right? Seems a reasonable idea.

trigger: none

pool:
  vmImage: 'windows-latest'

parameters:
 - name: WeatherSettings.DefaultTemperatureType
   type: string
   default: 'Fahrenheit'
 - name: WeatherSettings.DefaultWindType
   type: string
   default: 'KMH'
 - name: WeatherSettings.DefaultTemp
   type: number
   default: 12
 - name: WeatherSettings.ShowTemp
   type: boolean
   default: false

variables:
  appsettingsfile: appsettings.json
  
steps:
  - task: FileTransform@2
    displayName: "Transform Json"
    inputs:
      folderPath: '$(System.DefaultWorkingDirectory)/**/'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/$(appsettingsfile)'
  - bash: |
      cat $(appsettingsfile)
    displayName: "Show Json substitution"

Well that doesn’t work, the parameter values are not picked up by the FileTransform Task and so no substitution happens. So, what if we define the variables again and assign the values from the parameters.

trigger: none

pool:
  vmImage: 'windows-latest'

parameters:
 - name: WeatherSettings.DefaultTemperatureType
   type: string
   default: 'Fahrenheit'
 - name: WeatherSettings.DefaultWindType
   type: string
   default: 'KMH'
 - name: WeatherSettings.DefaultTemp
   type: number
   default: 12
 - name: WeatherSettings.ShowTemp
   type: boolean
   default: false

variables:
  appsettingsfile: appsettings.json
  WeatherSettings.DefaultTemperatureType: ${{ parameters.WeatherSettings.DefaultTemperatureType }}
  WeatherSettings.DefaultWindType: ${{ parameters.WeatherSettings.DefaultWindType }}
  WeatherSettings.DefaultTemp: ${{ parameters.WeatherSettings.DefaultTemp }}
  WeatherSettings.ShowTemp: ${{ parameters.WeatherSettings.ShowTemp }}
steps:
  - task: FileTransform@2
    displayName: "Transform Json"
    inputs:
      folderPath: '$(System.DefaultWorkingDirectory)/**/'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/$(appsettingsfile)'
  - bash: |
      cat $(appsettingsfile)
    displayName: "Show Json substitution"

That doesn’t work either, using nested parameter names threw an error “Key not found”. OK so what if we change the parameters to not use nested names.

trigger: none

pool:
  vmImage: 'windows-latest'

parameters:
 - name: DefaultTemperatureType
   type: string
   default: 'Fahrenheit'
 - name: DefaultWindType
   type: string
   default: 'KMH'
 - name: DefaultTemp
   type: number
   default: 12
 - name: ShowTemp
   type: boolean
   default: false

variables:
  appsettingsfile: appsettings.json
  WeatherSettings.DefaultTemperatureType: ${{ parameters.DefaultTemperatureType }}
  WeatherSettings.DefaultWindType: ${{ parameters.DefaultWindType }}
  WeatherSettings.DefaultTemp: ${{ parameters.DefaultTemp }}
  WeatherSettings.ShowTemp: ${{ parameters.ShowTemp }}
steps:
  - task: FileTransform@2
    displayName: "Transform Json"
    inputs:
      folderPath: '$(System.DefaultWorkingDirectory)/**/'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/$(appsettingsfile)'
  - bash: |
      cat $(appsettingsfile)
    displayName: "Show Json substitution"

Great!! this now works and provides the substitution. Ah! but the boolean values become a string and the casing is invalid JSON.

"WeatherSettings": {
        "DefaultTemperatureType": "Fahrenheit",
        "DefaultWindType": "KMH",
        "DefaultTemp": 12,
        "ShowTemp": "False"
    }

At the time of writing this is a known issue with the FileTransform Task but there is a work around. Simply change the boolean variables to be a string.

parameters:
 - name: DefaultTemperatureType
   type: string
   default: 'Fahrenheit'
 - name: DefaultWindType
   type: string
   default: 'KMH'
 - name: DefaultTemp
   type: number
   default: 12
 - name: ShowTemp
   type: string
   default: 'false'

This now has the correct output and adds the boolean value in the JSON.

 "WeatherSettings": {
        "DefaultTemperatureType": "Fahrenheit",
        "DefaultWindType": "KMH",
        "DefaultTemp": 12,
        "ShowTemp": false
    }

It’s good this is now working but it seems a bit excessive to add duplicate variable for each parameter in order to successfully get this to work. What could we do to improve that?

Well one thing we can do is to use a loop to turn all of the parameters into variables at runtime.

trigger: none

pool:
  vmImage: 'windows-latest'

parameters:
 - name: WeatherSettings.DefaultTemperatureType
   type: string
   default: 'Fahrenheit'
 - name: WeatherSettings.DefaultWindType
   type: string
   default: 'KMH'
 - name: WeatherSettings.DefaultTemp
   type: number
   default: 12
 - name: WeatherSettings.ShowTemp
   type: string
   default: 'false'

variables:
  appsettingsfile: appsettings.json
steps:
  - ${{ each item in parameters }}:
     - bash: |
           echo "##vso[task.setvariable variable=${{ item.key }}]${{ item.value }}"
       displayName: "Create Variable ${{ item.key }}"
  - task: FileTransform@2
    displayName: "Transform Json"
    inputs:
      folderPath: '$(System.DefaultWorkingDirectory)/**/'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/$(appsettingsfile)'
  - bash: |
      cat $(appsettingsfile)
    displayName: "Show Json substitution"

Note: The Azure Pipelines editor shows the loop underlined and reports an error “The first property must be task”.

But when running validate.

The result is OK. Hopefully this issue will get resolved in the future and not show as an error in the UI.

So the loop worked and provided all of the variables to be substituted. We could leave it there as it works, but we could also use some of the other parameter properties such as displayName and values to provide a nicer configuration. 

trigger: none

pool:
  vmImage: 'windows-latest'

parameters:
 - name: WeatherSettings.DefaultTemperatureType
   displayName: Temperature Type
   type: string
   default: 'Celsius'
   values:
     - 'Celsius'
     - 'Fahrenheit'
 - name: WeatherSettings.DefaultWindType
   displayName: Wind Type
   type: string
   default: 'MPH'
   values:
     - 'MPH'
     - 'KMH'
 - name: WeatherSettings.DefaultTemp
   displayName: Temperature
   type: number
   default: 30
 - name: WeatherSettings.ShowTemp
   displayName: Show Temperature
   type: string
   default: 'true'
   values:
     - 'true'
     - 'false'

variables:
  appsettingsfile: appsettings.json
  
steps:
  - ${{ each item in parameters }}:
    - bash: |
        echo "##vso[task.setvariable variable=${{ item.key }}]${{ item.value }}"
      displayName: "Create Variable ${{ item.key }}"
  - task: FileTransform@2
    displayName: "Transform Json"
    inputs:
      folderPath: '$(System.DefaultWorkingDirectory)/**/'
      xmlTransformationRules: ''
      jsonTargetFiles: '**/$(appsettingsfile)'
  - bash: |
      cat $(appsettingsfile)
    displayName: "Show Json substitution"

In the Azure DevOps UI we can see the parameters have nice names instead of the nested ones and we can choose expected values.

I found this technique works really well and have already used it in a pipeline. You can also use a condition to only create a variable for parameters that starts with or contains etc. e.g.

  - ${{ each item in parameters }}:
      - bash: |
            echo "##vso[task.setvariable variable=${{ item.key }}]${{ item.value }}"
        displayName: "Create Variable ${{ item.key }}"
        condition: startswith('${{ item.key }}', 'WeatherSettings')

One thought on “Azure Pipelines – Parameters + JSON File Substitution”

  1. Pingback: Azure Pipelines – Complex Types and Terraform | Coding With Taz

Comments are closed.