Azure Pipelines, IaC

Azure Pipelines – Complex Types and Terraform

I’ve been using Terraform for a while with Azure Pipelines and have always passed the pipeline parameters or variables to Terraform using the -var command line parameter. This has worked really well until I encountered a need to pass more complex objects into Terraform which supports objects, maps and lists.

The Problem

When attempting to pass complex objects into -var Azure Pipelines outputs errors like ‘object is not a string’. After trying a number of work arounds that failed I ended up changing my Terraform to take strings and then perform actions on them e.g. an array as string then using split in Terraform to re-create the array.

This lead me to thinking “There has to be a better way”. Naturally one option is to create a .tfvars.json file and then substitute the variables using the same technique I used in my previous article Azure Pipelines – Parameters + JSON File Substitution. This would work for the most part but would not solve using array types.

I started thinking, could I get a parameter into a script and then somehow workout if it was a complex object and then write code to extract the value into something useful like JSON. This lead me to a community post that mentioned a function convertToJson.

A Solution

Based on using convertToJson and combing the technique from my previous article I came up with a step to create a HCL formatted .auto.tfvars file. The only thing is that for objects the colons ‘:’ need converting to equals ‘=’.

- ${{ each item in parameters }}:
     - script: echo '${{ item.key }}=${{ replace(convertToJson(item.value), ':', '=')}}' >> parameters.auto.tfvars
       displayName: "JsonVar ${{ item.key }}"

The .auto.tfvars file is automatically loaded by Terraform which removes the need to specify any -var or -var-file options.

Example Pipeline

For my example pipeline I have used an object for Tags and an array for a list of Network addresses for use with a Network Security Group.
The initial pipeline setups up the parameters and the Azure Storage Account for my Terraform state files.

trigger: none

pr: none

parameters:
  - name: resourceGroup
    displayName: Resource Group
    type: string
    default: 'terraform-test'
  - name: resourceLocation
    displayName: Resource Location
    type: string
    default: 'uksouth'
  - name: projectName
    displayName: Project Tag Name
    type: string
    default: 'Demo'
  - name: tags
    type: object
    default:
      Project: "Demo"
      Environment: "Dev"
  - name: network_source_addresses
    displayName: Network Address List
    type: object
    default:
      - "192.168.1.20"
      - "192.168.1.254"

variables:
  subscription: 'My Subscription'
  terraformVersion: '0.14.6'
  terraformResourceGroup: 'test-deployment'
  terraformStorageName: 'demoterraformstore'
  terrformStorageSku: Standard_LRS
  terraformContainerName: 'terraform'
  terraformStateFilename: test.tfstate

pool:
  vmImage: "ubuntu-latest"

steps:
- task: AzureCLI@2
  displayName: 'Azure CLI'
  inputs:
    azureSubscription: $(subscription)
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      az group create --location ${{ parameters.resourceLocation }} --name $(terraformResourceGroup)
      az storage account create --name $(terraformStorageName) --resource-group $(terraformResourceGroup) --location ${{ parameters.resourceLocation }} --sku $(terrformStorageSku) --tags "project=${{ parameters.projectName }}"
      az storage container create --name $(terraformContainerName) --account-name $(terraformStorageName)
    addSpnToEnvironment: false
- template: deploy.yml
  parameters:
    resourceGroup: ${{ parameters.resourceGroup }}
    resourceLocation: ${{ parameters.resourceLocation }}
    tags: ${{ parameters.tags }}
    network_source_addresses: ${{ parameters.network_source_addresses }}
    secret_value: $(secret_value)

I separated the Terraform parts into a template so that the loop only uses the parameters that are needed for Terraform and not any others that are in the main pipeline.
Note: I use the Microsoft Terraform Azure Pipelines Extension to deploy the Terraform scripts.

parameters:
  - name: resourceGroup
    type: string
  - name: resourceLocation
    type: string
  - name: tags
    type: object
  - name: network_source_addresses
    type: object
  - name: secret_value
    type: string

steps:
- ${{ each item in parameters }}:
     - script: echo '${{ item.key }}=${{ replace(convertToJson(item.value), ':', '=')}}' >> parameters.auto.tfvars
       displayName: "JsonVar ${{ item.key }}"
- bash: |
    cat parameters.auto.tfvars
  displayName: "Debug show new file"
- task: TerraformInstaller@0
  displayName: 'Install Terraform'
  inputs:
    terraformVersion: $(terraformVersion)
  
- task: TerraformTaskV1@0
  displayName: 'Terraform Init'
  inputs:
    backendServiceArm: $(subscription)
    backendAzureRmResourceGroupName: '$(terraformResourceGroup)'
    backendAzureRmStorageAccountName: '$(terraformStorageName)'
    backendAzureRmContainerName: '$(terraformContainerName)'
    backendAzureRmKey:  '$(terraformStateFilename)'
- task: TerraformTaskV1@0
  displayName: 'Terraform Plan'
  inputs:
    command: plan    
    environmentServiceNameAzureRM: $(subscription)
- task: TerraformTaskV1@0
  displayName: 'Terraform Apply'
  inputs:
    command: apply
    commandOptions: -auto-approve
    environmentServiceNameAzureRM: $(subscription)

Conclusion

I think this is a nice technique for using complex types in Azure Pipelines for use with Terraform deployments or anything else that would benefit from this idea. This was a fun problem to try and solve and I hope that sharing this helps others who have encountered the same problem.