I’ve been using Terraform for a while with Azure Pipelines and have always passed the pipeline parameters or variables to Terraform using the -var command line parameter. This has worked really well until I encountered a need to pass more complex objects into Terraform which supports objects, maps and lists.
The Problem
When attempting to pass complex objects into -var Azure Pipelines outputs errors like ‘object is not a string’. After trying a number of work arounds that failed I ended up changing my Terraform to take strings and then perform actions on them e.g. an array as string then using split in Terraform to re-create the array.
This lead me to thinking “There has to be a better way”. Naturally one option is to create a .tfvars.json file and then substitute the variables using the same technique I used in my previous article Azure Pipelines – Parameters + JSON File Substitution. This would work for the most part but would not solve using array types.
I started thinking, could I get a parameter into a script and then somehow workout if it was a complex object and then write code to extract the value into something useful like JSON. This lead me to a community post that mentioned a function convertToJson.
A Solution
Based on using convertToJson and combing the technique from my previous article I came up with a step to create a HCL formatted .auto.tfvars file. The only thing is that for objects the colons ‘:’ need converting to equals ‘=’.
- ${{ each item in parameters }}:
- script: echo '${{ item.key }}=${{ replace(convertToJson(item.value), ':', '=')}}' >> parameters.auto.tfvars
displayName: "JsonVar ${{ item.key }}"
The .auto.tfvars file is automatically loaded by Terraform which removes the need to specify any -var or -var-file options.
Example Pipeline
For my example pipeline I have used an object for Tags and an array for a list of Network addresses for use with a Network Security Group.
The initial pipeline setups up the parameters and the Azure Storage Account for my Terraform state files.
trigger: none
pr: none
parameters:
- name: resourceGroup
displayName: Resource Group
type: string
default: 'terraform-test'
- name: resourceLocation
displayName: Resource Location
type: string
default: 'uksouth'
- name: projectName
displayName: Project Tag Name
type: string
default: 'Demo'
- name: tags
type: object
default:
Project: "Demo"
Environment: "Dev"
- name: network_source_addresses
displayName: Network Address List
type: object
default:
- "192.168.1.20"
- "192.168.1.254"
variables:
subscription: 'My Subscription'
terraformVersion: '0.14.6'
terraformResourceGroup: 'test-deployment'
terraformStorageName: 'demoterraformstore'
terrformStorageSku: Standard_LRS
terraformContainerName: 'terraform'
terraformStateFilename: test.tfstate
pool:
vmImage: "ubuntu-latest"
steps:
- task: AzureCLI@2
displayName: 'Azure CLI'
inputs:
azureSubscription: $(subscription)
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
az group create --location ${{ parameters.resourceLocation }} --name $(terraformResourceGroup)
az storage account create --name $(terraformStorageName) --resource-group $(terraformResourceGroup) --location ${{ parameters.resourceLocation }} --sku $(terrformStorageSku) --tags "project=${{ parameters.projectName }}"
az storage container create --name $(terraformContainerName) --account-name $(terraformStorageName)
addSpnToEnvironment: false
- template: deploy.yml
parameters:
resourceGroup: ${{ parameters.resourceGroup }}
resourceLocation: ${{ parameters.resourceLocation }}
tags: ${{ parameters.tags }}
network_source_addresses: ${{ parameters.network_source_addresses }}
secret_value: $(secret_value)
I separated the Terraform parts into a template so that the loop only uses the parameters that are needed for Terraform and not any others that are in the main pipeline.
Note: I use the Microsoft Terraform Azure Pipelines Extension to deploy the Terraform scripts.
parameters:
- name: resourceGroup
type: string
- name: resourceLocation
type: string
- name: tags
type: object
- name: network_source_addresses
type: object
- name: secret_value
type: string
steps:
- ${{ each item in parameters }}:
- script: echo '${{ item.key }}=${{ replace(convertToJson(item.value), ':', '=')}}' >> parameters.auto.tfvars
displayName: "JsonVar ${{ item.key }}"
- bash: |
cat parameters.auto.tfvars
displayName: "Debug show new file"
- task: TerraformInstaller@0
displayName: 'Install Terraform'
inputs:
terraformVersion: $(terraformVersion)
- task: TerraformTaskV1@0
displayName: 'Terraform Init'
inputs:
backendServiceArm: $(subscription)
backendAzureRmResourceGroupName: '$(terraformResourceGroup)'
backendAzureRmStorageAccountName: '$(terraformStorageName)'
backendAzureRmContainerName: '$(terraformContainerName)'
backendAzureRmKey: '$(terraformStateFilename)'
- task: TerraformTaskV1@0
displayName: 'Terraform Plan'
inputs:
command: plan
environmentServiceNameAzureRM: $(subscription)
- task: TerraformTaskV1@0
displayName: 'Terraform Apply'
inputs:
command: apply
commandOptions: -auto-approve
environmentServiceNameAzureRM: $(subscription)
Conclusion
I think this is a nice technique for using complex types in Azure Pipelines for use with Terraform deployments or anything else that would benefit from this idea. This was a fun problem to try and solve and I hope that sharing this helps others who have encountered the same problem.
