Azure, Bicep, DevOps, IaC

Bicep – Variable JSON Files

Introduction

Recently I was looking into others ways to provide inputs to Bicep. In previous posts I have used various techniques to pass parameters in via Azure Pipelines, either updating a parameters JSON file or creating the whole file in the pipeline (I bundled those techniques into one post if you want to take a look).

If you are not sure about creating a parameters file to use with Bicep I recommend the Visual Studio Code extension Azure Resource Manager (ARM) Tools as this provides an easy way to create them. Also there is a PowerShell module written by Stefan Ivemo (and others) that has a command ‘New-BicepParameterFile’ that creates a parameter file from a Bicep file (see github and/or the PowerShell gallery)

In other Infrastructure as Code (IaC) frameworks like Terraform, there are ways to load variable files in and they are usually in the same language as the framework. Bicep however doesn’t currently support variables files in the Bicep language, but as I discovered, it does have a function called loadTextContent, this coupled with the json function allows JSON files to be loaded in and therefore could be used as variable files.

So, in this post I am going to explore using these functions to load variables into Bicep.

NOTE: I will be using az bicep version 0.6.1

The Code

Let’s start with something small like creating a storage account.

First create some settings for the account, name, sku, etc. in a JSON file (vars_storage.json) e.g.

{
    "name": "stmvarsdemodevweu",
    "sku": "Standard_LRS",
    "kind": "StorageV2",
    "properties": {
        "minimumTlsVersion": "TLS1_2"
    }
}

Next create the bicep file (deploy.bicep) to create the storage account and load in the variables from the json file using loadTextContent and json functions:

param location string = resourceGroup().location 
var storageConfig = json(loadTextContent('vars_storage.json'))

resource storageaccount 'Microsoft.Storage/storageAccounts@2021-09-01' = {
  name: storageConfig.name
  location: location
  kind: storageConfig.kind
  sku: {
    name: storageConfig.sku
  }
  properties: {
    minimumTlsVersion: storageConfig.properties.minimumTlsVersion
  }
}

NOTE: With the Visual Studio Code Bicep extension you get intellisense on the json properties as well which is really nice

I already created the resource group in Azure and so just need to deploy the Bicep via the Azure CLI

 az deployment group create --resource-group rg-varsdemo-dev-weu --template-file deploy.bicep

Now, what if we wanted multiple variable files? as you can imagine you can just load in another file.

Let’s add a Virtual network to Bicep.

As before, starting with the JSON file (vars_network.json) define the network configuration

{
    "name": "vnet-varsdemo-dev-weu",
    "addressPrefixes": [
        "10.72.0.0/16"
    ],
    "subnets": [
        {
            "name": "Subnet-1",
            "properties": {
                "addressPrefix": "10.72.252.0/24"
            }
        },
        {
            "name": "Subnet-2",
            "properties": {
                "addressPrefix": "10.72.254.0/24"
            }
        }
    ]
}

And then add the Virtual Network to Bicep and load in the new json file into a variable

param location string = resourceGroup().location 
var storageConfig = json(loadTextContent('vars_storage.json'))
var networkConfig = json(loadTextContent('vars_network.json'))

resource storageaccount 'Microsoft.Storage/storageAccounts@2021-09-01' = {
  name: storageConfig.name
  location: location
  kind: storageConfig.kind
  sku: {
    name: storageConfig.sku
  }
  properties: {
    minimumTlsVersion: storageConfig.properties.minimumTlsVersion
  }
}

resource virtualNetwork 'Microsoft.Network/virtualNetworks@2019-11-01' = {
  name: networkConfig.name
  location: location
  properties: {
    addressSpace: {
      addressPrefixes: networkConfig.addressPrefixes
    }
    subnets: networkConfig.subnets
  }
}

Multiple Environments

Often when we use IaC we are deploying to multiple environments. Can files be dynamically loaded by a parameter value? something like this?

param location string = resourceGroup().location
param env string
var storageConfig = json(loadTextContent('vars_${env}_storage.json'))
var networkConfig = json(loadTextContent('vars_${env}_network.json'))

Unfortunately this is not a supported configuration in Bicep

The reason for this is that Param is determined at runtime and loadTextContent is determined at compile time

It is possible however to define environments in the file and load the desired section at runtime.

NOTE: One downside of using this way of loading environments is that Visual Studio Code loses intellisense on the JSON properties due to the runtime env parameter

Let’s change the JSON files to support multiple environments

vars_storage.json

{
    "dev": {
        "name": "stmvarsdemodevweu",
        "sku": "Standard_LRS",
        "kind": "StorageV2",
        "properties": {
            "minimumTlsVersion": "TLS1_2"
        }
    },
    "prod": {
        "name": "stmvarsdemoprodweu",
        "sku": "Premium_LRS",
        "kind": "StorageV2",
        "properties": {
            "minimumTlsVersion": "TLS1_2"
        }
    }
}

vars_network.json

{
    "dev": {
        "name": "vnet-varsdemo-dev-weu",
        "addressPrefixes": [
            "10.72.0.0/16"
        ],
        "subnets": [
            {
                "name": "Subnet-1",
                "properties": {
                    "addressPrefix": "10.72.252.0/24"
                }
            },
            {
                "name": "Subnet-2",
                "properties": {
                    "addressPrefix": "10.72.254.0/24"
                }
            }
        ]
    },
    "prod": {
        "name": "vnet-varsdemo-dev-weu",
        "addressPrefixes": [
            "10.74.0.0/16"
        ],
        "subnets": [
            {
                "name": "Subnet-1",
                "properties": {
                    "addressPrefix": "10.74.252.0/24"
                }
            },
            {
                "name": "Subnet-2",
                "properties": {
                    "addressPrefix": "10.74.254.0/24"
                }
            }
        ]
    }
}

And finally add the loading of the environment in Bicep

param location string = resourceGroup().location
param env string
var storageConfig = json(loadTextContent('vars_storage.json'))['${env}'] 
var networkConfig = json(loadTextContent('vars_network.json'))['${env}'] 

resource storageaccount 'Microsoft.Storage/storageAccounts@2021-09-01' = {
  name: storageConfig.name
  location: location
  kind: storageConfig.kind
  sku: {
    name: storageConfig.sku
  }
  properties: {
    minimumTlsVersion: storageConfig.properties.minimumTlsVersion
  }
}

resource virtualNetwork 'Microsoft.Network/virtualNetworks@2019-11-01' = {
  name: networkConfig.name
  location: location
  properties: {
    addressSpace: {
      addressPrefixes: networkConfig.addressPrefixes
    }
    subnets: networkConfig.subnets
  }
}

As before this can now be deployed into Azure via the Azure CLI adding the env parameter

Dev

az deployment group create --resource-group rg-varsdemo-dev-weu --template-file deploy.bicep --parameters env=dev

Prod

az deployment group create --resource-group rg-varsdemo-prod-weu --template-file deploy.bicep --parameters env=prod

Conclusion

You can of course provide all the variables inside the Bicep file including multiple environments and that is a valid way of defining values. However if you have a lot of variables, using a separate file whether that be a parameters file or following this technique can reduce the noise inside the Bicep file. Separate files also allow values to be overridden by a CI/CD pipeline, share variables between multiple Bicep files and it might be easier to maintain configuration if it is in separate files.

I think loading files in is a nice way of injecting values into Bicep, its a shame that other file formats like YAML are not currently supported or using Bicep style variable/parameters files. Hopefully these features will become part of Bicep in the future.

Well I hope you found this useful and discovered another way to get inputs into Bicep. Bicep is certainly improving more and more and I look forward to future releases.

Happy IaCing !!

Azure, Azure Pipelines, Bicep, DevOps, IaC

Dynamic Multistage Azure Pipelines Part 2

In Part 1 I went through the idea of a base template to handle dynamic multistage pipelines and ended with a template to deal with multiple subscriptions and environments. In this part I am going to show real use by deploying some IaC (Infrastructure as Code) to create an Web App in Azure and then deploying an application to the Web App using that base template.

To recap, the base template looked liked this with one difference, I’ve added a QA environment:

parameters:
- name: environments
  type: object
  default:
  - name: 'dev'
    subscriptions:
      - subscription: 'Dev Subscription'
        regions:
          - location: 'eastus'
            locationShort: 'eus'
  - name: 'qa'
    subscriptions:
      - subscription: 'QA Subscription'
        regions:
          - location: 'eastus'
            locationShort: 'eus'
          - location: 'westus'
            locationShort: 'wus'
  - name: 'prod'
    subscriptions:
      - subscription: 'Prod Subscription'
        regions:
          - location: 'eastus'
            locationShort: 'eus'
          - location: 'westus'
            locationShort: 'wus'
- name: buildSteps
  type: stepList
  default: []
- name: releaseSteps
  type: stepList
  default: []
- name: customReleaseTemplate
  type: string
  default: ''
- name: variableGroupPrefix
  type: string
  default: ''
- name: variableFilePrefix
  type: string
  default: ''
 
stages:
- stage: build
  displayName: 'Build/Package Code or IaC'
  jobs:
  - job: build
    displayName: 'Build/Package Code'
    steps: ${{ parameters.buildSteps }}
 
- ${{ each env in parameters.environments }}:
  - stage: ${{ env.name }}
    displayName: 'Deploy to ${{ env.name }}'
    condition: succeeded()
    variables:
      - ${{ if ne(parameters.variableFilePrefix, '') }}:
        - template: ${{ parameters.variableFilePrefix }}_${{ env.name }}.yml@self
      - ${{ if ne(parameters.variableGroupPrefix, '') }}:
        - group: ${{ parameters.variableGroupPrefix }}_${{ env.name }}
    jobs:
    - ${{ each sub in env.subscriptions }}:
      - ${{ each region in sub.regions }}:
        - ${{ if ne(parameters.customReleaseTemplate, '') }}:
          - template: ${{ parameters.customReleaseTemplate }}
            parameters:
               env: ${{ env.name }}
               location: ${{ region.location }}
               locationShort: ${{ region.locationShort }}
               subscription: ${{ sub.subscription }}
        - ${{ else }}:
          - deployment: deploy_${{ region.locationShort }}
            displayName: 'Deploy app to ${{ env.name }} in ${{ region.location }}'
            environment: ${{ env.name }}_${{ region.locationShort }}
            strategy:
              runOnce:
                deploy:
                  steps:
                  - ${{ parameters.releaseSteps }}

My repository for this contains several folders, one called infrastructure which contains my IaC code (in this case Bicep) and one called src which holds a very basic .NET 6 web app.

Deploy IaC

As most of the configuration is in the base template this pipeline is going to extend the environments template and then add steps to package up the infrastructure and run the IaC via a custom release template deploy-infrastructure.yml.

trigger: none
pr: none

pool: 
  vmImage: 'ubuntu-latest'

resources:
  repositories:
    - repository: templates
      type: git
      name: shared-templates
      ref: main

extends:
  template: environments.yml@templates
  parameters:
    buildSteps:
    - task: CopyFiles@2
      displayName: 'Copy Files to output'
      inputs:
        contents: |
          **/infrastructure/**
        targetFolder: $(Build.ArtifactStagingDirectory)
    - task: ArchiveFiles@2
      displayName: 'Archive Infrastructure'
      inputs:
        rootFolderOrFile: $(Build.ArtifactStagingDirectory)
        includeRootFolder: false
        archiveType: 'zip'
        archiveFile: '$(Build.ArtifactStagingDirectory)/infrastructure$(Build.BuildId).zip'
    - publish: $(Build.ArtifactStagingDirectory)/infrastructure$(Build.BuildId).zip
      displayName: 'Upload Infrastructure package'
      artifact: infrastructure
    customReleaseTemplate: deploy-infrastructure.yml@self

The custom release template is mainly used so that I can dynamically load in some parameters files that will build the parameters JSON file for Bicep to use (if you want to know how I create the parameters JSON see my post on Passing Parameters to Bicep Technique #3).

deploy-infrastructure.yml

parameters:
- name: env
  type: string
- name: location
  type: string
- name: locationShort
  type: string
- name: subscription
  type: string

jobs:
- deployment: deploy_${{ parameters.locationShort }}
  displayName: 'Deploy app to ${{ parameters.env }} in ${{ parameters.location }}'
  environment: ${{ parameters.env }}_${{ parameters.locationShort }}
  variables:
    resourceGroupName: 'myapp-${{ parameters.env }}-${{ parameters.locationShort }}-rg'
    templateFileName: 'infrastructure/main.bicep'
    templateParametersFileName: 'infrastructure/main.parameters.json'
  strategy: 
    runOnce:
      deploy:
        steps:
        - task: ExtractFiles@1
          displayName: 'Extract Infrastructure'
          inputs:
            archiveFilePatterns: '$(Pipeline.Workspace)/**/*.zip'
            destinationFolder: $(Build.SourcesDirectory)
            cleanDestinationFolder: false
        - template: params_${{ parameters.env }}.yml
          parameters:
            linuxFxVersion: 'DOTNETCORE|6.0'
            env: ${{ parameters.env }}
            appname: 'myapp'
        - task: AzureCLI@2
          displayName: 'Deploy Infrastructure'
          inputs:
            azureSubscription: ${{ parameters.subscription }}
            scriptType: 'pscore'
            scriptLocation: inlineScript
            inlineScript: |
              az group create --name $(resourceGroupName) --location ${{ parameters.location }}
              az deployment group create --name "infrastructure-deploy" --resource-group $(resourceGroupName) --template-file $(templateFileName) --parameters $(templateParametersFileName)

The parameters template will create the parameters JSON file, each file is very similar but the main difference is the sku being used, dev has ‘F1’, qa has ‘B1’ and prod has ‘S1’. This is what the prod parameters file looks like:

parameters:
- name: sku
  type: string
  default: 'S1'
- name: linuxFxVersion
  type: string 
- name: env 
  type: string
- name: appname
  type: string
- name: tags
  type: object
  default:
    environment: 'Production'
    project: 'myproj'

steps:
- template: createParametersFile.yml@templates
  parameters:
    paramsJson: '${{ convertToJson(parameters) }}'
    parameterFilePath: $(templateParametersFileName)

Deploy Application

As with the deployment of IaC most of the configuration is in the base template and so again this pipeline is quite small.

trigger: none
pr: none

pool: 
  vmImage: 'ubuntu-latest'

resources:
  repositories:
    - repository: templates
      type: git
      name: shared-templates
      ref: main

extends:
  template: environments.yml@templates
  parameters:    
    buildSteps:
    - script: | 
        cd src
        dotnet build --configuration Release
      displayName: 'dotnet build Release'
    - task: DotNetCoreCLI@2
      displayName: 'dotnet publish'
      inputs:
        arguments: --configuration Release -o $(Build.ArtifactStagingDirectory)
        command: 'publish'
        publishWebProjects: true
    - publish: $(Build.ArtifactStagingDirectory)
      artifact: code
      displayName: 'Publish Code'
    customReleaseTemplate: deploy-app.yml@self

deploy-app.yml

parameters:
- name: env
  type: string
- name: location
  type: string
- name: locationShort
  type: string
- name: subscription
  type: string

jobs:
- deployment: deploy_${{ parameters.locationShort }}
  displayName: 'Deploy app to ${{ parameters.env }} in ${{ parameters.location }}'
  environment: ${{ parameters.env }}_${{ parameters.locationShort }}
  strategy: 
    runOnce:
      deploy:
        steps:
        - task: AzureWebApp@1
          inputs:
            azureSubscription: '${{ parameters.subscription }}'
            appType: 'webAppLinux'
            appName: 'app-myapp-${{ parameters.env }}-${{ parameters.locationShort }}'
            package: '$(Pipeline.Workspace)/**/*.zip'

Conclusion

The infrastructure and application both run the same steps and so have the same stages output

Resource Groups and Web Apps created by the IaC

Using this type of template provides a very powerful base template and allows the pipelines to focus on building and deploying removing the need to worry too much about the environments that are needed.

I hope showing this has been helpful and who knows it may spark some other ideas and ways of using base templates. Happy deployments 🙂