Azure, Azure Pipelines, Bicep, DevOps, IaC

Automate Maintaining a Private Bicep Module Registry with Azure Pipelines

After using IaC (Infrastructure as Code) on multiple projects it soon becomes apparent that you are performing the same actions over and over and you might want to start sharing those actions or even standardising them. IaC frameworks like Terraform, Pulumi, etc. provide a registry for sharing modules and I wondered if Bicep had something similar, it turns out there is using ACR (Azure Container Registry).

In this post we are going to use an ACR to create a private Bicep registry for sharing modules and a build pipeline to publish modules into the ACR when new modules are added or existing ones are changed.

Objectives:

  • Trigger on changes to Bicep files in the main branch
  • Add modules to the registry only if they do not already exist
  • Publish a new version of each changed module to the registry

Automate Publishing Files

First thing that we need is a Azure Container Registry, if you don’t already have one provisioned you can create one using the Azure CLI e.g.

az group create --name devops-rg --location "UK South"
az acr create --resource-group devops-rg --name devcrdevopsuks --sku Basic

Next we will need a repository that contains the Bicep modules we want to share. This could be an existing repository or a new repository. For the purpose of this, we can create a repository with a simple structure, a single folder called modules that contain the Bicep files we want to share e.g.

Now we have an ACR and a repository we can start going through those objectives.

Trigger on changes to Bicep files in the main branch

Azure Pipeline triggers can be defined to handle this objective by adding the branch name and including the paths e.g.

trigger:
  branches:
    include:
    - main
  paths:
    include:
    - '*.bicep'

Add modules to the registry only if they do not already exist

To achieve this objective we will first need a list of what is in the registry and compare that against the modules in our repository.

Initially there will be no modules in the registry but as ones are added we will want to only return the Bicep modules. It is a good idea to prefix the modules e.g. with ‘bicep/’ when adding them as you may use the ACR for other things not just Bicep modules.

We can use the Azure CLI again to get the list from the registry and filter on the prefix e.g.

az acr repository list --name $(registryName) --query "[?contains(@, '$(modulePrefix)')]" -o tsv

Combine that with some PowerShell to compare the entries, we can then publish the modules not in the registry e.g.

$version = Get-Date -f 'yyyy-MM-dd'
$publishedModules = $(az acr repository list --name $(registryName) --query "[?contains(@, '$(modulePrefix)')]" -o tsv)
Get-ChildItem -Recurse -Path ./*.bicep | Foreach-Object {
    $filename = ($_ | Resolve-Path -Relative) -replace "^./" -replace '\..*'
    # Check if module already exists in the registry
    If (-not $publishedModules.Contains(("$(modulePrefix)" + $filename))) {
        Write-Host "Adding new module $filename with version $version"
        az bicep publish --file $_ --target br:$(registryName).azurecr.io/bicep/${filename}:${version}
    }
}

Note: For the version using the date provides Bicep/ARM style version numbering e.g. 2022-01-23

Publish a new version of each changed module to the registry

For this objective we need to get the list of changed files from the repository. We can use the Git command diff-tree to provide a list of changes since the last commit e.g.

git diff-tree --no-commit-id --name-only --diff-filter=ad -r $(Build.SourceVersion)

This command shows the name of the files changed without the commit id and using lowercase a and d filters instruct this to not include Add or Delete changes (for further information on diff-tree see the docs).

Following this we need to filter to just Bicep file changes and then publish those changes like before e.g.

git diff-tree --no-commit-id --name-only --diff-filter=ad -r $(Build.SourceVersion) | Where-Object {$_.EndsWith('.bicep')} | Foreach-Object {
    $moduleName = ($_ | Resolve-Path -Relative) -replace "^./" -replace '\..*'
    Write-Host "Updating module $moduleName with version $version"
    az bicep publish --file $_ --target br:$(registryName).azurecr.io/$(modulePrefix)${moduleName}:${version}
}

Now we have the full PowerShell script we can add that to the pipeline using the AzureCLI task, include an install step for Bicep and then the complete pipeline looks like this:

trigger:
  branches:
    include:
    - main
  paths:
    include:
    - '*.bicep'

pr: none

variables:
  isMain: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')]
  modulePrefix: 'bicep/'

jobs:
- job: modules
  displayName: 'Publish Bicep Modules'
  condition: eq(variables.isMain, 'true')
  pool:
    vmImage: ubuntu-latest
  steps:
  - task: AzureCLI@2
    displayName: 'Publish/Update Modules to Registry'
    inputs:
      azureSubscription: $(azureSubscription)
      scriptType: 'pscore'
      scriptLocation: inlineScript
      inlineScript: |
        az bicep install
        $version = Get-Date -f 'yyyy-MM-dd'
        $publishedModules = $(az acr repository list --name $(registryName) --query "[?contains(@, '$(modulePrefix)')]" -o tsv)
        Get-ChildItem -Recurse -Path ./*.bicep | Foreach-Object {
          $filename = ($_ | Resolve-Path -Relative) -replace "^./" -replace '\..*'
          # Check if module already exists in the registry
          If (-not $publishedModules.Contains(("$(modulePrefix)" + $filename))) {
            Write-Host "Adding new module $filename with version $version"
            az bicep publish --file $_ --target br:$(registryName).azurecr.io/bicep/${filename}:${version}
          }
        }

        git diff-tree --no-commit-id --name-only --diff-filter=ad -r $(Build.SourceVersion) | Where-Object {$_.EndsWith('.bicep')} | Foreach-Object {
          $moduleName = ($_ | Resolve-Path -Relative) -replace "^./" -replace '\..*'
          Write-Host "Updating module $moduleName with version $version"
          az bicep publish --file $_ --target br:$(registryName).azurecr.io/$(modulePrefix)${moduleName}:${version}
        }

Consume Registry Modules

Now we have the shared modules in the registry, how do we use them? As it turns out it’s quite simple as shown in the Microsoft docs e.g.

module storageModule 'br:devcrdevopsuks.azurecr.io/bicep/modules/storage:2022-01-23' = {

Using the registry name and the module path can make this quite long and a lot to type in every time. We can however use a Bicep config file to create an alias and include the registry and module path (see the Microsoft docs for more detail) e.g.

{
  "moduleAliases": {
    "br": {
      "DevOps": {
        "registry": "devcrdevopsuks.azurecr.io",
        "modulePath": "bicep/modules"
      }
    }
  }
}

Now the name is more concise e.g.

module storageModule 'br/DevOps:storage:2022-01-23' = {

Conclusion

I have to say I like the option of using an ACR as a Bicep registry and by automating the maintenance of adding/updating the modules it makes sharing changes very easy.

The only thing that bothered me was that (at the time of writing) Visual Studio Code does not provide intellisense on which modules are available in the registry. Hopefully this will change in the future but in the meantime this handy PowerShell script will output the information about the registry modules and available versions

$items = @()
$(az acr repository list --name devcrdevopsuks --query "[?contains(@, 'bicep/')]" -o tsv) | ForEach-Object {
    $items += [PSCustomObject]@{
        Moddule = $_
        Tags = $(az acr repository show-tags --name devcrdevopsuks --repository $_ -o tsv)
        }    
}
Write-Output $items
Azure Pipelines

MonoRepos with Azure Pipelines

Talking with a friend of mine a few weeks ago, we discussed monorepos and how to support them in Azure Pipelines using either GitHub or Azure repos. My friend had multiple mirco services in a monorepo in GitHub and wanted to avoid checking out the whole repo each time to build only a part of it.

After lots of research and reading I came across a git command sparse-checkout, this command has been in git since 2.25 and is marked as experimental.

At the time of writing Azure Pipelines ubuntu-latest and windows-latest showed git 2.33.1 as the git version in use on the Microsoft Build agents.

I went back to my friend to tell him what I had found and he had come to the same conclusion and we decided to share this in case anyone else was wondering the same.

So how do we use this command in an Azure Pipeline. First thing to do is stop Azure Pipelines checking out the code automatically using the checkout step with a value of none.

- checkout: none

Then we need to manually setup connecting to the git repo, we can do this in a script step performing the same steps that happen under the checkout step and include the sparse-checkout configuration at the same time.

- script: |
    git init
    git sparse-checkout init --cone
    git sparse-checkout add $(folders)
    git remote add origin $(Build.Repository.Uri)
    git config core.sparsecheckout true
    git config gc.auto 0
    git config --get-all http.$(Build.Repository.Uri).extraheader
    git config --get-all http.proxy
    git config http.version HTTP/1.1

The sparse-checkout add uses a space separated list of folder names for the ones you wish to checkout. Azure Pipelines has some predefined build variables which are really handy for getting the URI, etc.

Now that’s setup we can perform a fetch and a checkout to grab the source code and only get the files we need based on the sparse-checkout configuration.

git fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules --verbose --depth=1 origin
git checkout --progress $(Build.SourceBranchName)

Another handy predefined variable is used to get the branch name.

So, the full pipeline to just perform the checkout part of a monorepo build looks like this:

trigger:
  - main

pool:
  vmImage: ubuntu-latest # or windows-latest

variables:
  folders: 'folder1 folder2'

steps:
- checkout: none
- script: |
    git init
    git sparse-checkout init --cone
    git sparse-checkout add $(folders)
    git remote add origin $(Build.Repository.Uri)
    git config core.sparsecheckout true
    git config gc.auto 0
    git config --get-all http.$(Build.Repository.Uri).extraheader
    git config --get-all http.proxy
    git config http.version HTTP/1.1
    git fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules --verbose --depth=1 origin
    git checkout --progress $(Build.SourceBranchName)
  displayName: 'Clone Partial git repo'

The configuration core.sparseCheckoutCone allows a more restrictive pattern set to be added, for more information see the git documentation.

This all worked fine using a GitHub repository that is connected via the GitHub Azure Pipelines App.

Note: GitHub Apps are the officially recommended way to connect to GitHub (see the docs)

It is a real shame that this just doesn’t work as is for Azure Pipelines but fortunately only a minor change is needed in order for it to work :). You just need to add a git config line or change the git fetch command to include a token and that token can be accessed by a special predefined variable System.AccessToken (see the Microsoft docs for more information).

git config addition:

git config http.$(Build.Repository.Uri).extraheader "AUTHORIZATION: bearer $(System.AccessToken)"

update to the git fetch:

git -c http.extraheader="AUTHORIZATION: bearer $(System.AccessToken)" fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules --depth=1 origin

This was ran using both Microsoft ubuntu and windows agents and it was successful in checking out only the folders specified in the folder list as well as any root files using GitHub and Azure repos.

It has been a lot of fun finding out about the sparse-checkout feature of git and building a pipeline to take advantage of it with GitHub and Azure Repos. Hopefully others will find it useful too and the future of this feature in git is one to watch.

I hope sharing this helps with building monorepos and highlighting the sparse-checkout git feature.