Category: Azure Pipelines

Azure Pipelines, IaC

Azure Pipelines – Deploy AKS with Bicep

Posted on by Mark Johnson

In this post we are going to look at deploying an AKS cluster using Azure Pipelines YAML and Bicep.

If you are new to AKS then take a look at the video series AKS Zero to Hero from Richard Hooper (aka PixelRobots) and Gregor Suttie as well as the learning path from Brendan Burns.

If you are new to Pipelines and Bicep then checkout this Microsoft Learn course to give an introduction.

So, on to creating the AKS cluster using Bicep.

The resources we are going to deploy are:

  • Virtual Network
  • Log Analytics Workspace
  • AKS Cluster
  • Container Registry

We are also going to add Azure AD groups to lockdown the cluster administration and connect the container registry to allow AKS to pull containers from the registry.

Bicep

So let’s start with creating a module for the Virtual network, we need a name for the network and subnet as well as some address prefixes and tags.

@description('The virtual network name')
param vnetName string
@description('The name of the subnet')
param subnetName string
@description('The virtual network address prefixes')
param vnetAddressPrefixes array
@description('The subnet address prefix')
param subnetAddressPrefix string
@description('Tags for the resources')
param tags object

resource vnet 'Microsoft.Network/virtualNetworks@2019-11-01' = {
  name: vnetName
  location: resourceGroup().location
  properties: {
    addressSpace: {
      addressPrefixes: vnetAddressPrefixes
    }
    subnets: [
      {
        name: subnetName
        properties: {
          addressPrefix: subnetAddressPrefix
        }
      }      
    ]
  }
  tags: tags
}

output subnetId string = '${vnet.id}/subnets/${subnetName}'

The next module then is the AKS cluster itself, there is a lot of settings that you might want to control but I’ve added defaults for some of them. This module also includes creation of an Log Analytics workspace and the renaming of the AKS resource group that normally is prefixed with MC_ to something inline with the used naming convention.

@description('The environment prefix of the Managed Cluster resource e.g. dev, prod, etc.')
param prefix string
@description('The name of the Managed Cluster resource')
param clusterName string
@description('Resource location')
param location string = resourceGroup().location
@description('Kubernetes version to use')
param kubernetesVersion string = '1.20.7'
@description('The VM Size to use for each node')
param nodeVmSize string
@minValue(1)
@maxValue(50)
@description('The number of nodes for the cluster.')
param nodeCount int
@maxValue(100)
@description('Max number of nodes to scale up to')
param maxNodeCount int
@description('The node pool name')
param nodePoolName string = 'linux1'
@minValue(0)
@maxValue(1023)
@description('Disk size (in GB) to provision for each of the agent pool nodes. This value ranges from 0 to 1023. Specifying 0 will apply the default disk size for that agentVMSize')
param osDiskSizeGB int
param nodeAdminUsername string
@description('Availability zones to use for the cluster nodes')
param availabilityZones array = [
  '1'
  '2'
  '3'
]
@description('Allow the cluster to auto scale to the max node count')
param enableAutoScaling bool = true
@description('SSH RSA public key for all the nodes')
@secure()
param sshPublicKey string
@description('Tags for the resources')
param tags object
@description('Log Analytics Workspace Tier')
@allowed([
  'Free'
  'Standalone'
  'PerNode'
  'PerGB2018'
  'Premium'
])
param workspaceTier string
@allowed([
  'azure'  
])
@description('Network plugin used for building Kubernetes network')
param networkPlugin string = 'azure'
@description('Subnet id to use for the cluster')
param subnetId string
@description('Cluster services IP range')
param serviceCidr string = '10.0.0.0/16'
@description('DNS Service IP address')
param dnsServiceIP string = '10.0.0.10'
@description('Docker Bridge IP range')
param dockerBridgeCidr string = '172.17.0.1/16'
@description('An array of AAD group object ids for administration')
param adminGroupObjectIDs array = []

resource logAnalyticsWorkspace 'Microsoft.OperationalInsights/workspaces@2020-10-01' = {
  name: '${prefix}-oms-${clusterName}-${resourceGroup().location}'
  location: location
  properties: {
    sku: {
      name: workspaceTier
    }
  }
  tags: tags
}

resource aksCluster 'Microsoft.ContainerService/managedClusters@2021-03-01' = {
  name: '${prefix}-aks-${clusterName}-${location}'
  location: location
  identity: {
    type: 'SystemAssigned'
  }
  tags: tags  
  properties: {
    nodeResourceGroup: 'rg-${prefix}-aks-nodes-${clusterName}-${location}'
    kubernetesVersion: kubernetesVersion
    dnsPrefix: '${clusterName}-dns'
    enableRBAC: true    
    agentPoolProfiles: [
      {        
        name: nodePoolName
        osDiskSizeGB: osDiskSizeGB
        osDiskType: 'Ephemeral'        
        count: nodeCount
        enableAutoScaling: enableAutoScaling
        minCount: nodeCount
        maxCount: maxNodeCount
        vmSize: nodeVmSize        
        osType: 'Linux'
        type: 'VirtualMachineScaleSets'
        mode: 'System'
        availabilityZones: availabilityZones
        enableEncryptionAtHost: true
        vnetSubnetID: subnetId
      }
    ]
    networkProfile: {      
      loadBalancerSku: 'standard'
      networkPlugin: networkPlugin
      serviceCidr: serviceCidr
      dnsServiceIP: dnsServiceIP
      dockerBridgeCidr: dockerBridgeCidr
    }
    aadProfile: !empty(adminGroupObjectIDs) ? {
      managed: true
      adminGroupObjectIDs: adminGroupObjectIDs
    } : null
    addonProfiles: {
      azurepolicy: {
        enabled: false
      }
      omsAgent: {
        enabled: true
        config: {
          logAnalyticsWorkspaceResourceID: logAnalyticsWorkspace.id
        }
      }   
    }
    linuxProfile: {      
      adminUsername: nodeAdminUsername
      ssh: {
        publicKeys: [
          {
            keyData: sshPublicKey
          }
        ]
      }      
    }
  }

  dependsOn: [
    logAnalyticsWorkspace    
  ]
}

output controlPlaneFQDN string = reference('${prefix}-aks-${clusterName}-${location}').fqdn
output clusterPrincipalID string = aksCluster.properties.identityProfile.kubeletidentity.objectId

The final module is building an Azure Container Registry and assigning the ACR Pull role for the cluster

@description('The name of the container registry')
param registryName string
@description('The principal ID of the AKS cluster')
param aksPrincipalId string
@description('Tags for the resources')
param tags object

@allowed([
  'b24988ac-6180-42a0-ab88-20f7382dd24c' // Contributor
  'acdd72a7-3385-48ef-bd42-f606fba81ae7' // Reader
])
param roleAcrPull string = 'b24988ac-6180-42a0-ab88-20f7382dd24c'

resource containerRegistry 'Microsoft.ContainerRegistry/registries@2019-05-01' = {
  name: registryName
  location: resourceGroup().location
  sku: {
    name: 'Standard'
  }
  properties: {
    adminUserEnabled: true
  }
  tags: tags
}

resource assignAcrPullToAks 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
  name: guid(resourceGroup().id, registryName, aksPrincipalId, 'AssignAcrPullToAks')
  scope: containerRegistry
  properties: {
    description: 'Assign AcrPull role to AKS'
    principalId: aksPrincipalId
    principalType: 'ServicePrincipal'
    roleDefinitionId: '/subscriptions/${subscription().subscriptionId}/providers/Microsoft.Authorization/roleDefinitions/${roleAcrPull}'
  }
}

output name string = containerRegistry.name

So now we have the all the modules lets setup the main bicep file to put it all together

@description('Naming prefix for the resources e.g. dev, test, prod')
param prefix string
@description('The public SSH key')
@secure()
param publicsshKey string
@description('The name of the cluster')
param clusterName string
@description('The location of the resources')
param location string = resourceGroup().location
@description('The admin username for the nodes in the cluster')
param nodeAdminUsername string
@description('An array of AAD group object ids to give administrative access.')
param adminGroupObjectIDs array = []
@description('The VM size to use in the cluster')
param nodeVmSize string
@minValue(1)
@maxValue(50)
@description('The number of nodes for the cluster.')
param nodeCount int = 1
@maxValue(100)
@description('Max number of nodes to scale up to')
param maxNodeCount int = 3
@description('Disk size (in GB) to provision for each of the agent pool nodes. This value ranges from 0 to 1023. Specifying 0 will apply the default disk size for that agentVMSize')
param osDiskSizeGB int
@description('Log Analytics Workspace Tier')
@allowed([
  'Free'
  'Standalone'
  'PerNode'
  'PerGB2018'
  'Premium'
])
param workspaceTier string
@description('The virtual network address prefixes')
param vnetAddressPrefixes array
@description('The subnet address prefix')
param subnetAddressPrefix string
@description('Tags for the resources')
param tags object

module vnet 'vnet.bicep' = {
  name: 'vnetDeploy'
  params: {
    vnetName: '${prefix}-vnet-${clusterName}-${location}'
    subnetName: '${prefix}-snet-${clusterName}-${location}'
    vnetAddressPrefixes: vnetAddressPrefixes
    subnetAddressPrefix: subnetAddressPrefix
    tags: tags
  }
}

module aks 'aks.bicep' = {
  name: 'aksDeploy'
  params: {
    prefix: prefix
    clusterName: clusterName    
    subnetId: vnet.outputs.subnetId
    nodeAdminUsername: nodeAdminUsername
    adminGroupObjectIDs: adminGroupObjectIDs
    nodeVmSize: nodeVmSize
    nodeCount: nodeCount
    maxNodeCount: maxNodeCount
    osDiskSizeGB: osDiskSizeGB
    sshPublicKey: publicsshKey
    workspaceTier: workspaceTier
    tags: tags
  }

  dependsOn: [
    vnet
  ]
}

module registry 'registry.bicep' = {
  name: 'registryDeploy'
  params: {
    registryName: 'acr${clusterName}'
    aksPrincipalId: aks.outputs.clusterPrincipalID
    tags: tags
  }

  dependsOn: [
    aks
  ]
}

As with ARM templates you can use a Json File to configure the parameters in bicep and so I’ve added one for this

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "tags": {
            "value": {
                "project": "mjdemo",
                "resource": "AKS"
            }
        },
        "prefix": {
            "value": "dev"
        },
        "clusterName": {
            "value": "mjdemo"
        },
        "nodeVmSize": {
            "value": "Standard_D2s_V3"
        },
        "osDiskSizeGB": {
            "value": 50
        },
        "nodeCount": {
            "value": 1
        },
        "maxNodeCount": {
            "value": 3
        },
        "nodeAdminUsername": {
            "value": "aksAdminUser"
        },
        "adminGroupObjectIDs": {
            "value": []
        },
        "publicsshKey": {
            "value": ""
        },
        "workspaceTier": {
            "value": "PerGB2018"
        },
        "vnetAddressPrefixes": {
            "value": []
        },
        "subnetAddressPrefix": {
            "value": ""
        }
    }
}

Pipeline

Now we have all the Bicep files and a parameters file, we can create an Azure Pipeline but first we are going to need an SSH Key and upload it to Azure Pipelines, one way to generate an SSH key is to use the ssh-keygen command in Bash (I used Ubuntu in WSL) e.g.

ssh-keygen -q -t rsa -b 4096 -N '' -f aksKey

This will generate a private and public key pair, you can then upload the public key file e.g. aksKey.pub to Secure Files in Azure DevOps Pipelines (Pipelines->Library->Secure files)

We are going to add Azure AD Groups in this deployment and will need to assign the role ‘Azure Kubernetes Service Cluster User Role’ to each group, the Microsoft Docs detail how to do this.

Now we have the SSH key uploaded we can configure the parameters we want to set for our AKS cluster and network.

trigger: none
pr: none

pool:
  vmImage: ubuntu-latest

parameters:
  - name: azureSubscription
    type: string
    default: 'Sandbox'
  - name: location
    displayName: 'Resource Location'
    type: string
    default: 'uksouth'
  - name: prefix
    displayName: 'Environment Prefix'
    type: string
    default: 'prod'
  - name: clusterName
    displayName: 'Name of the AKS Cluster'
    type: string
    default: 'demo'
  - name: nodeVmSize
    displayName: 'VM Size for the Nodes'
    type: string
    default: 'Standard_D2s_V3'
    values:
      - 'Standard_D2s_V3'
      - 'Standard_DS2_v2'
      - 'Standard_D4s_V3'
      - 'Standard_DS3_v2'
      - 'Standard_DS4_v2'
      - 'Standard_D8s_v3'
  - name: osDiskSizeGB
    displayName: 'Size of OS disk (0 means use vm size)'
    type: number
    default: 50
  - name: nodeCount
    displayName: 'The number of nodes'
    type: number
    default: 3
  - name: maxNodeCount
    displayName: 'Max node to scale out to'
    type: number
    default: 10
  - name: workspaceTier
    displayName: Log Analytics Workspace Tier
    type: string
    default: 'PerGB2018'
    values:
      - 'Free'
      - 'Standalone'
      - 'PerNode'
      - 'PerGB2018'
      - 'Premium'
  - name: tags
    displayName: 'Tags'
    type: object
    default:
     Environment: "prod"
     Resource: "AKS"
     Project: "Demo"
  - name: nodeAdminUsername
    displayName: 'Admin username for the nodes'
    type: string
    default: 'adminUserName'
  - name: vnetAddressPrefixes
    displayName: 'Virtual Network Address Prefixes'
    type: object
    default: 
      - '10.240.0.0/16'
  - name: subnetAddressPrefix
    displayName: 'Subnet Address Prefix'
    type: string
    default: '10.240.0.0/20'
  - name: adGroupNames
    type: object
    default: 
      - 'demo-group'

variables:
  resourceGroupName: 'rg-${{ parameters.prefix }}-${{ parameters.clusterName }}-${{ parameters.location }}'

With the parameters set the next part is to build up the steps, starting with downloading the SSH key from the secure files using the DownloadSecureFile task

steps:
- task: DownloadSecureFile@1
  displayName: 'Download Public SSH Key'
  name: SSHfile
  inputs:
    secureFile: 'aksKey.pub'
- bash: |
    value=`cat $(SSHfile.secureFilePath)`
    echo '##vso[task.setvariable variable=publicsshKey;issecret=true]'$value
  displayName: Obtain SSH key value

Next we can get the object IDs for the groups

- task: AzureCLI@2
  displayName: 'Get AD Group Object Ids'
  inputs:
    azureSubscription: ${{ parameters.azureSubscription }}
    scriptType: pscore
    scriptLocation: inlineScript
    inlineScript: |    
      $objectIds = '${{ join(',',parameters.adGroupNames) }}'.Split(',') | ForEach { 
        "$(az ad group list --query "[?displayName == '$_'].{objectId:objectId}" -o tsv)" 
      }

      $output = ConvertTo-Json -Compress @($objectIds)
      Write-Host '##vso[task.setvariable variable=groupIds]'$output

This next section is taking those parameters and turning them into variables to then substitute the values in parameters json file

- template: objectparameters.yml
  parameters:
    tags: ${{ parameters.tags }}
    vnetAddressPrefixes: ${{ parameters.vnetAddressPrefixes }}
- template: parameters.yml
  parameters:
    prefix: ${{ parameters.prefix }}
    clusterName: ${{ parameters.clusterName }}
    nodeVmSize: ${{ parameters.nodeVmSize }}
    osDiskSizeGB: ${{ parameters.osDiskSizeGB }}
    nodeCount: ${{ parameters.nodeCount }}
    maxNodeCount: ${{ parameters.maxNodeCount }}
    nodeAdminUsername: ${{ parameters.nodeAdminUsername }}
    publicsshKey: $(publicsshKey)
    workspaceTier: ${{ parameters.workspaceTier }}    
    subnetAddressPrefix: ${{ parameters.subnetAddressPrefix }}
    adminGroupObjectIDs: $(groupIds)
- task: FileTransform@2
  displayName: "Transform Parameters"
  inputs:
    folderPath: '$(System.DefaultWorkingDirectory)'
    xmlTransformationRules: ''
    jsonTargetFiles: 'deploy.parameters.json'

If you need to debug the transform then you can add another step to output the file contents, I find this a useful technique to make sure the transform worked as expected

- bash: |
    cat deploy.parameters.json
  displayName: "Debug show parameters file"

If we put all that together then the final pipeline looks like this:

trigger: none
pr: none

pool:
  vmImage: ubuntu-latest

parameters:
  - name: azureSubscription
    type: string
    default: 'Sandbox'
  - name: location
    displayName: 'Resource Location'
    type: string
    default: 'uksouth'
  - name: prefix
    displayName: 'Environment Prefix'
    type: string
    default: 'prod'
  - name: clusterName
    displayName: 'Name of the AKS Cluster'
    type: string
    default: 'demo'
  - name: nodeVmSize
    displayName: 'VM Size for the Nodes'
    type: string
    default: 'Standard_D2s_V3'
    values:
      - 'Standard_D2s_V3'
      - 'Standard_DS2_v2'
      - 'Standard_D4s_V3'
      - 'Standard_DS3_v2'
      - 'Standard_DS4_v2'
      - 'Standard_D8s_v3'
  - name: osDiskSizeGB
    displayName: 'Size of OS disk (0 means use vm cache size)'
    type: number
    default: 50
  - name: nodeCount
    displayName: 'The number of nodes'
    type: number
    default: 3
  - name: maxNodeCount
    displayName: 'Max node to scale out to'
    type: number
    default: 10
  - name: workspaceTier
    displayName: Log Analytics Workspace Tier
    type: string
    default: 'PerGB2018'
    values:
      - 'Free'
      - 'Standalone'
      - 'PerNode'
      - 'PerGB2018'
      - 'Premium'
  - name: tags
    displayName: 'Tags'
    type: object
    default:
     Environment: "prod"
     Resource: "AKS"
     Project: "Demo"
  - name: nodeAdminUsername
    displayName: 'Admin username for the nodes'
    type: string
    default: 'adminUserName'
  - name: vnetAddressPrefixes
    displayName: 'Virtual Network Address Prefixes'
    type: object
    default: 
      - '10.240.0.0/16'
  - name: subnetAddressPrefix
    displayName: 'Subnet Address Prefix'
    type: string
    default: '10.240.0.0/20'
  - name: adGroupNames
    type: object
    default: 
      - 'demo-group'

variables:
  resourceGroupName: 'rg-${{ parameters.prefix }}-${{ parameters.clusterName }}-${{ parameters.location }}'

steps:
- task: DownloadSecureFile@1
  displayName: 'Download Public SSH Key'
  name: SSHfile
  inputs:
    secureFile: 'aksKey.pub'
- bash: |
    value=`cat $(SSHfile.secureFilePath)`
    echo '##vso[task.setvariable variable=publicsshKey;issecret=true]'$value
  displayName: Obtain SSH key value  
- task: AzureCLI@2
  displayName: 'Get AD Group Object Ids'
  inputs:
    azureSubscription: ${{ parameters.azureSubscription }}
    scriptType: pscore
    scriptLocation: inlineScript
    inlineScript: |    
      $objectIds = '${{ join(',',parameters.adGroupNames) }}'.Split(',') | ForEach { 
        "$(az ad group list --query "[?displayName == '$_'].{objectId:objectId}" -o tsv)" 
      }

      $output = ConvertTo-Json -Compress @($objectIds)
      Write-Host '##vso[task.setvariable variable=groupIds]'$output
- template: objectparameters.yml
  parameters:
    tags: ${{ parameters.tags }}
    vnetAddressPrefixes: ${{ parameters.vnetAddressPrefixes }}
- template: parameters.yml
  parameters:
    prefix: ${{ parameters.prefix }}
    clusterName: ${{ parameters.clusterName }}
    nodeVmSize: ${{ parameters.nodeVmSize }}
    osDiskSizeGB: ${{ parameters.osDiskSizeGB }}
    nodeCount: ${{ parameters.nodeCount }}
    maxNodeCount: ${{ parameters.maxNodeCount }}
    nodeAdminUsername: ${{ parameters.nodeAdminUsername }}
    publicsshKey: $(publicsshKey)
    workspaceTier: ${{ parameters.workspaceTier }}    
    subnetAddressPrefix: ${{ parameters.subnetAddressPrefix }}
    adminGroupObjectIDs: $(groupIds)
- task: FileTransform@2
  displayName: "Transform Parameters"
  inputs:
    folderPath: '$(System.DefaultWorkingDirectory)'
    xmlTransformationRules: ''
    jsonTargetFiles: 'deploy.parameters.json'
- task: AzureCLI@2
  displayName: 'Deploy AKS Cluster'
  inputs:
    azureSubscription: ${{ parameters.azureSubscription }}
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      az group create --name "$(resourceGroupName)" --location ${{ parameters.location }} 
      az deployment group create --name "${{ parameters.clusterName }}-deploy" --resource-group "$(resourceGroupName)" --template-file deploy.bicep --parameters deploy.parameters.json

The template file objectparameters.yml looks like this:

parameters: 
  - name: tags
    type: object
  - name: vnetAddressPrefixes
    type: object

steps:
- ${{ each item in parameters }}: 
  - bash: |
      value='${{ convertToJson(item.value) }}'
      echo '##vso[task.setvariable variable=parameters.${{ item.key }}.value]'$value
    displayName: "Create Variable ${{ item.key }}"

And the template file parameters.yml looks like this:

parameters: 
  - name: prefix
    type: string
  - name: clusterName
    type: string
  - name: nodeVmSize
    type: string
  - name: osDiskSizeGB
    type: number
  - name: nodeCount
    type: number
  - name: maxNodeCount
    type: number
  - name: nodeAdminUsername
    type: string
  - name: publicsshKey
    type: string
  - name: workspaceTier
    type: string
  - name: subnetAddressPrefix
    type: string
  - name: adminGroupObjectIDs
    type: string

steps:
- ${{ each item in parameters }}:  
    - bash: |
        echo '##vso[task.setvariable variable=parameters.${{ item.key }}.value]${{ item.value }}'
      displayName: "Create Variable ${{ item.key }}"

Now we have an AKS cluster setup we might want to deploy some applications to the cluster. CoderDave has a great video tutorial to do this with Azure Pipelines.

All the files shown above can be found on my GitHub

3
Architecture, Azure Pipelines, Diagrams

Azure Pipelines – Diagrams as Code

Posted on by Mark Johnson

Following on from my previous post on Architecture Diagrams I thought I would share my experiences with another tool, Diagrams.

Diagrams uses the Python language to describe diagrams, Python is not a language I use generally but it was simple enough to learn building diagrams.

The documentation describes how to get started and setup. I am a big fan of containers and so I created a container for using Diagrams.

The following dockerfile will create an environment: 

FROM python:alpine3.13
ENV APK_ADD "bash py3-pip graphviz ttf-freefont"

RUN apk upgrade --update && \
    apk add --no-cache --virtual .pipeline-deps readline linux-pam && \
    apk add --no-cache ${APK_ADD} && \
    # Install Diagrams
    pip --no-cache-dir install --upgrade pip && \
    pip --no-cache-dir install diagrams && \
    # Tidy up
    apk del .pipeline-deps

RUN echo "PS1='\n\[\033[01;35m\][\[\033[0m\]Diagrams\[\033[01;35m\]]\[\033[0m\]\n\[\033[01;35m\][\[\033[0m\]\[\033[01;32m\]\w\[\033[0m\]\[\033[01;35m\]]\[\033[0m\]\n \[\033[01;33m\]->\[\033[0m\] '" >> ~/.bashrc

CMD tail -f /dev/null

To build and run: (I used a windows 10 machine)

docker build -f diagrams.dockerfile -t my-diagrams
docker run -it --entrypoint=/bin/bash --volume $env:USERPROFILE\source\repos:/mycode my-diagrams

Diagram

With my new environment I can use an editor of my choice to create the diagrams, my current go to is Visual Studio Code and there is a extension for Python.

The purpose of using this tool was to draw a diagram of an Azure Tenant and Subscription setup, I needed something that would allow the diagram to be changed quickly as the multiple people were collaborating.

The code below shows a simple example of the diagram being created:

from diagrams import Cluster, Diagram
from diagrams.azure.general import Managementgroups
from diagrams.azure.general import Subscriptions
from diagrams.azure.identity import ActiveDirectory

with Diagram("Azure Tenant Design", show=False, direction="TB"):
    tenant = ActiveDirectory("Tenant AD")  
    topGroup = Managementgroups("Main\r\nManagement Group")
    sandbox = Subscriptions("Sandbox\r\nSubscription")

    with Cluster("Business Units"):
        with Cluster("Unit1"):
          mainGroup = Managementgroups("Unit1\r\nManagement Group")
          topGroup >> mainGroup
          with Cluster("Project1"):
            group = Managementgroups("Project1\r\nManagement Group")
            sub = [Subscriptions("Project1\r\nDev/Test\r\nSubscription"), Subscriptions("Project1\r\nProduction\r\nSubscription")]
            group - sub
            mainGroup >> group

          with Cluster("Project2"):
            group = Managementgroups("Project2\r\nManagement Group")
            sub = [Subscriptions("Project2\r\nDev/Test\r\nSubscription"), Subscriptions("Project2\r\nProduction\r\nSubscription")]
            group - sub
            mainGroup >> group

        with Cluster("Infrastructure"):
          group = Managementgroups("Infrastructure\r\nManagement Group")
          sub = [Subscriptions("Test\r\nSubscription"), Subscriptions("Infrastructure\r\nProduction\r\nSubscription")]
          group - sub
          topGroup >> group

    tenant >> topGroup >> sandbox

The diagram can be created by simply calling python and the name of the file (this code was executed from the container).

The code produces the following diagram:

Azure Pipelines

Now the diagram code is created, it would be good to be able to have a pipeline building the diagram and providing the image. Building the diagrams in an Azure Pipeline would be easier if I could use the container created earlier.

Fortunately I can, Azure Pipelines allows container jobs, but that means the dockerfile needs a few modifications to use it in Azure Pipelines. The Microsoft Docs explain in more detail but for this I need node installed, a special label and some additional packages.

The new dockerfile looks like this:

FROM node:lts-alpine3.13 AS node_base

RUN echo "NODE Version:" && node --version
RUN echo "NPM Version:" && npm --version

FROM python:alpine3.13

ENV NODE_HOME /usr/local/bin/node
COPY --from=node_base ["${NODE_HOME}", "${NODE_HOME}"] 

LABEL maintainer="Tazmainiandevil"
LABEL "com.azure.dev.pipelines.agent.handler.node.path"="${NODE_HOME}" 

ENV APK_ADD "bash sudo shadow py3-pip graphviz ttf-freefont"

RUN apk upgrade --update && \
    apk add --no-cache --virtual .pipeline-deps readline linux-pam && \
    apk add --no-cache ${APK_ADD} && \
    # Install Diagrams
    pip --no-cache-dir install --upgrade pip && \
    pip --no-cache-dir install diagrams && \
    # Tidy up
    apk del .pipeline-deps

RUN echo "PS1='\n\[\033[01;35m\][\[\033[0m\]Diagrams\[\033[01;35m\]]\[\033[0m\]\n\[\033[01;35m\][\[\033[0m\]\[\033[01;32m\]\w\[\033[0m\]\[\033[01;35m\]]\[\033[0m\]\n \[\033[01;33m\]->\[\033[0m\] '" >> ~/.bashrc

CMD tail -f /dev/null

Container Build

Using Azure Pipelines I can build the container and added it to an Azure Container Registry (if you need to know how to setup ACR see my previous post on Configuring ACR)

trigger: 
 branches:
    include:
    - main
 paths:
    include: 
     - diagrams.dockerfile

pr: none

variables:
- group: Azure Connections
- name: dockerFilePath
  value: diagrams.dockerfile
- name: imageRepository
  value: dac/diagrams

pool:
  vmImage: "ubuntu-latest"

steps:
  - task: Docker@2
    displayName: "Build Diagram Image"
    inputs:
      containerRegistry: "$(myContainerRegistry)"
      repository: '$(imageRepository)'
      command: 'buildAndPush'
      Dockerfile: '$(dockerfilePath)'
      tags: |
        $(Build.BuildNumber)
        latest

With the container added to my registry I can use it in a pipeline to create my diagrams.

Image Build

The pipeline needs to create an image as an artifact and only when on the main branch to make sure only the final diagrams are published and not ones in progress.

The YAML below defines the pipeline:

trigger: 
   - main

pr: none

variables:
  isMain: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')]

jobs:
- job: creatediagram
  displayName: Create Diagram  
  pool:
    vmImage: ubuntu-latest
  container:
    image: $(myContainerRegistry)/dac/diagrams:latest
    endpoint: 'My Registry Service Connection'
  variables:
    workspaceFolder: 'TenantDesign'
  steps:
  - script: | 
      python tenant.py
      cp *.png $(Build.ArtifactStagingDirectory)
    displayName: Run Python
  - publish: '$(Build.ArtifactStagingDirectory)'
    displayName: Publish Diagrams
    artifact: $(workspaceFolder)
    condition: eq(variables.isMain, true)

Conclusion

I found using Diagrams simple and the documentation was good to allow picking up what was needed quickly. I will certainly be looking at using it for other diagrams in the future. I like the fact that it is easy to use, open source and supports custom images so you are not limited to the provided icons (Custom Docs).

Diagrams GitHub

Diagrams Docs

I hope that others find this useful and use Diagrams as Code for their projects.

2