Azure Pipelines

Azure Pipelines – Shared Templates and Versioning

Sharing templates seems like a really simple thing to want to do especially when you have similar projects that use the same steps to build or deploy code. In this post I am going to go through how does Azure Pipelines support this and how can we version shared templates.

As a C# developer I build code the same way in multiple projects, I use GitVersion for versioning my code and the DotNetCoreCLI tasks to restore, build and publish my code into and artifact, for example

steps:
- task: gitversion/setup@0
  displayName: 'Install GitVersion'
  inputs:
    versionSpec: '5.x'
- task: gitversion/execute@0
  displayName: 'Determine Version'
  inputs:
    additionalArguments: '/updateprojectfiles'
- task: DotNetCoreCLI@2
  displayName: 'Restore Packages'
  inputs:
    command: 'restore'
    projects: '**/*.csproj'
- task: DotNetCoreCLI@2
  displayName: 'Build Code'
  inputs:
    command: 'build'
    projects: '**/*.csproj'
    arguments: '--no-restore --configuration release'
- task: DotNetCoreCLI@2
  displayName: 'Publish Code'
  inputs:
    command: 'publish'
    arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)/release'
    projects:  'src/**/*.csproj'
    publishWebProjects: false
    modifyOutputPath: false
    zipAfterPublish: true    
- publish: '$(Build.ArtifactStagingDirectory)'
  displayName: 'Publish App Artifact'
  artifact: 'myapp'

There is probably a dozen projects/repositories where this exists now and say I wanted to add a security scan to all my projects that may take some time. A shared template would allow this to be maintained and updated in one place.

So how can we create a shared template that is used across multiple repositories in Azure DevOps?

Firstly we need to create a repository that contains the shared template code, I called my repository ‘shared-templates’, I added a folder called ‘code’ and added the above code into a template called ‘build-code.yml’ and added a parameter to pass in the name of the artifact

parameters:
- name: artifactName
  type: string

Now I have added the first version of the code I am going to tag the repo with a version

The shared templates repository is now ready for use, let’s update a pipeline to now use this. The current pipeline looks something like this:

trigger: none
pool:
  vmImage: 'ubuntu-latest'

steps:
- task: gitversion/setup@0
  displayName: Install GitVersion
  inputs:
    versionSpec: '5.x'
- task: gitversion/execute@0
  displayName: Determine Version
  inputs:
    additionalArguments: '/updateprojectfiles'
- task: DotNetCoreCLI@2
  displayName: Restore Packages
  inputs:
    command: 'restore'
    projects: '**/*.csproj'
- task: DotNetCoreCLI@2
  displayName: Build Code
  inputs:
    command: build
    projects: '**/*.csproj'
    arguments: '--no-restore --configuration release'
- task: DotNetCoreCLI@2
  displayName: Publish Code
  inputs:
    command: publish
    arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)/release'
    projects:  'src/**/*.csproj'
    publishWebProjects: false
    modifyOutputPath: false
    zipAfterPublish: true    
- publish: '$(Build.ArtifactStagingDirectory)'
  displayName: 'Publish App Artifact'
  artifact: 'myapp'

In order to use the external repository we need to add a resources section to the pipeline to look for the shared-templates repository and which branch to look at

resources:
  repositories:
    - repository: templates
      type: git
      name: shared-templates
      ref: main

The next thing is to replace all the previous steps with a template step and then tag the repository with the version from GitVersion

trigger: none
pool:
  vmImage: 'ubuntu-latest'

resources:
  repositories:
    - repository: templates
      type: git
      name: shared-templates
      ref: main

steps:
- template: 'code/build-code.yml@templates'
  parameters:
    artifactName: 'myapp'
- bash: |
      git tag $(SemVer)
      echo 'Created tag'
      git push --tags
     echo 'tags pushed'
  displayName: 'Creating Tag'  

This will get the latest version of the shared-templates when it is ran, if we wanted to get a specific version we can reference a tag like one we created earlier. The above would then become

trigger: none
pool:
  vmImage: 'ubuntu-latest'

resources:
  repositories:
    - repository: templates
      type: git
      name: shared-templates
      ref: ref/tags/v1.0

steps:
- template: 'code/build-code.yml@templates'
  parameters:
    artifactName: 'myapp'
- bash: |
      git tag $(SemVer)
      echo 'Created tag'
      git push --tags
     echo 'tags pushed'
  displayName: 'Creating Tag'  

And that’s it, a way of sharing templates that can be used in multiple projects and can also be versioned to mitigate breaking changes in shared code.

I hope this has been useful, it’s certainly a technique I will be using more of in my Azure Pipelines.

Architecture, Azure Pipelines, Diagrams

Azure Pipelines – Diagrams as Code

Following on from my previous post on Architecture Diagrams I thought I would share my experiences with another tool, Diagrams.

Diagrams uses the Python language to describe diagrams, Python is not a language I use generally but it was simple enough to learn building diagrams.

The documentation describes how to get started and setup. I am a big fan of containers and so I created a container for using Diagrams.

The following dockerfile will create an environment:

FROM python:alpine3.13
ENV APK_ADD "bash py3-pip graphviz ttf-freefont"

RUN apk upgrade --update && \
    apk add --no-cache --virtual .pipeline-deps readline linux-pam && \
    apk add --no-cache ${APK_ADD} && \
    # Install Diagrams
    pip --no-cache-dir install --upgrade pip && \
    pip --no-cache-dir install diagrams && \
    # Tidy up
    apk del .pipeline-deps

RUN echo "PS1='\n\[\033[01;35m\][\[\033[0m\]Diagrams\[\033[01;35m\]]\[\033[0m\]\n\[\033[01;35m\][\[\033[0m\]\[\033[01;32m\]\w\[\033[0m\]\[\033[01;35m\]]\[\033[0m\]\n \[\033[01;33m\]->\[\033[0m\] '" >> ~/.bashrc

CMD tail -f /dev/null

To build and run: (I used a windows 10 machine)

docker build -f diagrams.dockerfile -t my-diagrams
docker run -it --entrypoint=/bin/bash --volume $env:USERPROFILE\source\repos:/mycode my-diagrams

Diagram

With my new environment I can use an editor of my choice to create the diagrams, my current go to is Visual Studio Code and there is a extension for Python.

The purpose of using this tool was to draw a diagram of an Azure Tenant and Subscription setup, I needed something that would allow the diagram to be changed quickly as the multiple people were collaborating.

The code below shows a simple example of the diagram being created:

from diagrams import Cluster, Diagram
from diagrams.azure.general import Managementgroups
from diagrams.azure.general import Subscriptions
from diagrams.azure.identity import ActiveDirectory

with Diagram("Azure Tenant Design", show=False, direction="TB"):
    tenant = ActiveDirectory("Tenant AD")  
    topGroup = Managementgroups("Main\r\nManagement Group")
    sandbox = Subscriptions("Sandbox\r\nSubscription")

    with Cluster("Business Units"):
        with Cluster("Unit1"):
          mainGroup = Managementgroups("Unit1\r\nManagement Group")
          topGroup >> mainGroup
          with Cluster("Project1"):
            group = Managementgroups("Project1\r\nManagement Group")
            sub = [Subscriptions("Project1\r\nDev/Test\r\nSubscription"), Subscriptions("Project1\r\nProduction\r\nSubscription")]
            group - sub
            mainGroup >> group

          with Cluster("Project2"):
            group = Managementgroups("Project2\r\nManagement Group")
            sub = [Subscriptions("Project2\r\nDev/Test\r\nSubscription"), Subscriptions("Project2\r\nProduction\r\nSubscription")]
            group - sub
            mainGroup >> group

        with Cluster("Infrastructure"):
          group = Managementgroups("Infrastructure\r\nManagement Group")
          sub = [Subscriptions("Test\r\nSubscription"), Subscriptions("Infrastructure\r\nProduction\r\nSubscription")]
          group - sub
          topGroup >> group

    tenant >> topGroup >> sandbox

The diagram can be created by simply calling python and the name of the file (this code was executed from the container).

The code produces the following diagram:

Azure Pipelines

Now the diagram code is created, it would be good to be able to have a pipeline building the diagram and providing the image. Building the diagrams in an Azure Pipeline would be easier if I could use the container created earlier.

Fortunately I can, Azure Pipelines allows container jobs, but that means the dockerfile needs a few modifications to use it in Azure Pipelines. The Microsoft Docs explain in more detail but for this I need node installed, a special label and some additional packages.

The new dockerfile looks like this:

FROM node:lts-alpine3.13 AS node_base

RUN echo "NODE Version:" && node --version
RUN echo "NPM Version:" && npm --version

FROM python:alpine3.13

ENV NODE_HOME /usr/local/bin/node
COPY --from=node_base ["${NODE_HOME}", "${NODE_HOME}"] 

LABEL maintainer="Tazmainiandevil"
LABEL "com.azure.dev.pipelines.agent.handler.node.path"="${NODE_HOME}" 

ENV APK_ADD "bash sudo shadow py3-pip graphviz ttf-freefont"

RUN apk upgrade --update && \
    apk add --no-cache --virtual .pipeline-deps readline linux-pam && \
    apk add --no-cache ${APK_ADD} && \
    # Install Diagrams
    pip --no-cache-dir install --upgrade pip && \
    pip --no-cache-dir install diagrams && \
    # Tidy up
    apk del .pipeline-deps

RUN echo "PS1='\n\[\033[01;35m\][\[\033[0m\]Diagrams\[\033[01;35m\]]\[\033[0m\]\n\[\033[01;35m\][\[\033[0m\]\[\033[01;32m\]\w\[\033[0m\]\[\033[01;35m\]]\[\033[0m\]\n \[\033[01;33m\]->\[\033[0m\] '" >> ~/.bashrc

CMD tail -f /dev/null

Container Build

Using Azure Pipelines I can build the container and added it to an Azure Container Registry (if you need to know how to setup ACR see my previous post on Configuring ACR)

trigger: 
 branches:
    include:
    - main
 paths:
    include: 
     - diagrams.dockerfile

pr: none

variables:
- group: Azure Connections
- name: dockerFilePath
  value: diagrams.dockerfile
- name: imageRepository
  value: dac/diagrams

pool:
  vmImage: "ubuntu-latest"

steps:
  - task: Docker@2
    displayName: "Build Diagram Image"
    inputs:
      containerRegistry: "$(myContainerRegistry)"
      repository: '$(imageRepository)'
      command: 'buildAndPush'
      Dockerfile: '$(dockerfilePath)'
      tags: |
        $(Build.BuildNumber)
        latest

With the container added to my registry I can use it in a pipeline to create my diagrams.

Image Build

The pipeline needs to create an image as an artifact and only when on the main branch to make sure only the final diagrams are published and not ones in progress.

The YAML below defines the pipeline:

trigger: 
   - main

pr: none

variables:
  isMain: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')]

jobs:
- job: creatediagram
  displayName: Create Diagram  
  pool:
    vmImage: ubuntu-latest
  container:
    image: $(myContainerRegistry)/dac/diagrams:latest
    endpoint: 'My Registry Service Connection'
  variables:
    workspaceFolder: 'TenantDesign'
  steps:
  - script: | 
      python tenant.py
      cp *.png $(Build.ArtifactStagingDirectory)
    displayName: Run Python
  - publish: '$(Build.ArtifactStagingDirectory)'
    displayName: Publish Diagrams
    artifact: $(workspaceFolder)
    condition: eq(variables.isMain, true)

Conclusion

I found using Diagrams simple and the documentation was good to allow picking up what was needed quickly. I will certainly be looking at using it for other diagrams in the future. I like the fact that it is easy to use, open source and supports custom images so you are not limited to the provided icons (Custom Docs).

Diagrams GitHub

Diagrams Docs

I hope that others find this useful and use Diagrams as Code for their projects.