Azure, Azure Pipelines, DevOps

IaC Ansible with Azure Pipelines

Posted on by Mark Johnson

In my previous article IaC ARM Templates with Azure Pipelines I showed creating a Azure Pipeline Release and YAML multi-stage pipeline. In this article I am going to show using Ansible to deploy IaC to Azure using Azure Pipelines.

The Ansible Playbook used for this example can be found on GitHub.

As I covered creating piplelines in the previous article I will skip some of those steps and show the steps needed for Ansible and again show a multi-stage pipleline in YAML.

Azure Pipelines – Releases

For deploying ARM templates I left the defaults for the agent, this time I am going to change the Agent Specification to use Ubuntu.

The first thing I am going to add to the pipleline is Azure CLI.

The purpose of the CLI step is to get the Azure connection details to supply to Ansible in order to be able to deploy to Azure.

I’ve used an inline script to configure some Azure Pipeline variables on the fly from the CLI response. For this to work the “Access service principle details in script” must be selected.

echo "##vso[task.setvariable variable=ARM_SUBSCRIPTION_ID]$(az account show --query="id" -o tsv)"
echo "##vso[task.setvariable variable=ARM_CLIENT_ID]$servicePrincipalId"
echo "##vso[task.setvariable variable=ARM_CLIENT_SECRET]$servicePrincipalKey"
echo "##vso[task.setvariable variable=ARM_TENANT_ID]$tenantId"

As this is running on a hosted agent Ansible needs installing.

Ansible can use environment variables, a credentials file or have the variables defined in the playbook. As we want to keep secrets out of playbooks the others are preferred. For some reason the environment variable approach does not work using the Ansible Task but creating the credential file works for either the Ansible or Bash tasks.

On the hosted agent the credential file is either blank or doesn’t exist so we need to create/update it using the variables created in the Azure CLI step. Once this has been updated we can then install Ansible with Azure modules.

touch ~/.azure/credentials
echo "[default]" >> ~/.azure/credentials  
echo "subscription_id=$(ARM_SUBSCRIPTION_ID)" >> ~/.azure/credentials   
echo "client_id=$(ARM_CLIENT_ID)" >> ~/.azure/credentials  
echo "secret=$(ARM_CLIENT_SECRET)" >> ~/.azure/credentials  
echo "tenant=$(ARM_TENANT_ID)" >> ~/.azure/credentials  

pip install ansible[azure]

Now Ansible is installed on the agent, I can execute a simple playbook. As with the ARM template example I am going to deploy a storage account into a resource group, both of which are created by the playbook.

ansible-playbook $(System.DefaultWorkingDirectory)/_codingwithtaz/azure/storage/storage.yml --extra-vars "storage_name=$(storageName) storage_resource_group=$(storageGroup) storage_location=$(storageLocation)"

There are some variables that setting up for this to provide the playbook with the required values for the storage account.

That is all that is needed to run the playbook, let’s see if our pipeline works.

And in Azure, it looks like the account and resource group were created as expected.

There is an alternative to running the Ansible Playbook using a script, there is a Microsoft Ansible Task. I will add this task and remove the script.

Now I need to select the playbook for the release.

Select the playbook file from the linked GitHub artifacts.

Now we can see the completed task, for this run I am using the agent machine which is configured as a hosted agent but this could be a private agent you have setup.

--extra-vars "storage_name=$(storageName) storage_resource_group=$(storageGroup) storage_location=$(storageLocation)"

Let’s change the storage account name.

Now, let’s try and run the pipeline again and see if it works.

And in Azure the storage account was created and with the new name.

Ansible Task

You may have noticed when configuring the Ansible task that for the Ansible location there are two settings “Agent machine” and “Remote machine”. The “Remote machine” can be a Virtual Machine you have already setup for running your Ansible scripts and want to use that. I am not going to go into setting this up but it’s worth noting that the task supports this.

For more information on this extension see the Marketplace.

Azure Pipelines YAML

The previous article goes into detail about creating the pipeline via the Azure DevOps portal, so here I will just include the YAML for both Ansible deployment variants. Both examples show a multi-stage and multi-job pipeline.

YAML code for the pipeline can be found on GitHub.

# A pipeline with no CI trigger
trigger: none

# No PR triggers
pr: none

stages:
- stage: build_test
  displayName: Build and Test
  jobs:
  - job: build_test
    pool:
      vmImage: 'windows-latest' # Currently Windows 2019 and Visual Studio 2019
    steps:
    - script: echo Code Built and Tested!

- stage: release
  displayName: Release
  dependsOn: build_test
  jobs:
   - job: deploy_ansible
     pool:
       vmImage: 'ubuntu-latest' # Currently Ubuntu 18.04
     steps:
      - task: AzureCLI@2
        displayName: 'Azure CLI '
        inputs:
          azureSubscription: 'Twisters Portal'
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            echo "##vso[task.setvariable variable=ARM_SUBSCRIPTION_ID]$(az account show --query="id" -o tsv)"
            echo "##vso[task.setvariable variable=ARM_CLIENT_ID]$servicePrincipalId"
            echo "##vso[task.setvariable variable=ARM_CLIENT_SECRET]$servicePrincipalKey"
            echo "##vso[task.setvariable variable=ARM_TENANT_ID]$tenantId"
          addSpnToEnvironment: true     

      - bash: |
          touch ~/.azure/credentials
          echo "[default]" >> ~/.azure/credentials  
          echo "subscription_id=$(ARM_SUBSCRIPTION_ID)" >> ~/.azure/credentials   
          echo "client_id=$(ARM_CLIENT_ID)" >> ~/.azure/credentials  
          echo "secret=$(ARM_CLIENT_SECRET)" >> ~/.azure/credentials  
          echo "tenant=$(ARM_TENANT_ID)" >> ~/.azure/credentials  
          
          pip install ansible[azure]
        displayName: 'Install Ansible Azure + Environment'

      - bash: |
          ansible-playbook ./azure/storage/storage.yml --extra-vars "storage_name=$(storageName) storage_resource_group=$(storageGroup) storage_location=$(storageLocation)"
        displayName: 'Run Playbook Script'              

   - job: deploy_build
     dependsOn: deploy_ansible
     pool:
       vmImage: 'windows-latest' # Currently Windows 2019 and Visual Studio 2019
     steps:
      - script: echo Deploying Code



# A pipeline with no CI trigger
trigger: none

# No PR triggers
pr: none

stages:
- stage: build_test
  displayName: Build and Test
  jobs:
  - job: build_test
    pool:
      vmImage: 'windows-latest' # Currently Windows 2019 and Visual Studio 2019
    steps:
    - script: echo Code Built and Tested!

- stage: release
  displayName: Release
  dependsOn: build_test
  jobs:
   - job: deploy_ansible
     pool:
       vmImage: 'ubuntu-latest' # Currently Ubuntu 18.04
     steps:
      - task: AzureCLI@2
        displayName: 'Azure CLI '
        inputs:
          azureSubscription: 'Twisters Portal'
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            echo "##vso[task.setvariable variable=ARM_SUBSCRIPTION_ID]$(az account show --query="id" -o tsv)"
            echo "##vso[task.setvariable variable=ARM_CLIENT_ID]$servicePrincipalId"
            echo "##vso[task.setvariable variable=ARM_CLIENT_SECRET]$servicePrincipalKey"
            echo "##vso[task.setvariable variable=ARM_TENANT_ID]$tenantId"
          addSpnToEnvironment: true     

      - bash: |
          touch ~/.azure/credentials
          echo "[default]" >> ~/.azure/credentials  
          echo "subscription_id=$(ARM_SUBSCRIPTION_ID)" >> ~/.azure/credentials   
          echo "client_id=$(ARM_CLIENT_ID)" >> ~/.azure/credentials  
          echo "secret=$(ARM_CLIENT_SECRET)" >> ~/.azure/credentials  
          echo "tenant=$(ARM_TENANT_ID)" >> ~/.azure/credentials  
          
          pip install ansible[azure]
        displayName: 'Install Ansible Azure + Environment'
            
      - task: Ansible@0
        inputs:
          ansibleInterface: 'agentMachine'
          playbookPathOnAgentMachine: './azure/storage/storage.yml'
          inventoriesAgentMachine: 'noInventory'
          args: '--extra-vars "storage_name=$(storageName) storage_resource_group=$(storageGroup) storage_location=$(storageLocation)"'
          failOnStdErr: false

   - job: deploy_build
     dependsOn: deploy_ansible
     pool:
       vmImage: 'windows-latest' # Currently Windows 2019 and Visual Studio 2019
     steps:
      - script: echo Deploying Code

Let’s run the pipeline and see the output.

And again check the Azure Portal to see if it really was successful.

I hope this article is useful and helps you get started running your own Ansible playbooks using Azure Pipelines.

2
Azure, Azure Pipelines, DevOps

IaC ARM Templates with Azure Pipelines

Posted on by Mark Johnson

Following on from my previous article Co-locate IaC with My Application, this article is about consistently deploying IaC (Infrastructure as Code) from a pipeline. For this example I am going to use Azure Pipelines and deploy some simple infrastructure into Azure using ARM (Azure Resource Manager) templates. If you haven’t used Azure DevOps you can sign up for a free account here.

This example will show how to create the release in Azure Pipelines Releases as well as the new YAML stages in Azure Pipelines.

Azure Pipelines – Releases

In the Azure DevOps Portal select a project, in this case I am going to use an already setup project ‘CodingWithTaz’ and then from the left hand menu under Pipelines select ‘Releases’.

I have no releases at the moment and so I need to create a ‘New Pipeline’.

A new pipeline starts with two sections, Artifacts and Stages. When creating a new pipeline ‘Select a template’ is automatically selected, as there isn’t a template for deploying templates so I’ll choose ‘Empty job’.

Now I have an empty job I am going to select ‘Add an artifact’.

Artifacts can be from multiple sources e.g. a build pipeline, Azure Repos, GitHub, etc. for this example I am going to connect a GitHub repository where the ARM templates I want to deploy are located.

I already have a GitHub service connection setup and so I will use that and connect to the codingwithtaz repository.

I can now select ‘1 job, 0 task’ link on the ‘Stage 1’ which allows the Agent to be configured.

For this release I am going to leave the defaults which will use a hosted agent ‘Azure Pipelines’.

The agent is configured and I need to now add tasks for the agent to run.

Selecting the ‘+’ on Agent job brings up ‘Add tasks’. There are a lot of tasks so I’ll search for ‘ARM’ to narrow the list and then select ‘Add’ on ‘ARM template deployment’ to add the task.

Now the task is added it shows that some settings need attention. Selecting the task allows it to be configured.

For this example I will use the Resource Group deployment scope and I already have an Azure Resource Manager connection to use named ‘Twisters Portal’. If you need to configure a connection then follow the instructions on the Microsoft Docs.

The action defaults to ‘Create or update resource group’ and so I will leave that. If the required resource group already exists you can select it from the drop down, otherwise enter the name of the resource group to be created. For this I am going to create a new resource group ‘My-test-app’ and use West Europe as the location for the resource group.

Having configured my connection and resource group, I now need to scroll down and configure the template I am going to use in this release.

Selecting the three dots on the right of ‘Template’ brings up the linked artifacts dialog. As I have configured a GitHub repository I can now navigate that to find the template I require, for this example I will use a simple template that creates a storage account.

I will now follow the same steps for the template parameters.

If you wish to see the templates used here they are available on GitHub and in the same path structure as shown in the above dialog.

This template allows me to override the storage account name, so I will override the StorageName with a variable called ‘storageName’ in the ‘Override template parameters’ box.

After adding the override parameter, I need to define the ‘storageName’ variable that I used. Selecting the variables tab will allow me to add the new ‘storageName’ variable and provide a value.

Note: Storage Account names have to be all lowercase, no spaces, no special characters and between 3 and 24 characters.

All the steps to deploy the ARM template are now configured and I can run the pipeline.

The template was successful and you can see the output in the Logs tab.

Let’s head over to the Azure Portal and see if the storage account was created.

Well the account was created successfully. You may notice that the storage name isn’t exactly the value configured, this is because in the ARM template the name is combined with a unique string and uses all 24 characters. This is useful when requiring a unique name.

Azure Pipelines YAML

If you prefer to use the new YAML pipeline to create your release as code then you can configure through Azure Pipelines and save the YAML file to your repository.

First of all I am going to select ‘Create Pipeline’

For the purpose of this example I am going to use the same repository as I did for the Releases. Selecting the GitHub connection which will guide through logging in to GitHub and then configuring Azure Pipelines to use the connection.

Once configured you will get to the review page which contains a basic YAML configuration ready to be saved into the repository.

Obviously this is not the build we want to create, so we need to change it achieve the same as the Releases Pipeline.

As with the Releases Pipeline there are some variables that need to be created to match the YAML code.

Note: the PortalSubscription is a secret variable.

To demonstrate using multi-stage pipelines I have added a Build and Test stage and then a Release stage. Each stage has different configuration and different type of agent (defined by vmImage).

Saving this will commit the YAML file to the repository in GitHub called azure-pipelines.yml.

Note: First time run will ask for confirmation of connection to the Azure Portal, each subsequent run will use the confirmed connection.

Once the code runs, the stages can be visualized in the view.

It would be nice to be able to visualize the stages without running the pipeline especially if running a parallel release or fan-out-fan-in pattern.

For further reading on YAML multi-stage pipelines have a look at the Microsoft devblog and the examples on GitHub.

For general information on Azure Pipelines the Microsoft docs are very useful.

3