ARM | Coding With Taz

Following on from my previous article Co-locate IaC with My Application, this article is about consistently deploying IaC (Infrastructure as Code) from a pipeline. For this example I am going to use Azure Pipelines and deploy some simple infrastructure into Azure using ARM (Azure Resource Manager) templates. If you haven’t used Azure DevOps you can sign up for a free account here.

This example will show how to create the release in Azure Pipelines Releases as well as the new YAML stages in Azure Pipelines.

Azure Pipelines – Releases

In the Azure DevOps Portal select a project, in this case I am going to use an already setup project ‘CodingWithTaz’ and then from the left hand menu under Pipelines select ‘Releases’.

I have no releases at the moment and so I need to create a ‘New Pipeline’.

A new pipeline starts with two sections, Artifacts and Stages. When creating a new pipeline ‘Select a template’ is automatically selected, as there isn’t a template for deploying templates so I’ll choose ‘Empty job’.

Now I have an empty job I am going to select ‘Add an artifact’.

Artifacts can be from multiple sources e.g. a build pipeline, Azure Repos, GitHub, etc. for this example I am going to connect a GitHub repository where the ARM templates I want to deploy are located.

I already have a GitHub service connection setup and so I will use that and connect to the codingwithtaz repository.

I can now select ‘1 job, 0 task’ link on the ‘Stage 1’ which allows the Agent to be configured.

For this release I am going to leave the defaults which will use a hosted agent ‘Azure Pipelines’.

The agent is configured and I need to now add tasks for the agent to run.

Selecting the ‘+’ on Agent job brings up ‘Add tasks’. There are a lot of tasks so I’ll search for ‘ARM’ to narrow the list and then select ‘Add’ on ‘ARM template deployment’ to add the task.

Now the task is added it shows that some settings need attention. Selecting the task allows it to be configured.

For this example I will use the Resource Group deployment scope and I already have an Azure Resource Manager connection to use named ‘Twisters Portal’. If you need to configure a connection then follow the instructions on the Microsoft Docs.

The action defaults to ‘Create or update resource group’ and so I will leave that. If the required resource group already exists you can select it from the drop down, otherwise enter the name of the resource group to be created. For this I am going to create a new resource group ‘My-test-app’ and use West Europe as the location for the resource group.

Having configured my connection and resource group, I now need to scroll down and configure the template I am going to use in this release.

Selecting the three dots on the right of ‘Template’ brings up the linked artifacts dialog. As I have configured a GitHub repository I can now navigate that to find the template I require, for this example I will use a simple template that creates a storage account.

I will now follow the same steps for the template parameters.

If you wish to see the templates used here they are available on GitHub and in the same path structure as shown in the above dialog.

This template allows me to override the storage account name, so I will override the StorageName with a variable called ‘storageName’ in the ‘Override template parameters’ box.

After adding the override parameter, I need to define the ‘storageName’ variable that I used. Selecting the variables tab will allow me to add the new ‘storageName’ variable and provide a value.

Note: Storage Account names have to be all lowercase, no spaces, no special characters and between 3 and 24 characters.

All the steps to deploy the ARM template are now configured and I can run the pipeline.

The template was successful and you can see the output in the Logs tab.

Let’s head over to the Azure Portal and see if the storage account was created.

Well the account was created successfully. You may notice that the storage name isn’t exactly the value configured, this is because in the ARM template the name is combined with a unique string and uses all 24 characters. This is useful when requiring a unique name.

Azure Pipelines YAML

If you prefer to use the new YAML pipeline to create your release as code then you can configure through Azure Pipelines and save the YAML file to your repository.

First of all I am going to select ‘Create Pipeline’

For the purpose of this example I am going to use the same repository as I did for the Releases. Selecting the GitHub connection which will guide through logging in to GitHub and then configuring Azure Pipelines to use the connection.

Once configured you will get to the review page which contains a basic YAML configuration ready to be saved into the repository.

Obviously this is not the build we want to create, so we need to change it achieve the same as the Releases Pipeline.

As with the Releases Pipeline there are some variables that need to be created to match the YAML code.

Note: the PortalSubscription is a secret variable.

To demonstrate using multi-stage pipelines I have added a Build and Test stage and then a Release stage. Each stage has different configuration and different type of agent (defined by vmImage).

Saving this will commit the YAML file to the repository in GitHub called azure-pipelines.yml.

Note: First time run will ask for confirmation of connection to the Azure Portal, each subsequent run will use the confirmed connection.

Once the code runs, the stages can be visualized in the view.

It would be nice to be able to visualize the stages without running the pipeline especially if running a parallel release or fan-out-fan-in pattern.

For further reading on YAML multi-stage pipelines have a look at the Microsoft devblog and the examples on GitHub.

For general information on Azure Pipelines the Microsoft docs are very useful.

Recently I found myself with a requirement to use Azure API Management (APIM) and had no idea where to start, so hit the known resources, Microsoft Docs, blogs, Azure Friday, etc. and I finally ended up with a solution and thought I would share.

Background

To give a little background to this, I work in an environment where the company’s Azure estate is managed by a separate team and they are responsible for creating and maintaining the main infrastructure, e.g. Virtual Networks, Firewalls, Application Gateways, Azure AD etc. Including an Azure API Management instance. I am sure many others work in a similar setup. A decision was made that that all APIs should go via the APIM and use Subscription Keys to authenticate.

As you can imagine this lead to many conversations about what is APIM? and how will we configure it for our APIs. A good start for this is the Microsoft Docs – API Management Key Concepts.

The application my team and I work on consists of a set of back-end APIs that are currently implemented using a mixture of Azure Service Fabric and Azure Functions.

Research

A bit of google searching led me to many articles and blogs on how to configure APIM with PowerShell. This is good but I found there is a lot of commands to get your head around. So I thought, what about ARM templates? surely they can be used for this. I however struggled to find much on this subject other than in the Microsoft docs and so decided to give this approach a go and see if I could create a template to configure the APIM. I also figured that as we are using Azure DevOps this would be a nice easy way to auto deploy the configuration to Azure.

Products and APIs

APIM configures Products and APIs. When there are many APIs it makes sense you want to group them under a product.

Each of the APIs for the application is deployed independently and so I needed a template to deploy the product configuration they will be part of.

Product

Configuring a Product at its basic level is one entry in resources that describes the product details.

Note: Full Product template configuration detail can be found here

 {
     "apiVersion": "[variables('apiManagementVersion')]",
     "type": "Microsoft.ApiManagement/service/products",
     "name": "[concat(parameters('ApiManagementInstanceName'), '/', parameters('ApiProductName'))]",
     "dependsOn": [],
     "properties": {
         "displayName": "[parameters('ApiProductDisplayName')]",
         "description": "[parameters('ApiProductDescription')]",
         "subscriptionRequired": "[variables('subscriptionRequired')]",
         "approvalRequired": "[variables('approvalRequired')]",
         "state": "published"
     }
 }

In order to get visibility of the product to developers you need to add a group to the product. The developers group is a built-in group and can be added simply by adding an inner resource to the products entry.

 {
     "apiVersion": "[variables('apiManagementVersion')]",
     "type": "Microsoft.ApiManagement/service/products",
     "name": "[concat(parameters('ApiManagementInstanceName'), '/', parameters('ApiProductName'))]",
     "dependsOn": [],
     "properties": {
         "displayName": "[parameters('ApiProductDisplayName')]",
         "description": "[parameters('ApiProductDescription')]",
         "subscriptionRequired": "[variables('subscriptionRequired')]",
         "approvalRequired": "[variables('approvalRequired')]",
         "state": "published"
     },
     "resources": [{
         "type": "Microsoft.ApiManagement/service/products/groups",
         "apiVersion": "[variables('apiManagementVersion')]",
         "name": "[concat(parameters('ApiManagementInstanceName'), '/',  parameters('ApiProductName'), '/developers')]",
         "dependsOn": [
             "[concat('Microsoft.ApiManagement/service/', parameters('ApiManagementInstanceName'), '/products/', parameters('ApiProductName'))]"
         ],
         "properties": {
             "displayName": "Developers",
             "description": "Developers is a built-in group. Its membership is managed by the system. Signed-in users fall into this group.",
             "builtIn": true,
             "type": "system"
         }
     }]
 }

So the full product template looks like this:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "APIManagementInstanceName": {
            "type": "string",
            "metadata": {
                "description": "The name of the API Management instance"
            }
        },
        "ApiProductName": {
            "type": "string",
            "metadata": {
                "description": "Name of the product the api should be associated with"
            }
        },
        "ApiProductDisplayName": {
            "type": "string",
            "metadata": {
                "description": "Display Name for the product the api should be associated with"
            }
        },
        "ApiProductDescription": {
            "type": "string",
            "metadata": {
                "description": "The product description"
            }
        },
        "ApiProductTagName" :{
            "type": "string",
            "metadata": {
                "description": "The product tag name"
            }
        }
    },
    "variables": {
        "apiManagementVersion": "2019-01-01",
        "approvalRequired": false,
        "subscriptionRequired": true,
        "allowTracing": true
    },
    "resources": [
        {
            "apiVersion": "[variables('apiManagementVersion')]",
            "type": "Microsoft.ApiManagement/service/products",
            "name": "[concat(parameters('ApiManagementInstanceName'), '/', parameters('ApiProductName'))]",
            "dependsOn": [],
            "properties": {
                "displayName": "[parameters('ApiProductDisplayName')]",
                "description": "[parameters('ApiProductDescription')]",
                "subscriptionRequired": "[variables('subscriptionRequired')]",
                "approvalRequired": "[variables('approvalRequired')]",
                "state": "published"
            },
            "resources": [
                {
                    "type": "Microsoft.ApiManagement/service/products/groups",
                    "apiVersion": "[variables('apiManagementVersion')]",
                    "name": "[concat(parameters('ApiManagementInstanceName'), '/',  parameters('ApiProductName'), '/developers')]",
                    "dependsOn": [
                        "[concat('Microsoft.ApiManagement/service/', parameters('ApiManagementInstanceName'), '/products/', parameters('ApiProductName'))]"
                    ],
                    "properties": {
                        "displayName": "Developers",
                        "description": "Developers is a built-in group. Its membership is managed by the system. Signed-in users fall into this group.",
                        "builtIn": true,
                        "type": "system"
                    }
                }             
            ]
        },
        {
            "apiVersion": "[variables('apiManagementVersion')]",
            "type": "Microsoft.ApiManagement/service/tags",
            "name": "[concat(parameters('APIManagementInstanceName'), '/', parameters('ApiProductName'))]",
            "dependsOn": [
                "[concat('Microsoft.ApiManagement/service/', parameters('APIManagementInstanceName'), '/products/', parameters('ApiProductName'))]"
            ],
            "properties": {
                "displayName": "[parameters('ApiProductTagName')]"
            }
        }        
    ]
}

API

Now the product is configured, the next step is to configure the APIs. All of the APIs we have use OpenAPI (formally swagger), so the template below shows how to configure this type of entry.

Note: Full API template configuration detail can be found here

{
    "apiVersion": "[variables('apiManagementVersion')]",
    "type": "Microsoft.ApiManagement/service/apis",
    "name": "[concat(parameters('ApiManagementInstanceName'), '/', parameters('ApiName'))]",
    "dependsOn": [],
    "properties": {
        "format": "swagger-link-json",
        "value": "[parameters('OpenApiUrl')]",
        "path": "[parameters('ApiPath')]",
        "protocols": "[parameters('ApiProtocols')]"
    }
}

Now the API needs to link to the Product that was created in the
product template earlier.

{
    "apiVersion": "[variables('apiManagementVersion')]",
    "type": "Microsoft.ApiManagement/service/products/apis",
    "name": "[concat(parameters('APIManagementInstanceName'), '/', parameters('ApiProductName'), '/',parameters('ApiName'))]",
    "dependsOn": [
        "[concat('Microsoft.ApiManagement/service/', parameters('ApiManagementInstanceName'), '/apis/', parameters('ApiName'))]"
    ],
    "properties": {}
}

As APIs are added to the APIM I can see it is going to get a little hard to navigate the without adding tags to the APIs. Adding a tag is easy and can be done with the following configuration:

{
    "apiVersion": "[variables('apiManagementVersion')]",
    "type": "Microsoft.ApiManagement/service/tags",
    "name": "[concat(parameters('APIManagementInstanceName'), '/', parameters('ApiProductTagName'))]",
    "dependsOn": [],
    "properties": {
        "displayName": "[parameters('ApiProductTagDisplayName')]"
    }
}, 
{
    "apiVersion": "[variables('apiManagementVersion')]",
    "type": "Microsoft.ApiManagement/service/apis/tags",
    "name": "[concat(parameters('APIManagementInstanceName'), '/', parameters('ApiName'), '/', parameters('ApiProductTagName'))]",
    "dependsOn": [
        "[concat('Microsoft.ApiManagement/service/', parameters('ApiManagementInstanceName'), '/apis/', parameters('ApiName'))]"
    ],
    "properties": {}
}

The final API template looks like this:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "APIManagementInstanceName": {
            "type": "string",
            "metadata": {
                "description": "The name of the API Management instance"
            }
        },
        "OpenApiUrl": {
            "type": "string",
            "metadata": {
                "description": "This is the URL to the swagger.json entry for your API"
            }
        },
        "ApiProtocols": {
            "type": "array",
            "metadata": {
                "description": "The array of allowed protocols for the API e.g. HTTP, HTTPS"
            }
        },
        "ApiProductName": {
            "type": "string",
            "metadata": {
                "description": "Name of the product the api should be associated with"
            }
        },
        "ApiName": {
            "type": "string",
            "metadata": {
                "description": "Name of the API entry in APIM"
            }
        },
        "ApiPath": {
            "type": "string",
            "metadata": {
                "description": "The path for the API Url Suffix"
            }
        },
        "ApiProductTagName" :{
            "type": "string",
            "metadata": {
                "description": "The product tag name"
            }
        },
        "ApiProductTagDisplayName" :{
            "type": "string",
            "metadata": {
                "description": "The product tag display name"
            }
        }
    },
    "variables": {
        "apiManagementVersion": "2019-01-01",       
        "subscriptionRequired": true
    },
    "resources": [{
            "apiVersion": "[variables('apiManagementVersion')]",
            "type": "Microsoft.ApiManagement/service/apis",
            "name": "[concat(parameters('ApiManagementInstanceName'), '/', parameters('ApiName'))]",
            "dependsOn": [],
            "properties": {
                "format": "swagger-link-json",
                "value": "[parameters('OpenApiUrl')]",
                "path": "[parameters('ApiPath')]",
                "protocols": "[parameters('ApiProtocols')]"
            }
        },
        {
            "apiVersion": "[variables('apiManagementVersion')]",
            "type": "Microsoft.ApiManagement/service/products/apis",
            "name": "[concat(parameters('APIManagementInstanceName'), '/', parameters('ApiProductName'), '/',parameters('ApiName'))]",
            "dependsOn": [
                "[concat('Microsoft.ApiManagement/service/', parameters('ApiManagementInstanceName'), '/apis/', parameters('ApiName'))]"
            ],
            "properties": {}
        },
        {
            "apiVersion": "[variables('apiManagementVersion')]",
            "type": "Microsoft.ApiManagement/service/tags",
            "name": "[concat(parameters('APIManagementInstanceName'), '/', parameters('ApiProductTagName'))]",
            "dependsOn": [],
            "properties": {
                "displayName": "[parameters('ApiProductTagDisplayName')]"
            }
        },
        {
            "apiVersion": "[variables('apiManagementVersion')]",
            "type": "Microsoft.ApiManagement/service/apis/tags",
            "name": "[concat(parameters('APIManagementInstanceName'), '/', parameters('ApiName'), '/', parameters('ApiProductTagName'))]",
            "dependsOn": [
                "[concat('Microsoft.ApiManagement/service/', parameters('ApiManagementInstanceName'), '/apis/', parameters('ApiName'))]"
            ],
            "properties": {}
        }
    ]
}

The templates can be deployed via PowerShell, Azure Cli or by a deployment pipeline like Azure DevOps.

I hope others find this is helpful, I personally have found this journey quite enjoyable and interesting.

I am looking deeper into API Management, specifically around security with OAuth 2.0 and MSI. I’ll add a post when I find out more.

Notes

The final templates shown above can be found on my github under the repository codingwithtaz.

If anyone is interested in using OpenAPI with Azure Functions v2, I used this nuget package to generate the swagger json, more details can be found here.

Coding With Taz

Coding With Taz

Thoughts and things I learn

Tag: ARM

IaC ARM Templates with Azure Pipelines