Azure

Working with Azure Table Storage

Posted on by Mark Johnson

I’ve been working with Azure Table Storage for a few years and find it really useful for storing logs or static data or even as a data recovery store. Table Storage is an incredibly cheap way to store data.

The first time I used Table Storage I thought it was great, but there were times it was slow and I had no idea why and then I couldn’t isolate it to perform unit testing.

Research

Why is it slow?

  • First off you need to design your data to be accessed quickly. A good place to start is the Storage Design Guide
  • Nagle’s Algorithm – I really had no idea about this or how much it mattered, fortunately there is a great article to explain (despite being from 2010 it’s still useful)
  • The default connection limit in ServicePointManager is 2

How Do I Unit Test?

  • I could use the Azure Storage Emulator to perform tests, but it feels wrong having an external process for my tests and my build server will need to run this emulator too. On top of that we consider it good practice to not rely on external entities for our tests
  • I could write a wrapper around the Table Storage API and use an interface into my code

Batching?

  • The Table Storage API provides the ability to bulk/batch inserts. But this type of insert requires the Partition Key to be the same for each entry. I have found this to be a problem when there is multiple partitions to insert at once.

Solution

I decided to build a generic wrapper than encompassed being able to isolate the storage and configure the settings e.g. Nagle’s Algorithm.

The Wrapper

The wrapper has a TableStoreFactory method that creates the table store connection, or it allows you to create a TableStore directly.

The code below shows a very small example of injecting the TableStoreFactory and changing the options from the defaults.

public class TableStorageClient
{
    private ITableStore<MyDto> _store;

    public TableStorageClient(ITableStoreFactory factory)
    {
        var options = new TableStorageOptions
        {
            UseNagleAlgorithm = true,
            ConnectionLimit = 100,
            EnsureTableExists = false
        };

        _store = factory.CreateTableStore<MyDto>("MyTable", "UseDevelopmentStorage=true", options);
    }
}

You could also inject the TableStore

public class TableStorageClient
{
    private ITableStore<MyDto> _store;

    public TableStorageClient(ITableStore<MyDto> store)
    {
        _store = store;
    }
}

Or simply create the store in code

var store = new TableStore<MyDto>("MyTable", "UseDevelopmentStorage=true", new TableStorageOptions());

Batching

To handle the batch insert with multiple partition keys, I added the ability to automatically split the batch by key and then insert them in batches of Partition Key and up to the Max 100 records per batch. Now I can just create my list of entries and call insert without having to worry about it.

var entries = new List<MyDto>
{
    new MyDto("John", "Smith") {Age = 21, Email = "john.smith@something.com"},
    new MyDto("Jane", "Smith") {Age = 28, Email = "jane.smith@something.com"},
    new MyDto("Bill", "Smith") { Age = 38, Email = "bill.smith@another.com"},
    new MyDto("Fred", "Jones") {Age = 32, Email = "fred.jones@somewhere.com"},
    new MyDto("Bill", "Jones") {Age = 45, Email = "bill.jones@somewhere.com"},
    new MyDto("Bill", "King") {Age = 45, Email = "bill.king@email.com"},
    new MyDto("Fred", "Bloggs") { Age = 32, Email = "fred.bloggs@email.com" }
};

_store.InsertAsync(entries)

Filtering

Another noteworthy feature is GetRecordsByFilter, this allows secondary data to be filtered before returning the result (the filtering is done by passing in a predicate). The downside here is that it is required to get all records and then perform the filter, testing showed ~1.3 seconds for 10,000 records but when using paging and returning the first 100 it was ~0.0300 seconds for 10,000 records. 

// Without paging
_store.GetRecordsByFilter(x => x.Age > 21 && x.Age < 25);

// With paging
_store.GetRecordsByFilter(x => x.Age > 21 && x.Age < 25, 0, 100);

If there is a need to perform an action as the data is read from the table then there is support for Observable

var theObserver = _store.GetAllRecordsObservable();
theObserver.Where(entry => entry.Age > 21 && entry.Age < 25).Take(100).Subscribe(item =>
{
   // Do something with the table entry
});

The end result can be found on github and it is available on nuget. Others have introduced additions to this and they can be found on here and here

Cosmos DB

Table Storage does not support secondary indexes and global distribution, there was a Premium tier for Table Storage but now it is known as Cosmos DB.

The wrapper shown here will work with Cosmos DB but it does not support everything. For more details take a look at the FAQ’s and the new Table API.

Azure

Configuring Azure API Management

Posted on by Mark Johnson

Recently I found myself with a requirement to use Azure API Management (APIM) and had no idea where to start, so hit the known resources, Microsoft Docs, blogs, Azure Friday, etc. and I finally ended up with a solution and thought I would share.

Background

To give a little background to this, I work in an environment where the company’s Azure estate is managed by a separate team and they are responsible for creating and maintaining the main infrastructure, e.g. Virtual Networks, Firewalls, Application Gateways, Azure AD etc. Including an Azure API Management instance. I am sure many others work in a similar setup. A decision was made that that all APIs should go via the APIM and use Subscription Keys to authenticate.

As you can imagine this lead to many conversations about what is APIM? and how will we configure it for our APIs. A good start for this is the Microsoft Docs – API Management Key Concepts.

The application my team and I work on consists of a set of back-end APIs that are currently implemented using a mixture of Azure Service Fabric and Azure Functions.

Research

A bit of google searching led me to many articles and blogs on how to configure APIM with PowerShell. This is good but I found there is a lot of commands to get your head around. So I thought, what about ARM templates? surely they can be used for this. I however struggled to find much on this subject other than in the Microsoft docs and so decided to give this approach a go and see if I could create a template to configure the APIM. I also figured that as we are using Azure DevOps this would be a nice easy way to auto deploy the configuration to Azure.

Products and APIs

APIM configures Products and APIs. When there are many APIs it makes sense you want to group them under a product.

Each of the APIs for the application is deployed independently and so I needed a template to deploy the product configuration they will be part of.

Product

Configuring a Product at its basic level is one entry in resources that describes the product details.

Note: Full Product template configuration detail can be found here

 {
     "apiVersion": "[variables('apiManagementVersion')]",
     "type": "Microsoft.ApiManagement/service/products",
     "name": "[concat(parameters('ApiManagementInstanceName'), '/', parameters('ApiProductName'))]",
     "dependsOn": [],
     "properties": {
         "displayName": "[parameters('ApiProductDisplayName')]",
         "description": "[parameters('ApiProductDescription')]",
         "subscriptionRequired": "[variables('subscriptionRequired')]",
         "approvalRequired": "[variables('approvalRequired')]",
         "state": "published"
     }
 }

In order to get visibility of the product to developers you need to add a group to the product. The developers group is a built-in group and can be added simply by adding an inner resource to the products entry.

 {
     "apiVersion": "[variables('apiManagementVersion')]",
     "type": "Microsoft.ApiManagement/service/products",
     "name": "[concat(parameters('ApiManagementInstanceName'), '/', parameters('ApiProductName'))]",
     "dependsOn": [],
     "properties": {
         "displayName": "[parameters('ApiProductDisplayName')]",
         "description": "[parameters('ApiProductDescription')]",
         "subscriptionRequired": "[variables('subscriptionRequired')]",
         "approvalRequired": "[variables('approvalRequired')]",
         "state": "published"
     },
     "resources": [{
         "type": "Microsoft.ApiManagement/service/products/groups",
         "apiVersion": "[variables('apiManagementVersion')]",
         "name": "[concat(parameters('ApiManagementInstanceName'), '/',  parameters('ApiProductName'), '/developers')]",
         "dependsOn": [
             "[concat('Microsoft.ApiManagement/service/', parameters('ApiManagementInstanceName'), '/products/', parameters('ApiProductName'))]"
         ],
         "properties": {
             "displayName": "Developers",
             "description": "Developers is a built-in group. Its membership is managed by the system. Signed-in users fall into this group.",
             "builtIn": true,
             "type": "system"
         }
     }]
 }

So the full product template looks like this:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "APIManagementInstanceName": {
            "type": "string",
            "metadata": {
                "description": "The name of the API Management instance"
            }
        },
        "ApiProductName": {
            "type": "string",
            "metadata": {
                "description": "Name of the product the api should be associated with"
            }
        },
        "ApiProductDisplayName": {
            "type": "string",
            "metadata": {
                "description": "Display Name for the product the api should be associated with"
            }
        },
        "ApiProductDescription": {
            "type": "string",
            "metadata": {
                "description": "The product description"
            }
        },
        "ApiProductTagName" :{
            "type": "string",
            "metadata": {
                "description": "The product tag name"
            }
        }
    },
    "variables": {
        "apiManagementVersion": "2019-01-01",
        "approvalRequired": false,
        "subscriptionRequired": true,
        "allowTracing": true
    },
    "resources": [
        {
            "apiVersion": "[variables('apiManagementVersion')]",
            "type": "Microsoft.ApiManagement/service/products",
            "name": "[concat(parameters('ApiManagementInstanceName'), '/', parameters('ApiProductName'))]",
            "dependsOn": [],
            "properties": {
                "displayName": "[parameters('ApiProductDisplayName')]",
                "description": "[parameters('ApiProductDescription')]",
                "subscriptionRequired": "[variables('subscriptionRequired')]",
                "approvalRequired": "[variables('approvalRequired')]",
                "state": "published"
            },
            "resources": [
                {
                    "type": "Microsoft.ApiManagement/service/products/groups",
                    "apiVersion": "[variables('apiManagementVersion')]",
                    "name": "[concat(parameters('ApiManagementInstanceName'), '/',  parameters('ApiProductName'), '/developers')]",
                    "dependsOn": [
                        "[concat('Microsoft.ApiManagement/service/', parameters('ApiManagementInstanceName'), '/products/', parameters('ApiProductName'))]"
                    ],
                    "properties": {
                        "displayName": "Developers",
                        "description": "Developers is a built-in group. Its membership is managed by the system. Signed-in users fall into this group.",
                        "builtIn": true,
                        "type": "system"
                    }
                }             
            ]
        },
        {
            "apiVersion": "[variables('apiManagementVersion')]",
            "type": "Microsoft.ApiManagement/service/tags",
            "name": "[concat(parameters('APIManagementInstanceName'), '/', parameters('ApiProductName'))]",
            "dependsOn": [
                "[concat('Microsoft.ApiManagement/service/', parameters('APIManagementInstanceName'), '/products/', parameters('ApiProductName'))]"
            ],
            "properties": {
                "displayName": "[parameters('ApiProductTagName')]"
            }
        }        
    ]
}

API

Now the product is configured, the next step is to configure the APIs. All of the APIs we have use OpenAPI (formally swagger), so the template below shows how to configure this type of entry.

Note: Full API template configuration detail can be found here

{
    "apiVersion": "[variables('apiManagementVersion')]",
    "type": "Microsoft.ApiManagement/service/apis",
    "name": "[concat(parameters('ApiManagementInstanceName'), '/', parameters('ApiName'))]",
    "dependsOn": [],
    "properties": {
        "format": "swagger-link-json",
        "value": "[parameters('OpenApiUrl')]",
        "path": "[parameters('ApiPath')]",
        "protocols": "[parameters('ApiProtocols')]"
    }
}

Now the API needs to link to the Product that was created in the
product template earlier.

{
    "apiVersion": "[variables('apiManagementVersion')]",
    "type": "Microsoft.ApiManagement/service/products/apis",
    "name": "[concat(parameters('APIManagementInstanceName'), '/', parameters('ApiProductName'), '/',parameters('ApiName'))]",
    "dependsOn": [
        "[concat('Microsoft.ApiManagement/service/', parameters('ApiManagementInstanceName'), '/apis/', parameters('ApiName'))]"
    ],
    "properties": {}
}

As APIs are added to the APIM I can see it is going to get a little hard to navigate the without adding tags to the APIs. Adding a tag is easy and can be done with the following configuration:

{
    "apiVersion": "[variables('apiManagementVersion')]",
    "type": "Microsoft.ApiManagement/service/tags",
    "name": "[concat(parameters('APIManagementInstanceName'), '/', parameters('ApiProductTagName'))]",
    "dependsOn": [],
    "properties": {
        "displayName": "[parameters('ApiProductTagDisplayName')]"
    }
}, 
{
    "apiVersion": "[variables('apiManagementVersion')]",
    "type": "Microsoft.ApiManagement/service/apis/tags",
    "name": "[concat(parameters('APIManagementInstanceName'), '/', parameters('ApiName'), '/', parameters('ApiProductTagName'))]",
    "dependsOn": [
        "[concat('Microsoft.ApiManagement/service/', parameters('ApiManagementInstanceName'), '/apis/', parameters('ApiName'))]"
    ],
    "properties": {}
}

The final API template looks like this:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "APIManagementInstanceName": {
            "type": "string",
            "metadata": {
                "description": "The name of the API Management instance"
            }
        },
        "OpenApiUrl": {
            "type": "string",
            "metadata": {
                "description": "This is the URL to the swagger.json entry for your API"
            }
        },
        "ApiProtocols": {
            "type": "array",
            "metadata": {
                "description": "The array of allowed protocols for the API e.g. HTTP, HTTPS"
            }
        },
        "ApiProductName": {
            "type": "string",
            "metadata": {
                "description": "Name of the product the api should be associated with"
            }
        },
        "ApiName": {
            "type": "string",
            "metadata": {
                "description": "Name of the API entry in APIM"
            }
        },
        "ApiPath": {
            "type": "string",
            "metadata": {
                "description": "The path for the API Url Suffix"
            }
        },
        "ApiProductTagName" :{
            "type": "string",
            "metadata": {
                "description": "The product tag name"
            }
        },
        "ApiProductTagDisplayName" :{
            "type": "string",
            "metadata": {
                "description": "The product tag display name"
            }
        }
    },
    "variables": {
        "apiManagementVersion": "2019-01-01",       
        "subscriptionRequired": true
    },
    "resources": [{
            "apiVersion": "[variables('apiManagementVersion')]",
            "type": "Microsoft.ApiManagement/service/apis",
            "name": "[concat(parameters('ApiManagementInstanceName'), '/', parameters('ApiName'))]",
            "dependsOn": [],
            "properties": {
                "format": "swagger-link-json",
                "value": "[parameters('OpenApiUrl')]",
                "path": "[parameters('ApiPath')]",
                "protocols": "[parameters('ApiProtocols')]"
            }
        },
        {
            "apiVersion": "[variables('apiManagementVersion')]",
            "type": "Microsoft.ApiManagement/service/products/apis",
            "name": "[concat(parameters('APIManagementInstanceName'), '/', parameters('ApiProductName'), '/',parameters('ApiName'))]",
            "dependsOn": [
                "[concat('Microsoft.ApiManagement/service/', parameters('ApiManagementInstanceName'), '/apis/', parameters('ApiName'))]"
            ],
            "properties": {}
        },
        {
            "apiVersion": "[variables('apiManagementVersion')]",
            "type": "Microsoft.ApiManagement/service/tags",
            "name": "[concat(parameters('APIManagementInstanceName'), '/', parameters('ApiProductTagName'))]",
            "dependsOn": [],
            "properties": {
                "displayName": "[parameters('ApiProductTagDisplayName')]"
            }
        },
        {
            "apiVersion": "[variables('apiManagementVersion')]",
            "type": "Microsoft.ApiManagement/service/apis/tags",
            "name": "[concat(parameters('APIManagementInstanceName'), '/', parameters('ApiName'), '/', parameters('ApiProductTagName'))]",
            "dependsOn": [
                "[concat('Microsoft.ApiManagement/service/', parameters('ApiManagementInstanceName'), '/apis/', parameters('ApiName'))]"
            ],
            "properties": {}
        }
    ]
}

The templates can be deployed via PowerShell, Azure Cli or by a deployment pipeline like Azure DevOps.

I hope others find this is helpful, I personally have found this journey quite enjoyable and interesting.

I am looking deeper into API Management, specifically around security with OAuth 2.0 and MSI. I’ll add a post when I find out more.

Notes

The final templates shown above can be found on my github under the repository codingwithtaz.

If anyone is interested in using OpenAPI with Azure Functions v2, I used this nuget package to generate the swagger json, more details can be found here.