Tag: DevOps

MLOps

Build Event-Driven ML Pipelines with Argo Workflows

Posted on by Mark Johnson

I’ve been diving into the world of MLOps lately, curious about how modern ML pipelines can be made more scalable and maintainable. Machine learning is so much more than just training a model, there’s data pre-processing, feature engineering, evaluation, deployment, and the ongoing need for everything to be reproducible.

As a DevOps engineer, I’ve spent years designing reliable workflows for CI/CD and infrastructure automation, but hadn’t explored how those same principles could apply to ML pipelines. That’s where Argo Workflows and Argo Events caught my attention. They’re lightweight, Kubernetes-native, and from what I’ve seen so far, they’re gaining real traction in the MLOps space.

This post is my first hands-on look at these tools, setting up Argo Workflows and Argo Events on a local cluster with kind and exploring how they might enable event-driven, reproducible ML pipelines.

🧠Why Argo Workflows for MLOps?

Traditional ML pipelines are often stitched together using ad-hoc scripts, cron jobs, or heavy frameworks like Kubeflow. Argo Workflows offers a Kubernetes-native, lightweight alternative for orchestrating ML pipelines with:

  • Containerised tasks: Each step runs in its own container for reproducibility.
  • DAG-based workflows: Easily express complex pipelines with dependencies.
  • Event-driven triggers: With Argo Events, workflows can be launched automatically when new data arrives or other events occur.
  • Parallel execution: Fan-out tasks for hyperparameter tuning, multi-model training, or batch inference.
  • Retry strategies & exit handlers: Add robustness with built-in error handling and graceful exits.
  • Artifact management: Integrate with MinIO or volume mounts to persist model files, metrics, or datasets across steps.

Compared to tools like Kubeflow, Argo is simpler and less opinionated, making it easier to integrate with tools like MLflow, Seldon Core, etc. Its flexibility lets you tailor pipelines to your needs without locking into rigid frameworks.

🛠️Initial Setup

Here’s the setup I used for experimenting locally:

Create a kind Cluster

kind create cluster --name mlops-local

Install Argo Workflows

Run these commands to add it to the cluster:

kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/latest/download/install.yaml

Install Argo Events

Run these commands to add it to the kind cluster:

kubectl create namespace argo-events
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install.yaml

With everything installed, let’s walk through building an event-driven pipeline step-by-step.

Setting Up Event-Driven Pipelines

The fun part of MLOps with Argo is how events can trigger workflows. Here’s a minimal example I tried using webhooks.

Define an EventBus

Once defined, load it into the kind cluster:

🔽📄eventbus.yaml
apiVersion: argoproj.io/v1alpha1
kind: EventBus
metadata:
  name: default
spec:
  nats:
    native:
      # minimum3.
      replicas: 3
      auth: token

kubectl apply -n argo-events -f eventbus.yaml

Roles and Service Account

You’ll need a role, rolebinding, and service account to allow Argo Events to trigger workflows.

🔽📄role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argo-workflow-role
  namespace: argo-events
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["patch", "create", "get", "list", "watch", "delete"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["argoproj.io"]
    resources: ["workflows"]
    verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
  - apiGroups: ["argoproj.io"]
    resources: ["workflows/finalizers"]
    verbs: ["update"]
  - apiGroups: ["argoproj.io"]
    resources: ["workflowtaskresults"]
    verbs: ["create","patch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["serviceaccounts"]
    verbs: ["get"]
🔽📄rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argo-workflowtaskresults-binding
  namespace: argo-events
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name:  argo-workflow-role
subjects:
  - kind: ServiceAccount
    name: operate-workflow-sa
    namespace: argo-events
🔽📄serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: operate-workflow-sa
  namespace: argo-events

With everything configured, load it into the kind cluster:

kubectl apply -n argo-events -f role.yaml
kubectl apply -n argo-events -f rolebinding.yaml
kubectl apply -n argo-events -f serviceaccount.yaml

Define a Webhook EventSource

This sets up a simple HTTP endpoint that triggers a workflow when called.

🔽📄event-source.yaml
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: webhook
  namespace: argo-events
spec:
  service:
    ports:
      - port: 12000
        targetPort: 12000
  webhook:
    trigger:
      port: "12000"
      endpoint: /trigger
      method: POST

Once defined, load it into your cluster:

kubectl apply -n argo-events -f event-source.yaml

Define a Sensor to Trigger Workflows

🔽📄sensor.yaml
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: ml-pipeline-sensor
  namespace: argo-events
spec:
  template:
    serviceAccountName: operate-workflow-sa
  dependencies:
    - name: training-event
      eventSourceName: webhook
      eventName: trigger
  triggers:
    - template:
        name: trigger-ml-pipeline
        k8s:
          operation: create
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                generateName: ml-pipeline-run-
              spec:
                workflowTemplateRef:
                  name: ml-pipeline-template
                arguments:
                  parameters:
                    - name: model
                      value: default-model
                    - name: dataset
                      value: default-dataset
          parameters:
            - src:
                dependencyName: training-event
                dataKey: body.model
              dest: spec.arguments.parameters.0.value
            - src:
                dependencyName: training-event
                dataKey: body.dataset
              dest: spec.arguments.parameters.1.value

Once defined, load it into your cluster:

kubectl apply -n argo-events -f sensor.yaml

Define a Workflow Template

This is a mock ML pipeline with train and evaluate steps with an example of using parameters

🔽📄workflowtemplate.yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: ml-pipeline-template
  namespace: argo-events
spec:
  entrypoint: pipeline
  serviceAccountName: operate-workflow-sa 
  templates:
    - name: pipeline
      dag:
        tasks:
          - name: train-model
            template: train
            arguments:
              parameters:
                - name: model
                  value: "{{workflow.parameters.model}}"
          - name: evaluate-model
            dependencies: [train-model]
            template: evaluate
            arguments:
              parameters:
                - name: dataset
                  value: "{{workflow.parameters.dataset}}"

    - name: train
      inputs:
        parameters:
          - name: model
      container:
        image: python:3.9
        command: ["python"]
        args: ["-c", "print('Training {{inputs.parameters.model}}...')"]

    - name: evaluate
      inputs:
        parameters:
          - name: dataset
      container:
        image: python:3.9
        command: ["python"]
        args: ["-c", "print('Evaluating {{inputs.parameters.dataset}}...')"]

Once defined, load it into your cluster:

kubectl apply -n argo-events -f workflowtemplate.yaml

📬Trigger Event

First expose the webhook so it can be actioned:

kubectl -n argo-events port-forward svc/webhook-eventsource-svc 12000:12000

Trigger it by sending a POST request via curl:

curl -d '{"model":"resnet","dataset":"imagenet"}' \
  -H "Content-Type: application/json" -X POST http://localhost:12000/trigger

👀Visualizing the Pipeline

Patch the argo-server for local access:

kubectl patch deployment argo-server --namespace argo --type='json' \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/args", "value": ["server","--auth-mode=server"]}]'

Then:

kubectl port-forward svc/argo-server -n argo 2746:2746

Navigate to https://localhost:2746 to visualise your pipeline. You should see the following configuration

Event Flow

Event Sources

Sensor

Workflow Templates

Workflow Runs

🌐Real-World MLOps Use Cases

This is where I see Argo Workflows + Events fitting into real ML pipelines:

  1. Event-Driven Model Training
  2. Continuous Model Evaluation (CME)
  3. ETL for ML Pipelines
  4. Model Deployment Automation

📚Further Reading & Resources

🚀 What’s Next

Argo Workflows and Argo Events have opened the door to scalable, event-driven ML pipelines, but there’s much more to explore:

📦 GitOps Delivery with ArgoCD

Pairing ArgoCD with Argo Workflows would enable declarative, version-controlled deployment of ML pipelines across environments. Imagine triggering new workflow templates from a Git commit and syncing changes automatically.

📡 Real-World Event Sources

How about connecting Argo Events to cloud-native services such as:

  • AWS SQS / SNS
  • Azure Service Bus
  • etc.

These integrations could allow upstream events to dynamically trigger ML pipelines.

🔧 Tool Integrations Ahead

What about adding further integration with popular tools:

  • MLflow – for experiment tracking and lifecycle management
  • KServe – to enable model serving within a Kubernetes-native stack
  • etc.

I hope this post sparked some ideas, go give Argo a spin and explore your own event-driven ML workflows!

Azure Pipelines, DevOps

Dynamic Multistage Azure Pipelines Part 1

Posted on by Mark Johnson

In a previous post I looked at multistage YAML pipelines. In this post I am going to look at dynamic multistage YAML pipelines.

What do I mean by dynamic multistage? What I mean is running multiple stages but all of the configuration is loaded dynamically from one or more sources e.g. parameters, variable templates, variable groups, etc..

Why?

What problem am I trying to solve with this? Firstly, reduce duplication, in a lot of cases the difference between dev and prod is just the configuration. Secondly, provide the ground work to get a base setup so that I can concentrate on what steps are needed in the pipeline and not worry about the environments.

Anything else? Well, I often have multiple projects that all need to deploy to the same set of environments, it would be good to share the configuration for that as well between projects.

Next Steps

Ok, I need a pipeline, lets start with something simple, a pipeline with an initial build stage and then multiple deployment stages defined by a parameter:

trigger: none 
pr: none 

pool:  
  vmImage: 'ubuntu-latest' 

parameters:
- name: stages
  type: object
  default:
    - 'dev'
    - 'prod'

stages:
- stage: build
  displayName: 'Build/Package Code or IaC'  
  jobs:  
  - job: build
    displayName: 'Build/Package Code'
    steps:
    # Steps to perform the build and/or package of code or IaC

- ${{ each stage in parameters.stages }}:
  - stage: ${{ stage }}
    displayName: 'Deploy to ${{ stage }}'
    jobs:
    - deployment: deploy_${{ stage }}
      displayName: 'Deploy app to ${{ stage }}'
      environment: ${{ stage }}
      strategy:
        runOnce:
          deploy:
            steps:
            # Steps to perform the deployment

This very small example achieves configuring multiple deployment stages, adding another stage to this would be very easy to do, just update the parameter to include a new stage name.

Now we have the basic configuration lets add loading of a variable group. This could be done by using dynamic naming or by changing the stages parameter.

I have a variable group for each environment, groupvars_dev, groupvars_prod with a single variable mygroupvar.

Dynamic Naming

I’ll add the variable group to the variables at the Stage level (this could also be done at the job level) and include the stage name dynamically.

- ${{ each stage in parameters.stages }}:
  - stage: ${{ stage }}
    displayName: 'Deploy to ${{ stage }}'
    variables:
      - group: groupvars_${{ stage }}
    jobs:
    - deployment: deploy_${{ stage }}
      displayName: 'Deploy app to ${{ stage }}'
      environment: ${{ stage }}
      strategy:
        runOnce:
          deploy:
            steps:
            - bash: |
                echo '$(mygroupvar)'
              displayName: 'Deploy Steps'

Parameter Change

Another way to define the dynamic group is to update the parameter object to provide additional configuration e.g.

parameters:
- name: stages
  type: object
  default:
    - name: 'dev'
      group: 'groupvars_dev'
    - name: 'prod'
      group: 'groupvars_prod'

   ...

- ${{ each stage in parameters.stages }}:
  - stage: ${{ stage.name }}
    displayName: 'Deploy to ${{ stage.name }}'
    variables:
      - group: ${{ stage.group }}
    jobs:
    - deployment: deploy_${{ stage.name }}
      displayName: 'Deploy app to ${{ stage.name }}'
      environment: ${{ stage.name }}
      strategy:
        runOnce:
          deploy:
            steps:
            - bash: |
                echo '$(mygroupvar)'
              displayName: 'Deploy Steps'

Both ways of adding the variable group dynamically achieved the same goal and loaded in the expected group when each stage ran.

Variable Templates

Variable groups are not the only way to dynamically load variables, you could also use variable templates, lets say I have variable templates for each environment, vars_dev.yml and vars_prod.yml

Using dynamic naming you can load the variables like this:

- ${{ each stage in parameters.stages }}:
  - stage: ${{ stage }}
    displayName: 'Deploy to ${{ stage }}'
    variables:
      - template: vars_${{ stage }}.yml
    jobs:
    - deployment: deploy_${{ stage }}
      displayName: 'Deploy app to ${{ stage }}'
      environment: ${{ stage }}
      strategy:
        runOnce:
          deploy:
            steps:
            - bash: |
                echo '$(myfilevar)'
              displayName: 'Deploy Steps'

Now with variable files and groups being added, updating to add a new stage becomes a little more complex as I would need to add those as well.

Shared Template

Now I have a dynamic multistage pipeline, how can I create a template to share with other projects?

Before I answer that I should say that I usually use a separate repository for shared templates that way I can version them. I covered this is a previous post if you want some more information.

Ok, on to the how, based on the above scenario wouldn’t it be great to have a really simple pipeline that concentrated on just the steps, like this?

trigger: none
pr: none

pool: 
  vmImage: 'ubuntu-latest'

resources:
  repositories:
    - repository: templates
      type: git
      name: shared-templates
      ref: main

extends:
  template: environments.yml@templates
  parameters:
    variableFilePrefix: 'vars'
    buildSteps:
        # Steps to perform the build and/or package of code or IaC
    releaseSteps:
       # Steps to perform the deployment

This could be your boilerplate code for multiple projects extending from a base template. You might be asking but how do I create such a template?

Lets convert what we started with into a template a bit at a time.

Firstly create a new file e.g. environments.yml to be the base template and add the parameters that make up the stage configuration

parameters:
- name: stages
  type: object
  default:
    - 'dev'
    - 'prod'

Next, add the build stage up to the steps

stages:
- stage: build
  displayName: 'Build/Package Code or IaC'  
  jobs:  
  - job: build
    displayName: 'Build/Package Code'
    steps:

At this point we need to be able to pass in the build steps, using the Azure Pipeline built-in type stepList we can add a parameter ‘buildSteps’:

parameters:
- name: stages
  type: object
  default:
    - 'dev'
    - 'prod'
- name: buildSteps  
  type: stepList  
  default: []

stages:
- stage: build
  displayName: 'Build/Package Code or IaC'  
  jobs:  
  - job: build
    displayName: 'Build/Package Code'
    steps: ${{ parameters.buildSteps }}

Next, add the dynamic stages up to the steps

- ${{ each stage in parameters.stages }}:
  - stage: ${{ stage }}
    displayName: 'Deploy to ${{ stage }}'
    jobs:
    - deployment: deploy_${{ stage }}
      displayName: 'Deploy app to ${{ stage }}'
      environment: ${{ stage }}
      strategy:
        runOnce:
          deploy:
            steps:

And then as before, add a stepList for the release steps

parameters:
- name: stages
  type: object
  default:
    - 'dev'
    - 'prod'
- name: buildSteps  
  type: stepList  
  default: []
- name: releaseSteps  
  type: stepList  
  default: []

stages:
- stage: build
  displayName: 'Build/Package Code or IaC'  
  jobs:  
  - job: build
    displayName: 'Build/Package Code'
    # Steps to perform the build and/or package of code or IaC
    steps: ${{ parameters.buildSteps }}

- ${{ each stage in parameters.stages }}:
  - stage: ${{ stage }}
    displayName: 'Deploy to ${{ stage }}'
    variables:
      - template: vars_${{ stage }}.yml
    jobs:
    - deployment: deploy_${{ stage }}
      displayName: 'Deploy app to ${{ stage }}'
      environment: ${{ stage }}
      strategy:
        runOnce:
          deploy:
            steps: ${{ parameters.releaseSteps }}

The next part is adding support for variable groups and/or templates. This can be achieved by the addition of 2 parameters for the name prefixes e.g.

- name: variableGroupPrefix  
  type: string  
  default: ''  
- name: variableFilePrefix  
  type: string  
  default: ''

There will also need to a be check to only load the group and/or file if the parameter is not empty ”.

parameters:
- name: stages
  type: object
  default:
    - 'dev'
    - 'prod'
- name: buildSteps  
  type: stepList  
  default: []
- name: releaseSteps  
  type: stepList  
  default: []
- name: variableGroupPrefix  
  type: string  
  default: ''  
- name: variableFilePrefix  
  type: string  
  default: ''

stages:
- stage: build
  displayName: 'Build/Package Code or IaC'  
  jobs:  
  - job: build
    displayName: 'Build/Package Code'
    # Steps to perform the build and/or package of code or IaC
    steps: ${{ parameters.buildSteps }}

- ${{ each stage in parameters.stages }}:
  - stage: ${{ stage }}
    displayName: 'Deploy to ${{ stage }}'
    variables:
      - ${{ if ne(parameters.variableGroupPrefix, '') }}:
        - group: ${{ parameters.variableGroupPrefix }}_${{ stage }}
      - ${{ if ne(parameters.variableFilePrefix, '') }}:
        - template: ${{ parameters.variableFilePrefix }}_${{ stage }}.yml
    jobs:
    - deployment: deploy_${{ stage }}
      displayName: 'Deploy app to ${{ stage }}'
      environment: ${{ stage }}
      strategy:
        runOnce:
          deploy:
            steps: ${{ parameters.releaseSteps }}

Note: If I was running this template from the same repository, loading of the variable file would be fine but when it’s in a separate repository there needs to be a slight adjustment to add @self on the end so it will load from the calling repository instead of the remote repository.

- template: ${{ parameters.variableFilePrefix }}_${{ stage }}.yml@self

And that is it, one base template that handles the desired configuration and ready for reuse.

Expanding the Concept

Lets say you had a requirement to deploy multiple projects IaC (Infrastructure as Code) and applications to multiple subscriptions and multiple regions in your Azure Estate. How nice would it be to be able to define that in a central configuration. Here is one possible configuration for such a requirement

parameters:
- name: environments
  type: object
  default:
  - name: 'dev'
    subscriptions:
      - subscription: 'Dev Subscription'
        regions:
          - location: 'westus'
            locationShort: 'wus'
  - name: 'prod'
    subscriptions:
      - subscription: 'Prod Subscription'
        regions:
          - location: 'eastus'
            locationShort: 'eus'
          - location: 'westus'
            locationShort: 'wus'
- name: buildSteps
  type: stepList
  default: []
- name: releaseSteps
  type: stepList
  default: []
- name: customReleaseTemplate
  type: string
  default: ''
- name: variableGroupPrefix
  type: string
  default: ''
- name: variableFilePrefix
  type: string
  default: ''

stages:
- stage: build
  displayName: 'Build/Package Code or IaC'
  jobs:
  - job: build
    displayName: 'Build/Package Code'
    steps: ${{ parameters.buildSteps }}

- ${{ each env in parameters.environments }}:
  - stage: ${{ env.name }}
    displayName: 'Deploy to ${{ env.name }}'
    condition: succeeded()
    variables:
      - ${{ if ne(parameters.variableFilePrefix, '') }}:
        - template: ${{ parameters.variableFilePrefix }}_${{ env.name }}.yml@self
      - ${{ if ne(parameters.variableGroupPrefix, '') }}:
        - group: ${{ parameters.variableGroupPrefix }}_${{ env.name }}
    jobs:
    - ${{ each sub in env.subscriptions }}:
      - ${{ each region in sub.regions }}:
        - ${{ if ne(parameters.customReleaseTemplate, '') }}:
          - template: ${{ parameters.customReleaseTemplate }}
            parameters:
               env: ${{ env.name }}
               location: ${{ region.location }}
               locationShort: ${{ region.locationShort }}
               subscription: ${{ sub.subscription }}
        - ${{ else }}:
          - deployment: deploy_${{ region.locationShort }}
            displayName: 'Deploy app to ${{ env.name }} in ${{ region.location }}'
            environment: ${{ env.name }}_${{ region.locationShort }}
            strategy:
              runOnce:
                deploy:
                  steps:
                  - ${{ parameters.releaseSteps }}

You may notice with this configuration there is an option for a custom release template where you could override the job(s) required, you would just need to make sure the template included the parameters supplied from the base template:

parameters:
- name: env
  type: string
- name: location
  type: string
- name: locationShort
  type: string
- name: subscription
  type: string

Then you can add the custom jobs for a given project.

Final Thoughts

Shared templates are so powerful to use and combined with the often forgotten about built-in types step, stepList, job, jobList, deployment, deploymentList, stage and stageList, really allows for some interesting templates to be created.

For additional information see the Azure Pipelines Parameters docs.

You are no doubt thinking, this all sounds very good but what about real application of such a template? In the next post I will use this last template to deploy some Infrastructure as Code to Azure and then deploy an application into that infrastructure to show real usage.

1