Argo Workflows

Last reviewed Aug 26, 2026 · 828 words

There are two Argos, and people install the wrong one weekly. Argo CD watches a git repository and makes a cluster match it. Argo Workflows, the subject here, runs jobs: it takes a graph of steps, launches each one as a Kubernetes pod, passes files between them and reports the result. It has nothing to do with deployment and only a passing relationship with CI. What it is superb at is parallel batch work on a cluster you already run: data pipelines, model training, nightly processing of 10,000 files where each file is its own pod. If that is not your problem, the rest of this page will save you an afternoon.

Every step is a pod, which is the strength and the tax

A workflow is a Kubernetes custom resource. The controller reads it, creates a pod for each step with the container image you named, waits for it, and moves on according to the dependency graph. The strength: steps scale to whatever the cluster has, any container image is a valid step with no plugin system to learn, and a workflow with 500 parallel tasks is just 500 pods. The tax: every step pays pod scheduling and image pull latency, so a 6-step workflow of one-second shell commands takes a minute, and everything you write is YAML. The catalogue's Hard rating and 1 GB minimum are fair; the controller itself is small, but it presumes a working cluster with storage, RBAC and an ingress, and that presumption is most of the difficulty.

A first workflow, and the YAML that follows

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: nightly-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: fetch
            template: sh
            arguments: { parameters: [{ name: cmd, value: "echo fetching" }] }
          - name: process
            depends: fetch
            template: sh
            arguments: { parameters: [{ name: cmd, value: "echo processing" }] }
    - name: sh
      inputs:
        parameters: [{ name: cmd }]
      container:
        image: alpine:3
        command: [sh, -c]
        args: ["{{inputs.parameters.cmd}}"]

Submit it with argo submit --watch nightly.yaml and the CLI draws the graph as it runs; the web UI on port 2746 shows the same with logs per step. The pattern scales in a predictable direction: WorkflowTemplate for reusable pieces, CronWorkflow for schedules, withItems and withParam for fan-out over a list, and retryStrategy on anything that talks to the network. A real pipeline of 15 steps with retries, artifacts and parameters runs 300 to 500 lines, and the catalogue's "YAML gets verbose" is if anything polite.

Artifacts need an S3 bucket, so bring MinIO

Steps do not share a filesystem. Anything one step produces for another travels as an artifact, and artifacts live in an object store configured in the workflow-controller-configmap: an S3 endpoint, bucket, and credentials secret. On a homelab that means running MinIO in the cluster and pointing Argo at it with insecure: true and the service address, which is 20 lines of config and a common first-hour stumble. Logs for finished pods go there too if you enable archiving, and without it they vanish when Kubernetes garbage-collects the pod. Install itself is kubectl apply of the release manifest into an argo namespace, or the argo/argo-workflows Helm chart; k3s on a couple of nodes is enough.

Against Airflow, and against CI servers

Apache Airflow defines pipelines in Python with a scheduler, a metadata database and a fixed pool of workers, and shines when the logic is in the code and the tasks are many but small. Argo pushes all of that onto Kubernetes: no database of its own, no worker pool, steps as containers. Pick Argo when tasks are heavyweight and containerised and you already operate a cluster; pick Airflow when your team thinks in Python and does not want to learn pod specs. Against CI, the comparison is harsher. Woodpecker CI or Gitea Actions give you git-triggered builds with a YAML file in the repo and a status check on the pull request in 10 minutes. Doing the same with Argo means adding Argo Events for webhooks, writing the build as a workflow and wiring the commit status yourself. It works, and larger platform teams do it, but the CI/CD category has tools that do it out of the box.

What I'd do

Install Argo Workflows if you have a Kubernetes cluster and a job that is genuinely parallel: processing a photo library, running an ML training sweep, re-encoding a media collection. Start with MinIO as the artifact store, write one WorkflowTemplate for your common container, and use CronWorkflow instead of host cron for anything that should run inside the cluster. For CI on a homelab forge, skip it and run Woodpecker. And if what you actually wanted was "deploy my manifests from git", close this tab and install Argo CD, which is a smaller, calmer project that happens to share a name.