Published on April 16, 2025
This is the third part of our series on Cluster API and how it can be a solution for managing large numbers of clusters at scale. For the previous parts on this series, see part 1 and part 2.
Table of Contents
In our previous posts, we looked at how Cluster API (CAPI) works, how we can use an infrastructure provider (CAPA) and bootstrap our own management cluster and its own managed clusters. Great, we have a working cluster… but a cluster with no workloads is not very useful. We desperately want to deploy to our clusters, but managing workloads across multiple Kubernetes clusters presents several challenges:
- Consistency - Ensuring all clusters run identical workloads
- Drift prevention - Maintaining synchronized configurations over time
- Scalability - Adding new clusters without increasing operational overhead
- Maintenance - Performing updates across the fleet efficiently
For this part, we’ll be looking at how we get our workloads in these clusters, and how we can solve these problems trying to manage multiple clusters all running the same workloads.
Installing workloads with ClusterResourceSet
ClusterResourceSet (CRS) is a CAPI extension that allows you to define Kubernetes resources as YAML within ConfigMaps and Secrets and automatically apply them to matching clusters. It’s a simple and straightforward way to deploy the same resources to multiple clusters. Here’s an example of how one would use a ClusterResourceSet to deploy a basic monitoring stack:
apiVersion: addons.cluster.x-k8s.io/v1alpha3
kind: ClusterResourceSet
metadata:
name: monitoring-stack
namespace: capi-system
spec:
strategy: Reconcile
clusterSelector:
matchLabels:
environment: production
cloud: openstack
resources:
- kind: ConfigMap
name: prometheus-yaml
- kind: ConfigMap
name: grafana-yaml
- kind: Secret
name: monitoring-certs
---
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-yaml
namespace: capi-system
data:
prometheus.yaml: |
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
spec:
containers:
- name: prometheus
image: prom/prometheus:v2.36.0
ports:
- containerPort: 9090
---
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-yaml
namespace: capi-system
data:
grafana.yaml: |
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
containers:
- name: grafana
image: grafana/grafana:9.0.0
ports:
- containerPort: 3000
One thing to note is that if a Secret is being used to hold configuration that’s to be installed, it must have the type addons.cluster.x-k8s.io/resource-set set for CAPI to reconcile the YAML, otherwise it will be ignored by the controller.
A ClusterResourceSet is mostly used to deploy resources on clusters for bootstrapping purposes. As an example, when the managed cluster is created, depending on the cluster type, it might be missing crucial cluster components such as networking via a CNI. For an EKS cluster, this is not an issue as they are pre-configured with the Amazon VPC CNI, but for CAPI-managed Kubeadm control planes, a CNI must be present before any workload runs in the cluster. For this purpose, a CRS can be defined to allow for bootstrapping necessary resources on a cluster.
By default, CAPI only applies the resource once and does not delete the resource if the ClusterResourceSet has been removed. However, it does have limited support for reconciliation via the spec.Strategy field, so it can keep a resource updated whenever its manifest is modified. However, there’s no drift detection and managing complex applications is unfeasible with this approach, and therefore this is not a recommended approach for installing and managing general-purpose workloads.
Declarative Configurations with ArgoCD
If a CRS is not the solution for deploying to our fleet of clusters, then what is? This is where a good continuous delivery (CD) pipeline comes into play. Once the cluster is bootstrapped with its CNI and is ready to accept workloads, a tool for automating a CD pipeline, such as Flux or ArgoCD, can perform the rest of the work of installing all the applications that must run on all the managed clusters. One benefit of using a GitOps-y tool like the aforementioned ones is that the desired cluster state can be expressed with a series of manifests in a Git repository, which provides an auditable, version-controlled deployment flow for all the clusters at the same time. For this particular example, we’ll be using ArgoCD and the “app of apps” pattern to install applications declaratively: a single ArgoCD application is installed on each cluster “manually” (i.e. with some method outside of ArgoCD) which contains the information about the other applications with the real workloads that need to be installed.
Alternative 1: ArgoCD running on the management cluster
This is a C&C (command and control) setup, where you have a single source of control for all the applications that get installed in the cluster. ArgoCD is installed on the management cluster, and every time a new managed cluster is created, the details for that cluster (API server endpoint, authentication) are added to ArgoCD so that it can then install all the applications on that cluster.
Note: the examples here are demonstrative, since the Application/Project layout will differ depending on the nature and organization of your workloads.
- Install ArgoCD on the management cluster.
- Register your clusters in ArgoCD. Alternatively, use a tool like SuperOrbital’s capargo to manage adding new clusters in ArgoCD for you.
- Create an Application manifest for your workload:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: standard-workload
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/your-org/standard-workloads
targetRevision: main
path: base
destination:
server: https://kubernetes.default.svc
namespace: workloads
syncPolicy:
automated:
prune: true
selfHeal: true
Once this is done, you can use an ApplicationSet to deploy that workload to multiple clusters, using the cluster generator field to automatically generate the cluster information based on the clusters that have already been registered with Argo CD. As an example:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: fleet-workload
namespace: argocd
spec:
generators:
- clusters:
selector:
matchLabels:
environment: production
template:
metadata:
name: '{{name}}-standard-workload'
spec:
project: default
source:
repoURL: https://github.com/your-org/standard-workloads
targetRevision: main
path: overlays/{{metadata.labels.region}}
destination:
server: '{{server}}'
namespace: workloads
syncPolicy:
automated:
prune: true
selfHeal: true
Of course, with every approach there are a few pros and cons:
Pros:
- Single-pane of glass overview of the state of all the applications on all clusters.
- ArgoCD is physically segregated from each cluster that it manages, which improves the security standing.
- The Cluster Generator feature becomes available for all the registered clusters, which allow for targeting specific clusters for a given Application via a label selector.
Cons:
- Only one ArgoCD in the management cluster. This all but ensures that ArgoCD must be installed in HA mode to ensure little downtime during cluster downtime or maintenance.
- Without the use of capargo, adding a new cluster to ArgoCD requires manual intervention to create a special secret with the API server endpoint and the kubeconfig credentials for ArgoCD to authenticate and access the workloads on the cluster.
- Depending on the number of clusters being managed, it could require a lot of resources to operate ArgoCD on the management cluster.
Fortunately, this is not the only approach we can take. What if instead we ran ArgoCD everywhere?
Alternative 2: ArgoCD running on all managed clusters
This is a distributed setup where ArgoCD is installed on each managed cluster and preconfigured with the appropriate bootstrap application to install all the workloads needed on each cluster. The central ArgoCD on the management cluster is only tasked with ensuring that all the other ArgoCDs are kept up-to-date and have the “app-of-apps” configuration updated. In this situation, we can leverage CAPI’s ClusterResourceSet to bootstrap ArgoCD on every cluster with the “app of apps” configuration.
Note: the YAML for the base ArgoCD install is quite large, so for the sake of brevity it is omitted from the example below.
apiVersion: addons.cluster.x-k8s.io/v1alpha3
kind: ClusterResourceSet
metadata:
name: argocd-bootstrap
namespace: capi-system
spec:
clusterSelector:
matchLabels:
argocd: enabled
resources:
- kind: ConfigMap
name: argocd-installer
---
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-installer
namespace: capi-system
data:
argocd.yaml: |
apiVersion: v1
kind: Namespace
metadata:
name: argocd
---
# ArgoCD YAML would be here...
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: cluster-workloads
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/your-org/standard-workloads
targetRevision: main
path: overlays/{{metadata.labels.region}}
destination:
server: https://kubernetes.default.svc
namespace: workloads
syncPolicy:
automated:
prune: true
selfHeal: true
With this setup, you can configure a “meta” ArgoCD instance to manage the ArgoCDs (but not their workloads) on the management cluster:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: argocd-fleet-manager
namespace: argocd
spec:
generators:
- clusters:
selector:
matchLabels:
argocd: enabled
template:
metadata:
name: '{{name}}-argocd-config'
spec:
project: default
source:
repoURL: https://github.com/your-org/argocd-fleet-config
targetRevision: main
path: clusters/{{name}}
destination:
server: '{{server}}'
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true
Alas, this is not a perfect solution either, so whichever approach you choose should fit best your specific use-case. Let’s review the pros and cons for running ArgoCD everywhere:
Pros:
- A simpler configuration on the management cluster, as the single ArgoCD on the management cluster does not need to maintain the workloads for the entire cluster fleet.
- Easier installation and maintenance process since each cluster installs its own ArgoCD, and it can come pre-baked with a directive to install workloads on itself. This also means that each ArgoCD itself consumes fewer resources as it only cares about a single cluster.
- As new versions and configurations of ArgoCD are developed and deployed, it’s much easier to roll out these changes to a few clusters at a time and ensure that no bugs are introduced.
Cons:
- With ArgoCD, since a Redis server contains caches of the manifests deployed in the cluster, it can become a security issue if access to the ArgoCD namespace within the cluster is not properly locked down.
- More difficulty in controlling the synchronization of workloads as they are deployed in different clusters.
- If each instance running on each cluster needs to be exposed via an Ingress, it could potentially incur more costs for each load balancer.
- No “single pane of glass” capability for a quick overview of the state of the world.
What’s next?
CAPI is a great cloud-agnostic tool to build a platform where you can go from zero infrastructure to a fleet of homogeneous, production-ready clusters with a bit of initial configuration and some manifests.
However, there are times when a company has different clusters for different use cases. Maybe some clusters only host CI/CD pipeline tools and their runners; others run specialized ML workloads. When the types of deployed clusters start to exceed the dozens, you’ll soon realize that specifying the resources for each cluster becomes an exercise in creating a lot of boilerplate code.
In a future article, we’ll get into ClusterClass objects and how they can help you simplify the management of hundreds of different kinds of clusters and help you lose the fear of the single unique snowflake clusters that cannot be recreated!
Subscribe (yes, we still ❤️ RSS) or join our mailing list below to stay updated!
hello@superorbital.io
855.SUPERORBITAL
LinkedIn
Github
We still ❤️ RSS
Contact us with security concerns