What is the Argo CD app of apps pattern and how does it work?
The app of apps pattern uses one Argo CD Application, the parent, whose source path contains other Application manifests instead of Deployments or Services. When Argo CD syncs the parent, it applies those manifests to the cluster, usually into the argocd namespace. Each child then reconciles its own source, such as a Helm chart, a Kustomize overlay or a plain directory for one service. To add a service, you commit a new child manifest. You do not need to run argocd app create by hand.
Propagation is narrower than it first appears. Syncing the parent only creates or updates the child Application objects. It does not deploy their workloads unless each child has its own automated sync policy. Health is similar: by default, current Argo CD does not aggregate a child's health into the parent. To get that, you add a custom health check for argoproj.io/Application.
Pruning does propagate, and that is where the risk sits. If a child manifest disappears from the parent's path and the parent prunes, Argo CD deletes that child Application. If the child carries the resources-finalizer.argocd.argoproj.io finalizer, Argo CD first deletes everything that child manages and only then removes the child.
A few mistakes can therefore cascade into every workload you run:
- deleting the parent with cascade enabled
- pointing it at the wrong path or revision
- a bad merge that empties the directory
Automated sync refuses to prune down to zero resources unless allowEmpty is set, but a partial wipe still goes through.
The pattern fits two groups. Platform teams use it to bootstrap a cluster from a single applied manifest. Backend teams use it to keep dozens of services declared in Git rather than in the UI.
# Parent: applied once by hand; its path holds only child Application manifests.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root
namespace: argocd
spec:
project: default
source:
repoURL:
targetRevision: main # a wrong branch here re-targets every child
path: apps # an emptied or wrong path prunes children
destination:
server:
namespace: argocd # child Application objects live here
syncPolicy:
automated:
prune: true # removing a file in apps/ deletes that child
allowEmpty: false # refuse to prune down to zero children
---
# Child: apps/orders.yaml, one per service.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: orders
namespace: argocd
finalizers:
# Delete this child's workloads before the child itself is removed.
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL:
targetRevision: main
path: services/orders/overlays/prod
destination:
server:
namespace: orders
syncPolicy:
automated: {} # without this, the parent's sync never deploys workloads
A parent Application that manages a directory of child Application manifests, and one child whose resources-finalizer causes its workloads to be deleted along with it when the parent prunes it.
How should you structure a GitOps repository for many backend services?
Split the repository by who owns a change and how often it happens. A layout that holds up well has three top-level areas. bootstrap/ contains the root Application and the per-cluster entry points. platform/ holds cluster add-ons such as ingress, cert management and monitoring. services/ gets one directory per backend service, each with a base/ and an overlays/ directory that has one folder per environment (dev, staging, prod). Helm users can use a single chart with values-dev.yaml, values-prod.yaml and so on. Either way, every environment difference becomes a small, reviewable file in the same commit as everything else.
Avoid one branch per environment. Hotfixes land on prod and never make it back, and cherry-picks pile up. Soon nobody can tell from a diff whether two environments differ on purpose. Directories on a single branch make each difference visible and keep promotion an ordinary pull request.
Overlays carry their own risk. Every patch is small, but they add up. After a year of one-off tweaks, a prod overlay can replace most of the base, and you end up with divergent copies of the same config. Keep overlays limited to replicas, resources, endpoints and image tags. When an overlay grows past that, move the change into the base or into a parameter.
| Model | Review load | Access control | Cross-cutting changes | Reconciliation load |
|---|---|---|---|---|
| Single config monorepo | High, concentrated on platform reviewers | Path-based via CODEOWNERS; coarse repo permissions | One PR | Every commit triggers refresh checks for all Applications watching the repo |
| Repo per team | Spread across teams | Clean per-team boundaries | Many coordinated PRs | Isolated per repo |
| Config beside service code | Lowest; reviewed with the feature | Inherits app repo permissions | Hardest; touches every service repo | Noisy; code-only commits also trigger refreshes |
# Repository layout (one branch, environments as directories):
# bootstrap/root-app.yaml # root Application, per-cluster entry points
# platform/ingress/ cert-manager/ monitoring/
# services/orders/base/ # Deployment, Service, shared config
# services/orders/overlays/dev/
# services/orders/overlays/staging/
# services/orders/overlays/prod/ # this file
#
# services/orders/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Everything except the deliberate prod differences comes from base.
resources:
- ../../base
# Allowed difference: replica count.
replicas:
- name: orders
count: 4
# Allowed difference: image tag. Promotion is a PR that changes this line.
images:
- name: registry.example.com/orders
newTag: "1.14.2"
# Allowed differences: resources and endpoints, in one small patch file.
# If this list keeps growing, move the change into base or a parameter
# before the overlay turns into a second copy of the service.
patches:
- path: resources-and-endpoints.yaml
target:
kind: Deployment
name: orders
A prod Kustomize overlay for one service that inherits everything from base and overrides only replicas, the image tag, and a single patch for resources and endpoints, with the repository layout shown in the header comments.
When should you replace hand-written child Applications with an ApplicationSet?
Hand-written child Applications work while the list is short and stable. They stop scaling when most of your PRs are copies of an existing child manifest with three fields changed, or when an audit finds children whose targetRevision or sync options have quietly drifted apart. At that point, generate the children instead of writing them.
An ApplicationSet pairs a generator, which produces parameters, with a template that turns each parameter set into an Application. For service fan-out, the Git generators matter most:
| Generator | Produces one Application per… | Typical use |
|---|---|---|
| Git directory | Matching folder in a repo | services/* becomes one app per service |
| Git file | Matching config file, with its fields as parameters | Per-service settings such as namespace, team or chart path |
| List | Hard-coded element | A small, fixed set of environments |
| Cluster | Cluster registered in Argo CD, filterable by label | Deploy to every cluster labelled env=staging |
| Matrix | Combination of two generators | Every service × every matching cluster |
With a directory or file generator, onboarding a service becomes a single PR that adds a folder or config file. Nobody has to edit the parent app or copy and adjust a child manifest, and reviewers check the service's own config instead of boilerplate.
The cost is indirection. Copy-paste becomes a template, and every per-service exception adds conditional fields or extra generator parameters until the template is harder to read than the manifests it replaced. When a rendered Application looks wrong, you can't open a file to see why. You have to work out which generator emitted which parameters and how the template combined them. If more than a handful of services need special handling, give them their own ApplicationSet or a hand-written child.
# One ApplicationSet replaces N hand-written child Applications.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: services-staging
namespace: argocd
spec:
goTemplate: true
# Fail rendering on a typo'd parameter instead of emitting an empty string
goTemplateOptions: ["missingkey=error"]
generators:
# Matrix: every service folder x every matching cluster
- matrix:
generators:
# Onboarding = add a folder under services/
- git:
repoURL:
revision: main
directories:
- path: services/*
# Fan out to every registered cluster labelled env=staging
- clusters:
selector:
matchLabels:
env: staging
template:
metadata:
# e.g. payments-staging-eu1: one Application per service per cluster
name: '{{.path.basename}}-{{.name}}'
spec:
project: default
source:
repoURL:
targetRevision: main # set once here, so children cannot drift apart
path: '{{.path.path}}'
destination:
server: '{{.server}}'
namespace: '{{.path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
An ApplicationSet whose matrix generator combines a Git directory generator over services/ with a cluster generator filtered by env=staging, so adding a service folder creates one Application on every staging cluster.*
Combining app of apps and ApplicationSets: one entry point per cluster
The structure that holds up at scale gives each cluster one root Application. That root is an app of apps, but its children are not individual services. It installs three kinds of objects: the AppProjects, the platform add-ons the cluster needs (ingress controller, cert management, monitoring agents), and the ApplicationSets. The ApplicationSets then generate one Application per service. Anyone asking "what runs on this cluster?" has a single place to start, and adding a service means adding a directory or a config entry, not editing the root.
AppProjects as the blast-radius boundary
Each team's ApplicationSet template should set spec.project to that team's AppProject, never to default. The AppProject is what Argo CD enforces at sync time, so it catches mistakes that code review misses:
sourceReposlimits which repositories the team's Applications can pull from.destinationspins the allowed clusters and namespaces, for examplepayments-*on the production cluster only.clusterResourceWhitelistis left empty for service teams, so they cannot create ClusterRoles, CRDs or Namespaces.
With these limits in place, a template typo that renders the wrong namespace fails to sync instead of overwriting another team's resources. The same applies to a pull request that adds a ClusterRoleBinding. The root Application and the add-ons belong in a separate, more privileged project that only the platform team can change.
Keep the tree shallow
Stop at two levels: root, then ApplicationSets and add-ons. Every extra layer of Applications that generate Applications makes a failed sync harder to trace. An engineer has to follow a chain of parent objects across directories to find which file produced a resource. New team members also have to learn that chain before they can make a safe change.
# AppProject: Argo CD enforces these limits at sync time for every
# Application that sets spec.project: payments
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: payments
namespace: argocd
spec:
description: Payments team services
sourceRepos:
- # only this repo
destinations:
- name: prod-eu-1 # only the production cluster...
namespace: 'payments-*' # ...and only the team's namespaces
clusterResourceWhitelist: [] # no ClusterRoles, CRDs or Namespaces
---
# ApplicationSet: one Application per directory under services/
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: payments-services
namespace: argocd
spec:
goTemplate: true
generators:
- git:
repoURL:
revision: main
directories:
- path: services/*
template:
metadata:
name: 'payments-{{.path.basename}}'
spec:
project: payments # never 'default'
source:
repoURL:
targetRevision: main
path: '{{.path.path}}'
destination:
name: prod-eu-1
namespace: 'payments-{{.path.basename}}'
A team AppProject that restricts source repos, destinations and cluster-scoped resources, and the ApplicationSet whose generated per-service Applications are bound to it, so a mis-rendered namespace or a stray ClusterRoleBinding fails to sync.
How do you handle ordering, dependencies and promotion between environments?
Ordering with sync waves
When a parent Application syncs its children, you can order them with the argocd.argoproj.io/sync-wave annotation on each child Application. Put CRDs in the lowest wave, operators and controllers next, shared infrastructure such as ingress and certificate management after that, and services last. Argo CD applies lower waves first, so a service's custom resources never reach a cluster that lacks their definitions. Waves wait on child health only if your installation assesses health for Application resources, so confirm that setting before you rely on the ordering.
Keep waves for cluster-level prerequisites. Do not use them to start the orders service after the payments service. That dependency also applies during restarts, node drains and partial outages, and Argo CD plays no part in any of those. Readiness probes, connection retries with backoff, and tolerance for unavailable peers handle it in every case, and they keep your wave numbers short and stable.
Promotion through pull requests
Promotion becomes a small diff: change an image tag or chart version in the dev overlay, merge, verify, then open the same change for staging and then production. A person can open these pull requests, a CI job can open them after tests pass, or an image-update tool can write the change to dev automatically. In each case Git records who promoted what and when, and a revert is one commit.
This is slower than CI deploying straight to a cluster, because each environment adds a review and a sync cycle. Releases that span several services also need a convention, such as one pull request that bumps every affected service or a shared release identifier, so the changes move between environments together.
What guardrails prevent cascading deletes and incident-time conflicts?
Removing an entry from Git should never be enough to delete a production service. Keep automated pruning off on the parent Application, or require review on the paths it watches. Otherwise a bad merge removes child Applications, and their finalizers then delete the workloads.
Set deletion behavior deliberately:
- Child finalizers. Add the resources finalizer only where cascading deletion is what you want.
- ApplicationSet deletion. Set
preserveResourcesOnDeletionon ApplicationSets that manage long-lived services. - ApplicationSet sync policy. Consider
create-onlyorcreate-updatepolicies so a generator change cannot delete Applications. - Pre-merge review. Render generator output in CI and diff the resulting Application list before merging. A typo in a directory glob looks harmless in YAML but shows up immediately in a diff.
Incident procedure
Agree on the steps before you need them:
- Disable automated sync and self-heal on the parent first, then on the affected child. If you only pause the child, the parent restores its sync policy on the next reconcile.
- Apply the hotfix and record the exact change in the incident ticket.
- Open a pull request that reproduces the change in Git.
- Re-enable sync only once the live state and Git show no diff.
Load and rate limits
Many Applications add controller and repo-server work, and polling adds load on your Git provider.
Illustrative estimate (assumed numbers, not a benchmark): Git requests per minute ≈ Applications × (60 ÷ refresh interval in seconds). With 300 Applications at a 180-second interval, that is 300 × (60 ÷ 180) = 100 requests/min. This assumes each refresh hits Git; caching usually lowers the real number.
Plan for this load as part of operating cost. Use webhooks so you can lengthen the polling interval, shard the application controller across clusters, and set CPU and memory limits based on observed repo-server manifest generation.
When should you avoid this approach, and what are the alternatives?
All of that tuning only matters if the structure pays for itself. With a handful of services in a single environment, a root Application, generators and sync waves add indirection without saving much. When a sync fails, an engineer has to trace root, then child, then manifest path, when they could have opened one file. The pattern also assumes someone owns the root and the generators. Without a team that reviews those changes, one bad merge can re-target or prune every service at once, and nobody is accountable for catching it.
| Approach | Setup cost | Time saved at scale | Audit trail | Blast radius of a bad change | Debugging difficulty |
|---|---|---|---|---|---|
| Hand-written Applications | Low | Low; work grows per service | Full Git history | One service | Easy |
| App of apps alone | Low to moderate | Moderate | Full Git history | Everything under the root | Moderate |
| ApplicationSets alone | Moderate | High for uniform services | Template changes only; rendered Applications are not in Git | Every generated Application | Harder |
| CI push deploys, no GitOps | Low if CI exists | Moderate | CI run logs, not desired state | One pipeline | Easy per run; drift goes unnoticed |
Start with hand-written Applications. Move to app of apps once registering new services becomes routine work. Use ApplicationSets once most services differ only by name, namespace or cluster. Push deploys are still reasonable for small teams that accept manual drift checks in exchange for having less to operate.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to argo cd app of apps, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.