What is the Argo CD App of Apps pattern and how does it work?
The App of Apps pattern uses Argo CD to manage its own configuration. You create one parent Application whose source path contains child Application manifests rather than Deployments or Services. When the parent syncs, Argo CD applies those manifests like any other Kubernetes resource. Each child Application then syncs its own workloads to its own destination. To add, change, or remove a child, you commit a file in that directory.
The parent's fields are usually simple:
spec.source.repoURLandspec.source.pathpoint at a directory such asapps/prod/.spec.destinationtargets the cluster and namespace where Argo CD runs, typicallyargocd. The children are custom resources that Argo CD reads from there, not from the workload clusters.- Each child sets its own
spec.destination, so one parent can fan out to many clusters and namespaces.
The resulting tree looks like this:
- root-prod (parent, path
apps/prod/, deploys intoargocdnamespace)- payments-prod → cluster
prod-eu, namespacepayments - orders-prod → cluster
prod-eu, namespaceorders - ingress-prod → cluster
prod-us, namespaceingress-system
- payments-prod → cluster
You get one bootstrap point per environment or cluster. When you rebuild a cluster or stand up a new region, you apply only the parent and Argo CD recreates everything else from Git. This matters most to platform teams that own several clusters or environments. They no longer apply dozens of Applications by hand or keep their state in sync manually, and each environment's full contents can be reviewed as a single directory.
# apps/root-prod.yaml: the only manifest you apply by hand per environment.
# Its source directory holds child Application manifests, not workloads.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root-prod
namespace: argocd # Applications are read from Argo CD's own namespace
finalizers:
# Deleting the parent also cascades to the child Applications it created
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: # placeholder repo
targetRevision: main
path: apps/prod # directory of child Application manifests
directory:
recurse: false # only files directly in apps/prod/
destination:
# The children are custom resources for Argo CD itself, so the parent
# deploys into the cluster where Argo CD runs, not a workload cluster.
server:
namespace: argocd
syncPolicy:
automated:
prune: true # removing a child file from Git deletes that Application
selfHeal: true # revert manual edits to child Applications
# Resulting tree after sync (each child sets its own spec.destination):
#
# root-prod (argocd ns, in-cluster)
# |-- payments-prod --> cluster prod-eu, ns payments
# |-- orders-prod --> cluster prod-eu, ns orders
# `-- ingress-prod --> cluster prod-us, ns ingress-system
A minimal parent Application that syncs the child Application manifests in apps/prod/ into the argocd namespace, with an ASCII tree showing each child fanning out to its own cluster and namespace.
How should you lay out the Git repository for multiple environments?
A layout that holds up across many services uses three top-level directories, each with a distinct job:
bootstrap/: the parent Application or Applications. These are applied once, by hand or by cluster provisioning.apps/dev/,apps/staging/,apps/prod/: child Application manifests, one file per service per environment.workloads/<service>/: either a Kustomizebase/withoverlays/<env>/, or a Helm chart with one values file per environment.
The layers are separate because they have different owners and change at different rates. The platform team owns bootstrap/ and apps/, which decide where a service runs and under which policy. Service teams own workloads/. An image tag bump lands in workloads/ several times a day. A file in apps/ changes only when a service is added or moved. bootstrap/ almost never changes.
The split also limits blast radius. A commit to workloads/api/overlays/staging cannot affect production. A CODEOWNERS rule on apps/prod/ can require platform review before anything there changes.
For the parent layer, choose between two structures:
| One parent per environment | Single root fanning out | |
|---|---|---|
| Bootstrap step | Apply one parent per cluster or environment | Apply one root once |
| Failure isolation | A broken prod parent does not affect dev | A bad root commit affects every environment |
| Fits | Separate clusters per environment, strict prod access | One management cluster, a small team |
| Adding an environment | New parent file plus bootstrap | New directory, picked up by the root |
In each apps/<env>/ directory, three fields should differ. The destination cluster points dev and prod at different API servers. The namespace follows a convention such as api-prod. The project field references an environment-specific AppProject, so a dev Application cannot deploy into the prod cluster even if someone copies the manifest by mistake.
# apps/dev/api.yaml: child Application for the api service in dev
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: api-dev
namespace: argocd # Application objects live where Argo CD runs
spec:
project: dev # AppProject with permissive dev policy
source:
repoURL:
targetRevision: main
path: workloads/api/overlays/dev # owned by the service team
destination:
server: # dev runs on the management cluster
namespace: api-dev # <service>-<env> convention
syncPolicy:
automated: {} # dev syncs every merged commit automatically
---
# apps/prod/api.yaml: same service, different cluster, namespace and project
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: api-prod
namespace: argocd
spec:
project: prod # AppProject restricting repos, clusters, namespaces
source:
repoURL:
targetRevision: main
path: workloads/api/overlays/prod
destination:
server: # separate prod API server
namespace: api-prod
# No automated sync: prod changes are synced manually after review
Two child Applications for the same service showing that only the destination cluster, namespace, project and overlay path change between the dev and prod directories.
When should you use ApplicationSet instead of hand-written child Applications?
Hand-written children work until the third environment. Each Application differs from its siblings by a few fields (target revision, value file, destination cluster, project), so engineers copy one file and edit it. Sooner or later, someone changes a sync option or adds an ignore rule in prod and forgets dev. The environments then drift because the manifests that define them have diverged, even when the application code has not.
An ApplicationSet replaces those copies with one template and a generator that supplies the parameters:
- List generator: a fixed set of environments written inline, such as dev, staging and prod. Each element carries its own cluster URL and values file. This works well for a small, stable set of environments.
- Cluster generator: one Application per cluster registered in Argo CD, optionally filtered by cluster labels. This fits when you add clusters more often than you add apps.
- Git directory generator: one Application per directory that matches a path pattern. A new service directory becomes a new Application with no further edits.
You can combine the approaches. The parent App of Apps syncs ApplicationSet resources, and each ApplicationSet generates the per-environment children. The cost is indirection. To see what was deployed, you have to render the template in your head, and a typo in a parameter fails at generation time rather than in review.
Set one ownership rule and enforce it: every Application has exactly one manager. If an ApplicationSet generates api-prod, nobody commits a hand-written api-prod, and the parent never manages the generated children directly.
| Approach | Duplication | Readability | Best fit |
|---|---|---|---|
| Hand-written children | High | Explicit, easy to diff | Few apps, few environments |
| ApplicationSet | Low | Requires rendering the template | Many uniform apps or clusters |
| Parent manages ApplicationSets | Low | Two layers to trace | Platform-wide bootstrap |
# Lives in the parent App of Apps repo path; the parent syncs this ApplicationSet,
# and the ApplicationSet (not the parent) owns the generated api-<env> Applications.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: api
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"] # fail generation on a typo'd parameter
generators:
- list:
elements:
# Only the fields that differ per environment go here.
- env: dev
server:
revision: main
- env: staging
server:
revision: main
- env: prod
server:
revision: release-1.4
template:
metadata:
name: 'api-{{.env}}' # never hand-write an Application with this name
spec:
project: '{{.env}}'
source:
repoURL:
targetRevision: '{{.revision}}'
path: chart
helm:
valueFiles:
- 'values-{{.env}}.yaml'
destination:
server: '{{.server}}'
namespace: api
# Shared settings are written once, so dev and prod cannot drift apart.
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
An ApplicationSet with a list generator that renders api-dev, api-staging and api-prod from one template, synced by the parent App of Apps so each generated Application has exactly one manager.
How do you promote changes between environments with GitOps?
Promotion should be an ordinary pull request. Each environment pins an exact image tag or chart version in its own values file, and nothing floats on latest. To promote, open a PR that copies the version already proven in staging into production. A typical promotion diff touches one line in envs/prod/values.yaml: - tag: 1.4.2 becomes + tag: 1.5.0. Reviewers see exactly what changes and where.
| Aspect | Directory per environment, one branch | Branch per environment |
|---|---|---|
| Promotion mechanism | PR editing envs/prod/ |
Merge or cherry-pick between branches |
| Drift risk | Low; differences are visible files | High; cherry-picks skip commits and branches diverge |
| Cross-environment diff | diff envs/staging envs/prod |
Branch diff mixes promoted and unpromoted history |
| Production gating | CODEOWNERS or path-based required checks | Branch protection on prod |
Prefer a single branch with a directory per environment. Branch-per-environment looks tidy at first, but cherry-picked fixes accumulate, and comparing branches becomes hard to read. The trade-off is that branch protection alone no longer guards production, because every environment lives on the same branch. Assign owners to envs/prod/** and require their review, or add CI checks that run only when production paths change.
Automated tag-bump tools can open or merge these PRs when CI publishes a new image. That speeds up lower environments, but it adds a component holding write access to your repository, so scope its credentials narrowly. Keep production on human-reviewed PRs. The payoff is time saved during incidents: git log answers who changed what and when, and rollback is a git revert of the promotion commit.
#!/usr/bin/env bash
# Promote the image tag already proven in staging to production as a reviewed PR.
# Assumes each values file pins a single line like " tag: 1.5.0".
set -euo pipefail
STAGING=envs/staging/values.yaml
PROD=envs/prod/values.yaml
# Read the exact tag pinned in staging. Never promote a floating tag like latest.
TAG=$(grep -E '^[[:space:]]+tag:' "$STAGING" | awk '{print $2}')
# Optional: see every remaining difference between the two environments.
diff -u "$STAGING" "$PROD" || true
# Start the promotion branch from an up-to-date main.
git switch main
git pull --ff-only
git switch -c "promote/prod-${TAG}"
# Change only the production values file.
sed -i -E "s/^([[:space:]]+tag:).*/\1 ${TAG}/" "$PROD"
# The PR diff should be a single changed line, e.g. 1.4.2 -> 1.5.0.
git diff -- "$PROD"
git commit -am "promote(prod): image tag ${TAG}"
git push -u origin "promote/prod-${TAG}"
# Open the PR in your Git host. A CODEOWNERS rule on envs/prod/** requires
# production owners to approve before merge. After merge, Argo CD syncs prod.
# Rollback during an incident is another reviewed change:
# git revert <promotion-commit-sha> # squash or rebase merges
# git revert -m 1 <merge-commit-sha> # merge commits
# Then push the revert on a branch and open a PR as usual.
A Git-only promotion workflow that copies the pinned staging image tag into envs/prod/values.yaml on its own branch, so the PR shows a single-line diff and rollback is a git revert of that commit.
Which sync policies, sync waves and AppProjects should each environment use?
| Environment | Automated sync | Prune | Self-heal | Who triggers sync |
|---|---|---|---|---|
| Dev | Yes | Yes | Yes | Argo CD, on every commit |
| Staging | Yes | Yes | Yes | Argo CD, after promotion PR merges |
| Production | No, or automated without prune | No | Optional | A human, after reviewing the diff |
In dev, automated with prune: true and selfHeal: true keeps the cluster an exact mirror of Git, so stale resources do not hide bugs. In production, omit the automated block or leave prune off. A bad commit then shows up as an OutOfSync diff for someone to review, instead of deletions that are already running.
Order dependencies with the argocd.argoproj.io/sync-wave annotation. Lower waves sync first, and the default is 0. Put CRDs and operators in a negative wave so they are healthy before any app that creates their custom resources. Otherwise the first sync fails because the CRD does not exist yet.
Give each environment its own AppProject. Set sourceRepos to the allowed repositories, and set destinations to the matching cluster and namespaces. A dev Application that points at the production cluster is then rejected by Argo CD itself, whatever the YAML says.
Be careful with prune on the parent. If someone deletes a child manifest from Git, the parent prunes that child Application. If the child has the resources finalizer, its workloads are deleted too.
Document how to pause sync during an incident. Run argocd app set <app> --sync-policy none on the child. If the parent self-heals, also pause the parent first, or it restores the child's automated policy and reverts your emergency fix.
What happens when you delete a child or parent Application?
Deletion in App of Apps runs through two layers: the parent prunes child Application objects, and each child decides whether its workloads go with it. That second decision is controlled by one entry in the child's metadata.finalizers list: resources-finalizer.argocd.argoproj.io.
When an Application carries this finalizer, deleting the Application object triggers a cascading delete. Argo CD removes every resource the Application manages, then releases the finalizer. Without it, only the Application object disappears and the workloads keep running, unmanaged. Teams are often surprised because many templates and examples add the finalizer by default. Removing a YAML file from the parent path looks like bookkeeping, but if the parent has prune: true, Argo CD deletes the child Application. The finalizer then deletes the child's Deployments, Services and PVCs.
| Action | Child has finalizer | Child has no finalizer |
|---|---|---|
| Child manifest removed, parent prunes | Workloads deleted | Workloads orphaned, keep running |
| Child manifest removed, parent doesn't prune | Child remains, parent shows OutOfSync | Same |
| Child renamed | Old workloads deleted, new ones created | New Application adopts existing resources |
| Parent deleted with its own finalizer | Children deleted, then their workloads | Children deleted, workloads orphaned |
| Parent deleted without finalizer | Children remain | Children remain |
Argo CD treats a rename as a delete followed by a create. If the child has the finalizer, the rename causes downtime.
To rename a child safely:
- Pause the parent's automated sync.
- Remove the finalizer from the old child, or run
argocd app delete <app> --cascade=false. - Commit the renamed manifest.
- Re-enable sync on the parent.
The new Application adopts the running resources.
To retire a child, keep the finalizer and remove the manifest deliberately. Use a reviewed commit, not a side effect of a refactor.
When should you avoid App of Apps, and what are the alternatives?
App of Apps pays off when you have several environments and a service list that keeps growing. With one cluster and three or four applications, a parent Application adds a layer of indirection that every engineer must understand before debugging a failed sync. In that case, commit a few Application manifests and apply them once. The pattern also fits poorly when the children are near-identical and differ only by cluster or path. Generators handle that case better.
| Approach | Fits | Main cost |
|---|---|---|
| Plain Applications | Single cluster, few apps | Copy-paste grows with every app |
| ApplicationSet alone | Many similar apps or clusters | Template logic is harder to read, and per-app exceptions get awkward |
| One Argo CD instance for all environments | Small team, few clusters | One failure point; the instance that deploys dev also holds production credentials |
| One Argo CD instance per environment | Strict isolation, regulated production | More instances to upgrade, monitor and configure consistently |
The cost is upfront: you design the repository layout, naming conventions and AppProject boundaries before anything deploys. In return, adding an environment becomes a directory plus a reviewed commit instead of a runbook. You also copy less and deploy by hand less often. If you never expect to add environments, that return may not arrive.
Verification checklist
- Run
argocd app geton the parent and on each child, or open the tree view in the UI. Every node should show Synced and Healthy. - Run
argocd app diffbefore any manual sync, and read what will change. - In a non-production environment, edit a managed field directly with kubectl. Confirm that Argo CD reports OutOfSync, and that self-heal reverts the change where it is enabled.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to argo cd app of apps, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.