What is the Argo CD App of Apps pattern and how does it work?

The App of Apps pattern uses Argo CD to manage its own configuration. You create one parent Application whose source path contains child Application manifests rather than Deployments or Services. When the parent syncs, Argo CD applies those manifests like any other Kubernetes resource. Each child Application then syncs its own workloads to its own destination. To add, change, or remove a child, you commit a file in that directory.

The parent's fields are usually simple:

  • spec.source.repoURL and spec.source.path point at a directory such as apps/prod/.
  • spec.destination targets the cluster and namespace where Argo CD runs, typically argocd. The children are custom resources that Argo CD reads from there, not from the workload clusters.
  • Each child sets its own spec.destination, so one parent can fan out to many clusters and namespaces.

The resulting tree looks like this:

  • root-prod (parent, path apps/prod/, deploys into argocd namespace)
    • payments-prod → cluster prod-eu, namespace payments
    • orders-prod → cluster prod-eu, namespace orders
    • ingress-prod → cluster prod-us, namespace ingress-system

You get one bootstrap point per environment or cluster. When you rebuild a cluster or stand up a new region, you apply only the parent and Argo CD recreates everything else from Git. This matters most to platform teams that own several clusters or environments. They no longer apply dozens of Applications by hand or keep their state in sync manually, and each environment's full contents can be reviewed as a single directory.

# apps/root-prod.yaml: the only manifest you apply by hand per environment.
# Its source directory holds child Application manifests, not workloads.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-prod
  namespace: argocd          # Applications are read from Argo CD's own namespace
  finalizers:
    # Deleting the parent also cascades to the child Applications it created
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL:   # placeholder repo
    targetRevision: main
    path: apps/prod          # directory of child Application manifests
    directory:
      recurse: false         # only files directly in apps/prod/
  destination:
    # The children are custom resources for Argo CD itself, so the parent
    # deploys into the cluster where Argo CD runs, not a workload cluster.
    server:
    namespace: argocd
  syncPolicy:
    automated:
      prune: true            # removing a child file from Git deletes that Application
      selfHeal: true         # revert manual edits to child Applications

# Resulting tree after sync (each child sets its own spec.destination):
#
#   root-prod (argocd ns, in-cluster)
#    |-- payments-prod --> cluster prod-eu, ns payments
#    |-- orders-prod   --> cluster prod-eu, ns orders
#    `-- ingress-prod  --> cluster prod-us, ns ingress-system

A minimal parent Application that syncs the child Application manifests in apps/prod/ into the argocd namespace, with an ASCII tree showing each child fanning out to its own cluster and namespace.

How should you lay out the Git repository for multiple environments?

A layout that holds up across many services uses three top-level directories, each with a distinct job:

  • bootstrap/: the parent Application or Applications. These are applied once, by hand or by cluster provisioning.
  • apps/dev/, apps/staging/, apps/prod/: child Application manifests, one file per service per environment.
  • workloads/<service>/: either a Kustomize base/ with overlays/<env>/, or a Helm chart with one values file per environment.

The layers are separate because they have different owners and change at different rates. The platform team owns bootstrap/ and apps/, which decide where a service runs and under which policy. Service teams own workloads/. An image tag bump lands in workloads/ several times a day. A file in apps/ changes only when a service is added or moved. bootstrap/ almost never changes.

The split also limits blast radius. A commit to workloads/api/overlays/staging cannot affect production. A CODEOWNERS rule on apps/prod/ can require platform review before anything there changes.

For the parent layer, choose between two structures:

One parent per environment Single root fanning out
Bootstrap step Apply one parent per cluster or environment Apply one root once
Failure isolation A broken prod parent does not affect dev A bad root commit affects every environment
Fits Separate clusters per environment, strict prod access One management cluster, a small team
Adding an environment New parent file plus bootstrap New directory, picked up by the root

In each apps/<env>/ directory, three fields should differ. The destination cluster points dev and prod at different API servers. The namespace follows a convention such as api-prod. The project field references an environment-specific AppProject, so a dev Application cannot deploy into the prod cluster even if someone copies the manifest by mistake.

# apps/dev/api.yaml: child Application for the api service in dev
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-dev
  namespace: argocd            # Application objects live where Argo CD runs
spec:
  project: dev                 # AppProject with permissive dev policy
  source:
    repoURL:
    targetRevision: main
    path: workloads/api/overlays/dev   # owned by the service team
  destination:
    server:    # dev runs on the management cluster
    namespace: api-dev                       # <service>-<env> convention
  syncPolicy:
    automated: {}              # dev syncs every merged commit automatically
---
# apps/prod/api.yaml: same service, different cluster, namespace and project
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-prod
  namespace: argocd
spec:
  project: prod                # AppProject restricting repos, clusters, namespaces
  source:
    repoURL:
    targetRevision: main
    path: workloads/api/overlays/prod
  destination:
    server:    # separate prod API server
    namespace: api-prod
  # No automated sync: prod changes are synced manually after review

Two child Applications for the same service showing that only the destination cluster, namespace, project and overlay path change between the dev and prod directories.

When should you use ApplicationSet instead of hand-written child Applications?

Hand-written children work until the third environment. Each Application differs from its siblings by a few fields (target revision, value file, destination cluster, project), so engineers copy one file and edit it. Sooner or later, someone changes a sync option or adds an ignore rule in prod and forgets dev. The environments then drift because the manifests that define them have diverged, even when the application code has not.

An ApplicationSet replaces those copies with one template and a generator that supplies the parameters:

  • List generator: a fixed set of environments written inline, such as dev, staging and prod. Each element carries its own cluster URL and values file. This works well for a small, stable set of environments.
  • Cluster generator: one Application per cluster registered in Argo CD, optionally filtered by cluster labels. This fits when you add clusters more often than you add apps.
  • Git directory generator: one Application per directory that matches a path pattern. A new service directory becomes a new Application with no further edits.

You can combine the approaches. The parent App of Apps syncs ApplicationSet resources, and each ApplicationSet generates the per-environment children. The cost is indirection. To see what was deployed, you have to render the template in your head, and a typo in a parameter fails at generation time rather than in review.

Set one ownership rule and enforce it: every Application has exactly one manager. If an ApplicationSet generates api-prod, nobody commits a hand-written api-prod, and the parent never manages the generated children directly.

Approach Duplication Readability Best fit
Hand-written children High Explicit, easy to diff Few apps, few environments
ApplicationSet Low Requires rendering the template Many uniform apps or clusters
Parent manages ApplicationSets Low Two layers to trace Platform-wide bootstrap
# Lives in the parent App of Apps repo path; the parent syncs this ApplicationSet,
# and the ApplicationSet (not the parent) owns the generated api-<env> Applications.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: api
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]  # fail generation on a typo'd parameter
  generators:
    - list:
        elements:
          # Only the fields that differ per environment go here.
          - env: dev
            server:
            revision: main
          - env: staging
            server:
            revision: main
          - env: prod
            server:
            revision: release-1.4
  template:
    metadata:
      name: 'api-{{.env}}'  # never hand-write an Application with this name
    spec:
      project: '{{.env}}'
      source:
        repoURL:
        targetRevision: '{{.revision}}'
        path: chart
        helm:
          valueFiles:
            - 'values-{{.env}}.yaml'
      destination:
        server: '{{.server}}'
        namespace: api
      # Shared settings are written once, so dev and prod cannot drift apart.
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

An ApplicationSet with a list generator that renders api-dev, api-staging and api-prod from one template, synced by the parent App of Apps so each generated Application has exactly one manager.

How do you promote changes between environments with GitOps?

Promotion should be an ordinary pull request. Each environment pins an exact image tag or chart version in its own values file, and nothing floats on latest. To promote, open a PR that copies the version already proven in staging into production. A typical promotion diff touches one line in envs/prod/values.yaml: - tag: 1.4.2 becomes + tag: 1.5.0. Reviewers see exactly what changes and where.

Aspect Directory per environment, one branch Branch per environment
Promotion mechanism PR editing envs/prod/ Merge or cherry-pick between branches
Drift risk Low; differences are visible files High; cherry-picks skip commits and branches diverge
Cross-environment diff diff envs/staging envs/prod Branch diff mixes promoted and unpromoted history
Production gating CODEOWNERS or path-based required checks Branch protection on prod

Prefer a single branch with a directory per environment. Branch-per-environment looks tidy at first, but cherry-picked fixes accumulate, and comparing branches becomes hard to read. The trade-off is that branch protection alone no longer guards production, because every environment lives on the same branch. Assign owners to envs/prod/** and require their review, or add CI checks that run only when production paths change.

Automated tag-bump tools can open or merge these PRs when CI publishes a new image. That speeds up lower environments, but it adds a component holding write access to your repository, so scope its credentials narrowly. Keep production on human-reviewed PRs. The payoff is time saved during incidents: git log answers who changed what and when, and rollback is a git revert of the promotion commit.

#!/usr/bin/env bash
# Promote the image tag already proven in staging to production as a reviewed PR.
# Assumes each values file pins a single line like "  tag: 1.5.0".
set -euo pipefail

STAGING=envs/staging/values.yaml
PROD=envs/prod/values.yaml

# Read the exact tag pinned in staging. Never promote a floating tag like latest.
TAG=$(grep -E '^[[:space:]]+tag:' "$STAGING" | awk '{print $2}')

# Optional: see every remaining difference between the two environments.
diff -u "$STAGING" "$PROD" || true

# Start the promotion branch from an up-to-date main.
git switch main
git pull --ff-only
git switch -c "promote/prod-${TAG}"

# Change only the production values file.
sed -i -E "s/^([[:space:]]+tag:).*/\1 ${TAG}/" "$PROD"

# The PR diff should be a single changed line, e.g. 1.4.2 -> 1.5.0.
git diff -- "$PROD"

git commit -am "promote(prod): image tag ${TAG}"
git push -u origin "promote/prod-${TAG}"
# Open the PR in your Git host. A CODEOWNERS rule on envs/prod/** requires
# production owners to approve before merge. After merge, Argo CD syncs prod.

# Rollback during an incident is another reviewed change:
#   git revert <promotion-commit-sha>        # squash or rebase merges
#   git revert -m 1 <merge-commit-sha>       # merge commits
# Then push the revert on a branch and open a PR as usual.

A Git-only promotion workflow that copies the pinned staging image tag into envs/prod/values.yaml on its own branch, so the PR shows a single-line diff and rollback is a git revert of that commit.

Which sync policies, sync waves and AppProjects should each environment use?

Environment Automated sync Prune Self-heal Who triggers sync
Dev Yes Yes Yes Argo CD, on every commit
Staging Yes Yes Yes Argo CD, after promotion PR merges
Production No, or automated without prune No Optional A human, after reviewing the diff

In dev, automated with prune: true and selfHeal: true keeps the cluster an exact mirror of Git, so stale resources do not hide bugs. In production, omit the automated block or leave prune off. A bad commit then shows up as an OutOfSync diff for someone to review, instead of deletions that are already running.

Order dependencies with the argocd.argoproj.io/sync-wave annotation. Lower waves sync first, and the default is 0. Put CRDs and operators in a negative wave so they are healthy before any app that creates their custom resources. Otherwise the first sync fails because the CRD does not exist yet.

Give each environment its own AppProject. Set sourceRepos to the allowed repositories, and set destinations to the matching cluster and namespaces. A dev Application that points at the production cluster is then rejected by Argo CD itself, whatever the YAML says.

Be careful with prune on the parent. If someone deletes a child manifest from Git, the parent prunes that child Application. If the child has the resources finalizer, its workloads are deleted too.

Document how to pause sync during an incident. Run argocd app set <app> --sync-policy none on the child. If the parent self-heals, also pause the parent first, or it restores the child's automated policy and reverts your emergency fix.

What happens when you delete a child or parent Application?

Deletion in App of Apps runs through two layers: the parent prunes child Application objects, and each child decides whether its workloads go with it. That second decision is controlled by one entry in the child's metadata.finalizers list: resources-finalizer.argocd.argoproj.io.

When an Application carries this finalizer, deleting the Application object triggers a cascading delete. Argo CD removes every resource the Application manages, then releases the finalizer. Without it, only the Application object disappears and the workloads keep running, unmanaged. Teams are often surprised because many templates and examples add the finalizer by default. Removing a YAML file from the parent path looks like bookkeeping, but if the parent has prune: true, Argo CD deletes the child Application. The finalizer then deletes the child's Deployments, Services and PVCs.

Action Child has finalizer Child has no finalizer
Child manifest removed, parent prunes Workloads deleted Workloads orphaned, keep running
Child manifest removed, parent doesn't prune Child remains, parent shows OutOfSync Same
Child renamed Old workloads deleted, new ones created New Application adopts existing resources
Parent deleted with its own finalizer Children deleted, then their workloads Children deleted, workloads orphaned
Parent deleted without finalizer Children remain Children remain

Argo CD treats a rename as a delete followed by a create. If the child has the finalizer, the rename causes downtime.

To rename a child safely:

  1. Pause the parent's automated sync.
  2. Remove the finalizer from the old child, or run argocd app delete <app> --cascade=false.
  3. Commit the renamed manifest.
  4. Re-enable sync on the parent.

The new Application adopts the running resources.

To retire a child, keep the finalizer and remove the manifest deliberately. Use a reviewed commit, not a side effect of a refactor.

When should you avoid App of Apps, and what are the alternatives?

App of Apps pays off when you have several environments and a service list that keeps growing. With one cluster and three or four applications, a parent Application adds a layer of indirection that every engineer must understand before debugging a failed sync. In that case, commit a few Application manifests and apply them once. The pattern also fits poorly when the children are near-identical and differ only by cluster or path. Generators handle that case better.

Approach Fits Main cost
Plain Applications Single cluster, few apps Copy-paste grows with every app
ApplicationSet alone Many similar apps or clusters Template logic is harder to read, and per-app exceptions get awkward
One Argo CD instance for all environments Small team, few clusters One failure point; the instance that deploys dev also holds production credentials
One Argo CD instance per environment Strict isolation, regulated production More instances to upgrade, monitor and configure consistently

The cost is upfront: you design the repository layout, naming conventions and AppProject boundaries before anything deploys. In return, adding an environment becomes a directory plus a reviewed commit instead of a runbook. You also copy less and deploy by hand less often. If you never expect to add environments, that return may not arrive.

Verification checklist

  • Run argocd app get on the parent and on each child, or open the tree view in the UI. Every node should show Synced and Healthy.
  • Run argocd app diff before any manual sync, and read what will change.
  • In a non-production environment, edit a managed field directly with kubectl. Confirm that Argo CD reports OutOfSync, and that self-heal reverts the change where it is enabled.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to argo cd app of apps, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.