How does Kubernetes pod startup work, gate by gate?

A pod passes through a fixed sequence of gates before it serves traffic. Each gate answers one question, and each has its own failure behaviour.

  1. Init containers run one at a time, in the order declared. Each must exit successfully before the next starts. App containers do not start until all of them have finished.
  2. Startup probe, if defined, runs once the app container starts. Liveness and readiness probes are disabled until it succeeds. This gives a slow process time to boot without being killed for it.
  3. Readiness probe decides whether the pod's IP is listed in Service endpoints. A failure removes the pod from traffic but does not restart it. It keeps running for the life of the container.
  4. Readiness gates add custom pod conditions, usually set by an external controller. The pod is Ready only when every container is ready and every gate condition is True.
  5. Liveness probe takes over after startup and restarts the container when it fails.

Most startup incidents come from one gate doing another gate's job. Common examples:

  • A liveness probe tuned to tolerate a slow boot, which then takes minutes to catch a real hang.
  • A readiness probe that checks a shared database, so every pod drops out of the Service at once when that database has a problem.
Gate What it blocks On failure Single concern it should own
Init containers App container start Retried per restartPolicy; pod stays in Init One-time preconditions before the app runs
Startup probe Liveness and readiness checks Container killed after failureThreshold "Has the process finished booting?"
Readiness probe Service endpoint membership Removed from endpoints, no restart "Can this pod handle a request right now?"
Readiness gates Pod Ready condition Pod stays not-Ready External signals the kubelet cannot see
Liveness probe Nothing; it acts after the fact Container restarted "Is the process stuck beyond recovery?"
+-------------+    +-------------+
| init-1      |--->| init-N      |  fail: retry per restartPolicy,
| run to exit |    | run to exit |        pod stays in Init
+-------------+    +-------------+
                          |  all exited 0: app container starts
                          v
                 +------------------+  fail x failureThreshold
                 | startup probe    |------------------------> kill container
                 | liveness and     |
                 | readiness held   |
                 +------------------+
                          | success
           +--------------+--------------+
           v                             v
+--------------------+        +--------------------+
| readiness probe    |        | liveness probe     |
| fail: leave        |        | fail: restart      |
| endpoints, no kill |        | container          |
+--------------------+        +--------------------+
           | containers ready
           v
+--------------------+    +----------------------------+
| Pod Ready only if  |<---| readiness gates: custom    |
| containers ready   |    | conditions set by external |
| AND gates all True |    | controller                 |
+--------------------+    +----------------------------+
           | Ready
           v
   Service endpoints: pod receives traffic

Pod startup in order: init containers run one after another, the startup probe holds back liveness and readiness, then readiness and readiness gates together decide whether the pod is in Service endpoints, while liveness handles restarts.

When should you use a Kubernetes startup probe for slow starting pods?

Liveness probes run as soon as the container starts, so a slow-starting process can be killed before it is ever ready. The traditional fix was to raise initialDelaySeconds on the liveness probe, but that has two problems. The delay applies to every container start, so a pod that comes up in 20 seconds still goes unchecked for the full delay. And the delay is a fixed guess: if one start runs past it, the kubelet restarts a healthy container in the middle of initialization, often in a loop.

A startup probe solves both. Liveness and readiness checks are held off until the startup probe succeeds once. After that, the startup probe stops running and the normal probes take over. Fast starts finish early, and slow starts get the time they need.

Size the window as failureThreshold × periodSeconds, plus any initialDelaySeconds. Base it on your measured worst-case startup, not the typical one. Worst cases include a cold cache, a large migration, JIT warmup, or a node pulling dependencies under contention.

Illustrative estimate: assume a 4-minute worst case. With periodSeconds: 10 and failureThreshold: 30, the startup budget is 300 seconds, about 5 minutes, which leaves a minute of margin. Once startup succeeds, liveness can stay tight. For example, periodSeconds: 10 with failureThreshold: 3 catches a hang in roughly 30 seconds.

The trade-off is detection time during startup. A pod that hangs while initializing will not be restarted until the full 5-minute budget runs out. Keep the window close to your measured worst case rather than padding it indefinitely.

The example below configures all three probes together.

apiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  containers:
    - name: app
      image: registry.example.com/orders-api:1.0.0
      ports:
        - containerPort: 8080
      # Startup budget: failureThreshold x periodSeconds = 30 x 10 = 300s.
      # Sized against an assumed 4-minute worst case (illustrative), not the typical start.
      # Liveness and readiness do not run until this probe succeeds once.
      startupProbe:
        httpGet:
          path: /healthz
          port: 8080
        periodSeconds: 10
        failureThreshold: 30
      # Tight after startup: about 3 x 10s = ~30s to detect a hang and restart.
      # No initialDelaySeconds needed; the startup probe covers slow starts.
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        periodSeconds: 10
        failureThreshold: 3
      # Readiness gates traffic. Failing it removes the pod from Service endpoints
      # without restarting the container.
      readinessProbe:
        httpGet:
          path: /ready
          port: 8080
        periodSeconds: 5
        failureThreshold: 3

A pod spec where a startup probe allows a 300-second startup budget, then hands off to a tight liveness probe and a separate readiness probe.

What should liveness and readiness probes check, and what should they avoid?

Liveness answers one question: is this process wedged beyond recovery? Keep the check shallow and local, such as an event loop that responds or a worker thread that is not deadlocked. Never include downstream dependencies. If liveness checks the database, a database outage fails the probe on every replica at the same moment, and the kubelet restarts all of them. The outage gets worse, and the restarts produce a reconnect storm when the database comes back.

Readiness can reflect whether the pod is able to serve: dependencies reachable, caches warm, configuration loaded. A readiness failure removes the pod from Service endpoints without restarting it, so the pod is still running when the condition clears. Use this carefully. If every replica checks the same shared dependency, a brief blip removes all endpoints at once, and clients get connection errors instead of degraded responses. Often it is better to stay ready and return a fallback or a fast error for the affected routes.

Probes run inside the app's resource limits. When the container is being CPU-throttled or paused for garbage collection, a 1-second timeout fails at exactly the moment the pod is busiest. Set timeouts with that headroom in mind.

Probe type What it proves Cost per check Notes
httpGet on a dedicated endpoint Request path and handler work Low Keep the handler cheap. Do not reuse a heavy route.
tcpSocket Port is accepting connections Lowest Misses a hung application behind an open socket.
exec Whatever the command checks Highest Forks a process each time. Overhead grows with node density and replica count.
apiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  containers:
    - name: app
      image: registry.example.com/orders-api:1.4.2
      ports:
        - containerPort: 8080
      resources:
        limits:
          cpu: "500m"     # probes share this budget with real traffic
          memory: "512Mi"
      livenessProbe:
        # Shallow and local: handler only confirms the event loop responds.
        # It must not touch the database or any other downstream service.
        httpGet:
          path: /livez
          port: 8080
        periodSeconds: 10
        timeoutSeconds: 3     # headroom for CPU throttling and GC pauses
        failureThreshold: 3   # tolerate a few slow checks before a restart
      readinessProbe:
        # May reflect ability to serve (config loaded, caches warm).
        # Failure removes the pod from endpoints; it is not restarted.
        httpGet:
          path: /readyz
          port: 8080
        periodSeconds: 5
        timeoutSeconds: 3
        failureThreshold: 3

A pod spec with a shallow, dependency-free liveness endpoint and a separate readiness endpoint, using timeouts and failure thresholds that leave headroom for CPU throttling and GC pauses.

When should you use init containers, and when should you avoid them?

Init containers run one at a time, in order, and each must exit successfully before the next one starts. Only after the last one succeeds do the app containers start. That makes them a good fit for one-time prerequisites:

  • Waiting for a dependency, such as a DNS name resolving or a port opening.
  • Rendering configuration files from templates or secrets into a shared volume.
  • Fixing ownership or permissions on a mounted volume.
  • Fetching artifacts, models or plugins the main image should not bundle.

Make them idempotent. If the pod restarts, its init containers can run again. A step that fails or corrupts state on a second run will eventually break a pod.

Set resource requests. The scheduler uses the pod's effective request, which is the larger of two values: the biggest request of any single init container, or the sum of the app containers' requests. An init container with no requests, or an oversized one, skews where the pod lands.

Know when to avoid them:

  • Dependency waits encode ordering assumptions that are invisible outside the manifest. They only cover startup. If the database goes away an hour later, the init container does nothing, so the application still needs its own retry and backoff logic.
  • Non-idempotent work, such as schema migrations, can run more than once. It can also race across replicas that start at the same time. Run migrations as a separate Job and gate the rollout on that Job completing.

The example below waits for a database service before the app container starts.

apiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  initContainers:
    # Block app startup until the database Service name resolves.
    # Idempotent: re-running after a pod restart is harmless.
    - name: wait-for-db
      image: busybox:1.36
      command:
        - sh
        - -c
        - |
          until nslookup orders-db.default.svc.cluster.local; do
            echo "waiting for orders-db"; sleep 2
          done
      # Explicit requests keep the pod's effective request predictable:
      # max(largest init request, sum of app requests).
      resources:
        requests:
          cpu: 10m
          memory: 16Mi
        limits:
          memory: 32Mi
  containers:
    - name: app
      image: registry.example.com/orders-api:1.4.2
      ports:
        - containerPort: 8080
      env:
        - name: DB_HOST
          value: orders-db.default.svc.cluster.local
      # The app still needs its own retry/backoff: this init container
      # only covers startup, not a database outage later on.
      resources:
        requests:
          cpu: 250m
          memory: 256Mi

A pod whose init container waits for the database Service's DNS name to resolve before the app container starts, with small explicit resource requests so it does not distort the pod's effective scheduling request.

What are readiness gates and who should use them?

Probes and init containers only see what happens inside the pod. Sometimes the pod is healthy but traffic still can't reach it, for example because an external load balancer hasn't registered it as a target yet. Readiness gates handle that case. In spec.readinessGates you list one or more entries, each with a conditionType such as example.com/lb-target-registered. The kubelet then marks the pod Ready only when every container passes its readiness probe and every listed condition in status.conditions is True.

The kubelet never sets these conditions. An external controller watches the pods, does its work (registers the target, programs a route, warms a cache), and patches the pod's status with an entry such as type: example.com/lb-target-registered, status: "True", plus a lastTransitionTime. Until that entry exists, Kubernetes treats it as False. The pod keeps ContainersReady=True but stays Ready=False.

Who needs this

Platform teams that connect pods to traffic systems outside the cluster, where "the process is up" and "the load balancer will send it requests" are different events. Most application teams don't need readiness gates. A good readiness probe covers their case.

The risk

If the controller is missing, crashlooping, lacks RBAC to patch pod status, or watches a different condition name, pods stay NotReady indefinitely. A rolling update waits for new pods to become Ready, so the rollout stalls. Container logs look clean, probes pass, and there are often no events. The only clue is in kubectl describe pod, where the Readiness Gates section shows the condition as False or <none>.

How do probes affect rollout safety and zero-downtime deploys?

A Deployment does not track whether your application works. It tracks whether new pods are available, meaning Ready for at least minReadySeconds. maxUnavailable and maxSurge are enforced against that count. If readiness passes before the process can serve real traffic, the controller scales down old pods based on false information, and the rollout removes capacity it still needs.

minReadySeconds covers a narrower failure: a pod that becomes Ready and then fails a few seconds later, for example when a connection pool exhausts or a cache warm-up crashes. Without it, the controller counts that pod as available right away and moves on to the next batch. With it, the pod has to stay Ready for the whole window before it counts. Set the window longer than the time your known post-start failures take to appear. It slows each rollout step, so keep it as short as that allows. If a rollout stalls past progressDeadlineSeconds, the Deployment reports it as failed but does not roll back automatically.

Readiness has no role on the way out. When a pod is deleted, its removal from endpoints and the SIGTERM to the container happen concurrently. Load balancers and node proxies may keep routing to the pod briefly after SIGTERM arrives. If the process exits immediately, in-flight and newly routed requests fail. To close that gap, add a short preStop delay, then handle SIGTERM by refusing new work and draining existing requests before exiting.

Illustrative estimate (assumed numbers, not a benchmark). Required grace period ≥ preStop delay + longest request drain + exit margin. With an assumed 5s preStop, 20s drain and 5s margin, you need terminationGracePeriodSeconds of at least 30. The preStop time counts against this budget, so it does not extend it.

Tuning startup gets new pods in safely. Getting old pods out safely needs its own shutdown handling.

How do you verify pod startup configuration before it reaches production?

Before a manifest ships, confirm in staging that each gate behaves the way you configured it:

  • Read events with kubectl describe pod <pod>. Unhealthy events name the probe that failed (startup, liveness or readiness) and show its output. Init containers list their State, Last State, exit code and restart count. A rising restart count with liveness failures usually means the startup probe ends too early.
  • Inspect conditions with kubectl get pod <pod> -o yaml. Under status.conditions, check Initialized, ContainersReady and Ready. If you declared spec.readinessGates, each gate's condition type must appear with status: "True". If containers are ready but the pod is not, a gate condition is missing or false, so look at the controller that should set it.
  • Read init failures with kubectl logs <pod> -c <init-container>. Add --previous if the container has already restarted.
  • Force a cold start. Schedule onto a node that does not have the image cached, with empty application caches. Time how long the pod takes to reach Ready, and compare that with failureThreshold × periodSeconds on the startup probe. Leave clear headroom between the two.

Giving each concern to one mechanism saves debugging time. You see fewer false restarts and fewer stalled rollouts. When something fails, the failure points to a single place, so diagnosis starts from the right component instead of guesswork.

Concern Owning mechanism Alternatives
Slow first boot Startup probe Long initialDelaySeconds (wastes time on fast boots)
Deadlocked process Liveness probe None advisable; don't check dependencies
Ready for traffic Readiness probe Readiness gate for external signals
One-time setup Init container Job, or in-app retry logic
External readiness Readiness gate Readiness probe calling the external system
Clean shutdown preStop + grace period In-app SIGTERM handling

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes startup probe, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.