Ephemeral Environments for Microservices: Per-PR Integration Testing

What an ephemeral environment is and why microservice teams use one

A per-PR ephemeral environment is a short-lived, isolated deployment of your system, created when a pull request opens and destroyed automatically when it merges or closes. It is built from the same manifests, images, and configuration templates that ship to production, with only the parameters changed: namespace, hostnames, replica counts, resource limits. That shared source matters. If the preview runs on hand-maintained YAML that drifts from production, a passing test tells you little.

The reason to run one is to catch integration bugs before merge. Unit tests and mocks check a service against your assumptions about its neighbours. They do not catch a renamed field in a JSON payload, a changed gRPC enum, a missing environment variable, a new required header, or a migration that breaks a downstream reader. Those failures appear only when real services talk to each other over a real network with real configuration.

The teams that benefit most run many independently deployed services and share one staging environment. In that setup, staging becomes a queue. Engineers wait for a slot, then find someone else's half-finished change has broken the environment, and lose time working out whose bug they are looking at. Giving each PR its own environment removes the queue and shortens the debug loop. When a test fails, the only change present is yours, so the cause is usually obvious. Reviewers can also open a running instance instead of reasoning from the diff alone.

Choosing an isolation model on Kubernetes

There are three common ways to carve a PR's environment out of Kubernetes. They differ mainly in what still leaks between concurrent PRs.

Namespace per PR is the usual default because it is cheap and fast to create. Its weak points are cluster-scoped objects and node capacity. Two PRs that modify the same CRD or operator overwrite each other. A heavy test run can starve its neighbours unless every namespace has a ResourceQuota.

Cluster per PR removes those collisions. That matters when the change itself is a CRD, an admission webhook or an operator. The cost is provisioning time, which often becomes the longest step in the feedback loop.

Request-level routing deploys only the changed services next to a shared baseline. An ingress or mesh sidecar reads a header carrying the PR identifier and sends matching requests to the PR's version. All other requests go to baseline. This only works if every service copies that header from inbound to outbound calls. For async hops, the header must also go into message metadata, and consumers must route on it. If one service drops the header, downstream calls silently reach baseline code, and tests can pass against code that isn't yours. To catch this, have tests confirm that the PR version actually handled each request, for example through a response header or a trace attribute.

Copy Shared baseline: every service, data store, broker, cache
 PR-42 deploys only changed services; mesh routes on header pr-id=42

 ┌────────┐ pr-id=42 ┌────────────┐ pr-id=42 ┌────────────┐
 │ Test   │─────────▶│ orders     │─────────▶│ payments   │
 │ client │          │ (PR-42)    │  copied  │ (PR-42)    │
 └────────┘          └─────┬──────┘          └─────┬──────┘
     ▲                     │ async: pr-id          │ header DROPPED
     │                     │ in msg metadata       │ on outbound call
     │                     ▼                       ▼
     │              ┌──────────────┐        ┌────────────┐ ┌────────────┐
     │              │ shared broker│        │ ledger     │ │ ledger     │
     │              │ (all PRs)    │        │ (baseline) │ │ (PR-42)    │
     │              └──────┬───────┘        │ gets PR-42 │ │ never hit  │
     │                     │ consumer       └────────────┘ └────────────┘
     │                     │ routes on pr-id
     │                     ▼
     │              ┌──────────────┐
     │              │ shipping     │
     │              │ (PR-42)      │
     │              └──────────────┘
     │
     └── assert every hop: served-by=PR-42 (response header / trace attr)

Request-level routing against a shared baseline: the PR header carries traffic through sync and async hops to PR-42 services, but when payments drops it, the call silently reaches baseline ledger, which per-hop served-by assertions catch.

Deciding what runs real, what is shared, and what is stubbed

Derive each PR's topology from its diff. Deploy every service the PR changes, plus the direct dependencies its tests exercise, and route everything else to the shared baseline. Each additional real service raises fidelity, but it also adds startup time, cluster capacity, and another way for the environment to fail for reasons unrelated to the change.

Version skew appears when the baseline lags main, or when a PR depends on an unmerged change in another service. Redeploy the baseline on every merge to main, and log the baseline version with each test run so a failure can be traced to the code that actually ran.

Fakes drift because providers change their APIs without notice. Run one contract suite against both the fake and the provider's sandbox on a schedule, and fail the build when the two disagree.

Shared brokers cause a quieter problem: baseline consumers pick up messages a PR publishes, and the reverse also happens. Give each PR its own prefix for topics, queues, and cache keys. Use a dedicated instance when the PR changes broker configuration or a service can't take a configurable prefix.

Stateful dependencies: databases, seed data, and secrets

Databases need the same treatment, but a prefix is not enough. If two PRs share one database, a migration in one branch changes the schema the other branch tests against. Rows written by one test run also become fixtures for another. Failures then depend on which PRs happen to be open, and that is the cross-talk ephemeral environments are supposed to remove.

Give each environment its own database, created by running the branch's migrations against an empty instance and then loading deterministic seed data. Use fixed IDs, fixed timestamps, and no random values, so every run starts from the same state. Running migrations from zero also tests the migrations on every PR. Restoring a snapshot is faster when the schema is large, but snapshots drift from the migration history and need their own refresh process. Use them only when migration time actually becomes a bottleneck.

Configuration should come from templates, not be copied by hand. Derive hostnames, connection strings, and callback URLs from the PR number or namespace name.

Secrets should exist only in their own namespace. Each one should grant the minimum needed: a test-mode payment key, not an account-wide key.

The CI identity should only be able to create and delete preview namespaces and deploy into them. It should not be able to read secrets elsewhere in the cluster.

Keep production credentials and copied production data out of preview environments entirely. These environments run unreviewed code by design.

If preview URLs are publicly reachable, put them behind authentication. Unprotected URLs get crawled and shared.

Copy#!/usr/bin/env bash
# Provision an isolated preview environment for one PR.
set -euo pipefail

PR_NUMBER="${PR_NUMBER:?PR_NUMBER is required}"
GIT_SHA="${GIT_SHA:?GIT_SHA is required}"
NS="pr-${PR_NUMBER}"                          # every name derives from the PR number
HOST="pr-${PR_NUMBER}.preview.example.internal"  # templated hostname, never hand-edited

# Idempotent namespace creation, so CI reruns do not fail.
kubectl create namespace "$NS" --dry-run=client -o yaml | kubectl apply -f -
kubectl label namespace "$NS" preview=true --overwrite

# Secrets exist only in this namespace. Values come from CI's preview-only store:
# a throwaway DB password and a test-mode payment key, never production credentials.
kubectl -n "$NS" create secret generic app-secrets \
  --from-literal=DATABASE_URL="postgres://app:${PREVIEW_DB_PASSWORD}@postgres.${NS}.svc.cluster.local:5432/app" \
  --from-literal=PAYMENT_API_KEY="${PAYMENT_TEST_KEY}" \
  --dry-run=client -o yaml | kubectl apply -f -

# Dedicated, empty database for this PR. Nothing is shared with other PRs.
kubectl -n "$NS" apply -f k8s/postgres.yaml
kubectl -n "$NS" rollout status statefulset/postgres --timeout=180s

# Run this branch's migrations from zero, which also tests them on every PR.
kubectl -n "$NS" create job migrate \
  --image="registry.example.com/app-migrate:${GIT_SHA}" -- ./migrate up
kubectl -n "$NS" wait --for=condition=complete job/migrate --timeout=300s

# Deterministic seed: fixed IDs and timestamps, no random values, no production copies.
kubectl -n "$NS" create job seed \
  --image="registry.example.com/app-migrate:${GIT_SHA}" -- ./seed fixtures/seed.sql
kubectl -n "$NS" wait --for=condition=complete job/seed --timeout=120s

# Render the app manifests (ingress host, callback URLs) from a template.
export NS HOST GIT_SHA
envsubst < k8s/app.yaml.tmpl | kubectl -n "$NS" apply -f -

# Teardown when the PR closes: kubectl delete namespace "pr-${PR_NUMBER}"

A CI provisioning script that gives each PR its own namespace, namespace-scoped test-only secrets, a fresh database built from the branch's migrations plus deterministic seed data, and configuration rendered from templates keyed on the PR number.

Implementing the lifecycle in CI

Wire the lifecycle to pull request events so every step is automatic and repeatable.

1. Create or update on open and push. Run the same deploy job for both events. Make it idempotent: apply manifests or upgrade a release in a namespace derived from the PR number, so a re-run converges on the desired state instead of failing because resources already exist.
2. Label everything. Attach the PR number, owner, and creation timestamp as labels on the namespace and its resources. Cleanup, cost reporting, and debugging all depend on being able to select "everything belonging to PR 1423."
3. Wait on readiness, not time. Replace fixed sleeps with kubectl rollout status per deployment, or kubectl wait --for=condition=Available, each with an explicit --timeout. A sleep is too long on fast runs and too short on slow ones. A readiness gate fails clearly when a pod never becomes ready.
4. Test against the environment URL. Pass the URL to the suite as configuration. Generate unique test data per run, such as IDs prefixed with the PR number and commit SHA, so retries and parallel jobs don't collide.
5. Collect evidence, then report. Before teardown, export pod logs, events, and trace IDs as CI artifacts. Post a summary to the PR with links to them.
6. Tear down on merge or close. Delete the namespace.

Teardown jobs will sometimes fail or never run. As a backup, schedule a reaper that deletes labeled namespaces older than a TTL, or ones whose PR is no longer open.

Copy#!/usr/bin/env bash
# Runs on PR open and push. Safe to re-run: every step converges.
set -euo pipefail

PR="${PR_NUMBER:?}"; SHA="${COMMIT_SHA:?}"; OWNER="${PR_AUTHOR:?}"
NS="pr-${PR}"
ENV_URL=""
mkdir -p artifacts

# Create the namespace once. Set created-at only on create so the TTL reaper sees true age.
if ! kubectl get namespace "$NS" >/dev/null 2>&1; then
  kubectl create namespace "$NS"
  kubectl label namespace "$NS" created-at="$(date +%s)"
fi
kubectl label namespace "$NS" --overwrite ephemeral=true pr="$PR" owner="$OWNER"

# Declarative apply: a re-run updates existing resources instead of failing.
kubectl apply -n "$NS" -f k8s/
kubectl label -n "$NS" --overwrite deployments,services --all pr="$PR" owner="$OWNER"

# Collect evidence on any exit (pass or fail) before a later job tears down.
collect() {
  kubectl get events -n "$NS" --sort-by=.lastTimestamp > artifacts/events.txt || true
  for p in $(kubectl get pods -n "$NS" -o name); do
    kubectl logs -n "$NS" "$p" --all-containers > "artifacts/${p#pod/}.log" || true
  done
}
trap collect EXIT

# Gate on readiness with explicit timeouts instead of sleeping.
for d in $(kubectl get deployments -n "$NS" -o name); do
  kubectl rollout status "$d" -n "$NS" --timeout=180s
done

# Unique data prefix per run so retries and parallel jobs don't collide.
TEST_BASE_URL="$ENV_URL" TEST_DATA_PREFIX="pr${PR}-${SHA:0:8}-${GITHUB_RUN_ATTEMPT:-1}" \
  ./scripts/run-integration-tests.sh --junit artifacts/results.xml

# Teardown (separate job on merge/close): kubectl delete namespace "pr-${PR}" --ignore-not-found

An idempotent per-PR deploy-and-test script that labels the namespace and resources, waits on rollout status with timeouts, runs the suite with run-unique test data, and always exports logs and events as CI artifacts.

Impact on cost and engineering time

Spend scales with two numbers: how many environments exist at once and how long each one lives. Bound both.

Set resource requests and limits on every container. Requests let the scheduler pack nodes densely, and limits stop a runaway test from starving its neighbours. In each namespace, add a LimitRange so pods that omit requests get sane defaults. Add a ResourceQuota so a single PR cannot claim a whole node pool. For lifetime, scale Deployments to zero after a stretch with no commits or traffic, and let the reaper enforce a hard TTL.

Illustrative estimate (assumed numbers, not a benchmark)

Formula: environments/day × pods/env × vCPU request/pod × hours alive × price per vCPU-hour

Assume 30 environments a day, 12 pods each, 0.25 vCPU per pod, 6 hours alive, and $0.04 per vCPU-hour:

30 × 12 × 0.25 × 6 × 0.04 = $21.60/day in CPU.

Memory uses the same formula with GiB-hours. Cutting idle lifetime from 6 hours to 2 lowers the CPU figure to $7.20, which is why auto-expiry usually saves more than tuning requests.

Cost is only half the trade-off. An environment that takes 15 minutes to become ready slows every review.

The warm buffer and pre-pulled images cost money while idle. That spend is the dial you turn to trade dollars for feedback time.

Copy# Per-PR namespace guardrails: defaults for pods that omit resources,
# plus a hard ceiling so one PR cannot claim a whole node pool.
apiVersion: v1
kind: LimitRange
metadata:
  name: pr-defaults
  namespace: pr-1234            # one namespace per pull request
spec:
  limits:
    - type: Container
      defaultRequest:           # applied when a container sets no requests
        cpu: 250m               # matches the 0.25 vCPU used in the estimate
        memory: 256Mi
      default:                  # applied when a container sets no limits
        cpu: 500m
        memory: 512Mi
      max:                      # reject any single container above this
        cpu: "1"
        memory: 1Gi
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: pr-budget
  namespace: pr-1234
spec:
  hard:
    pods: "15"                  # 12 expected pods plus headroom for Jobs
    requests.cpu: "4"           # 12 x 250m = 3 vCPU, plus slack
    requests.memory: 4Gi
    limits.cpu: "8"
    limits.memory: 8Gi
    persistentvolumeclaims: "2" # e.g. one database volume, one spare

A LimitRange and ResourceQuota for a single PR namespace that fill in default requests and limits and cap the total CPU, memory, pod count and volume claims the environment can use.

When to avoid ephemeral environments and what to use instead

Per-PR environments pay off when many services change independently and integration bugs keep reaching staging. Some systems never reach that point.

• Small systems. With two or three services, one engineer can run the whole stack on a laptop. The orchestration you would build and maintain costs more than the bugs it catches.
• Dependencies you cannot duplicate. A licensed database, a mainframe link, or a vendor sandbox with one tenant cannot run once per PR. If the most important interaction has to be stubbed, the environment tests your stubs instead of the real integration.
• Slow provisioning. If a preview takes longer to come up than engineers will wait, they merge without looking at it. A preview nobody waits for still costs money and catches nothing.

These approaches complement preview environments. Contract tests in particular reduce how much each preview has to prove.

Even a well-built preview environment does not reproduce production. It runs at small scale, on synthetic traffic, with state that is minutes old. It cannot show you how a migration behaves on real data volumes, how retries and timeouts behave under load, or what happens when a dependency degrades partway through a request. Those still need a pre-release stage, load testing, and progressive rollout.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to ephemeral environments microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.