Secrets Management Backend vs Env Vars: Config vs Secret Guide

Config vs secret: where to draw the line

A value is a secret if exposing it causes harm on its own. Three tests cover most cases. Does it grant access to a system? Does it let someone act as your service? Does it let someone decrypt or forge data? Passwords, API keys, OAuth tokens, TLS private keys and JWT signing keys all pass at least one test. So does a connection string with embedded credentials: postgres://app:hunter2@db:5432/orders is a password with a hostname attached.

Configuration changes behaviour but grants nothing. Hostnames, feature flags, timeouts, pool sizes and log levels are visible to anyone who can watch the system run.

The twelve-factor test is whether the codebase could be made open source at any moment without compromising credentials. It catches hardcoded secrets. Once secrets are out of the code, it says nothing about who can read them, how they are audited, or how they are rotated.

Misclassification costs you in both directions:

• Everything treated as a secret: changing a timeout requires an access request, and ordinary values disappear from code review.
• Credentials treated as config: they get pasted into dashboards, tickets and debug logs, where nobody tracks them.

Why env vars are a delivery mechanism, not a secrets manager

That failure mode is easy to fall into because env vars feel like a secrets store. They aren't one. An environment variable is a string the kernel copies into a process when it starts, and that's all it is. The twelve-factor methodology recommends env vars because they keep config out of the codebase and work in any language (12factor.net/config). That solves a delivery problem. It doesn't solve the problem of protecting a secret.

The limits show up once you have more than a few services. Access control stops at the process boundary, so every library and subprocess in the process can read every variable. Nothing records which code read a value or when. Values have no versions, so after a bad change you can't tell what the old value was or who set it. There's also no rotation: to pick up a new value, the process has to restart.

How values leak

• Crash dumps contain the process's memory, and the environment is part of that memory.
• Debug and diagnostic endpoints sometimes print the whole environment to help with troubleshooting.
• Child processes get a copy of the parent's environment by default. That includes shell-outs, helper binaries and anything those programs start in turn.
• Build and CI logs pick up values when a script echoes them, runs in verbose mode or fails with its full context printed.
• /proc/<pid>/environ on Linux shows the environment a process was started with. Only the same user or root can read it. On a shared host, though, the "same user" can be every service running under that account.

The example below shows a child process inheriting a variable and the same value being read back from /proc.

This is acceptable when env vars are the last step of delivery: something trustworthy fetches the secret and injects it just before the process starts. For a small team running one service with low-value credentials, plain env vars are often good enough. Once you have several services, a value that has to be rotated or an auditor asking who read it, you need a backend behind them.

Copy#!/usr/bin/env bash
# Shows that a child process inherits env vars and that the environment
# can be read back from /proc. Run this on Linux.

# Placeholder value. Never put a real credential in a demo.
export DB_PASSWORD='example-not-a-real-secret'

# 1. Children inherit by default. The parent didn't pass anything explicitly.
bash -c 'echo "child sees: DB_PASSWORD=$DB_PASSWORD"'

# Grandchildren inherit it too, through every level of shell-outs.
bash -c 'bash -c "echo \"grandchild sees: \$DB_PASSWORD\""'

# Opting out has to be done on purpose, one call at a time.
env -u DB_PASSWORD bash -c 'echo "scrubbed child sees: ${DB_PASSWORD:-<unset>}"'

# 2. Start a long-running process that stands in for a service.
sleep 300 &
pid=$!

# The file is mode 0400 and owned by the process's user. Only that user
# or root can read it, but on a shared host that may be every service.
ls -l "/proc/$pid/environ"

# Entries are NUL-separated. Turn them into lines and pick out the secret.
tr '\0' '\n' < "/proc/$pid/environ" | grep '^DB_PASSWORD='

# Changing the variable in the parent now does not touch the copy that
# /proc reports. That copy is what the process was started with.
unset DB_PASSWORD
tr '\0' '\n' < "/proc/$pid/environ" | grep '^DB_PASSWORD='

kill "$pid"

In this example, a child and a grandchild process inherit an exported variable, the variable is withheld from one child on purpose with env -u, and a running process's startup environment is read back from /proc//environ, where it is still there after the parent unsets it.

What a secrets management backend must provide and how it works

…a dedicated secrets management backend. At minimum it must encrypt values at rest and enforce identity-based access policies scoped to each service. It must log every read and write, keep versions so a bad change can be rolled back, and expose APIs for rotation and revocation. Stronger backends also issue short-lived, per-workload credentials on demand, such as a database user that expires in an hour, which leaves no long-lived credential to steal.

Workload identity and the secret-zero problem

If a service needs a static bootstrap token to log in to the backend, that token is just another secret you have to deliver somehow. Workload identity removes it. The service presents an identity the platform already vouches for, such as an orchestrator service account token, a cloud instance role or an mTLS client certificate. The backend then verifies that identity with whoever issued it.

Copy[Platform] --issues identity--> [Workload]
                                    |
                     1. present identity
                                    v
                            [Secrets backend] --verify--> [Identity issuer]
                                    |
                     2. evaluate policy (service, env)
                     3. issue secret / short-lived credential
                                    |
                                    +--> [Audit log: who, what, when]

What it does not fix

The backend becomes both an availability dependency and a security dependency. If services fetch secrets at startup, an outage of the backend stops them from booting, so plan for redundancy and caching. Encryption and audit logs also do nothing against a compromised workload that is authorized to read a secret, because the attacker reads it through the same legitimate path. You still need least privilege and fast revocation.

Policy granularity needs care as well. Twelve-factor's advice against named environments still applies to plain config, but backend policies are usually scoped by environment or by service. A broad rule like "prod reads prod/*" grants too much. One policy per secret per service becomes too much work to maintain.

Copy+----------------------+   1. issue identity  +-----------------------+
| Platform             |--------------------->| Workload              |
| SA token / instance  |                      | (service A, env prod) |
| role / mTLS cert     |                      +-----------------------+
+----------------------+                         |               ^
           ^                         2. present  |               |
           |                           identity  v               |
           |               +--------------------------------+    |
           +--3. verify ---| Secrets backend                |    |
                           |  auth: verify identity         |    | 4. secret /
                           |            |                   |    | short-lived
                           |            v                   |    | credential
                           |  policy: service=A, env=prod   |    |
                           |  may read db/creds?            |    |
                           |            | allow             |    |
                           |            v                   |    |
                           |  issue secret or short-lived   |----+
                           |  credential (TTL, revocable)   |
                           +--------------------------------+
                                        | 5. record every read/write
                                        v
                           +--------------------------------+
                           | Audit log: who, what, when     |
                           +--------------------------------+

The workload authenticates with a platform-issued identity instead of a static bootstrap token, and the backend verifies it, checks a service- and environment-scoped policy, issues a secret or short-lived credential, and records the access in the audit log.

Delivery patterns: env injection, file mounts, runtime fetch

A secret reaches the process in one of three ways, and each one trades portability against how quickly it picks up changes.

Env injection. A deploy or init step reads the backend and sets env vars before the process starts. The app needs no code change, which makes this the most portable option. The values are fixed for the life of the process, so rotating a secret means restarting or redeploying. Env vars are also inherited by child processes and often end up in crash reports and debug output.

File mounts. Secrets are written to an in-memory filesystem such as tmpfs and protected with file permissions. When a secret changes, the file can be updated in place. The app only sees the new value if it re-reads the file, either on a timer or when it detects a change.

Runtime fetch. The app calls the backend through an SDK, a sidecar or a local agent. This allows fine-grained access control and live updates. If the fetch happens at startup, though, a backend outage becomes failed deploys and failed autoscaling. Caches and agents reduce that risk, but they bring back local copies, and those copies can go stale and add exposure.

Use env injection for secrets that rarely rotate. Use file mounts when you need rotation without a restart. Use runtime fetch for short-lived or per-request credentials, and only when the backend is as available as your deploy pipeline.

Designing for secret rotation

Rotation is a design requirement. If an app can only pick up a new credential by restarting, every rotation becomes a deploy, and teams put off deploys.

Allow two valid credentials at once

The backing system must accept both the old and new credential during an overlap window. Most databases support this through two users or two passwords per role. Without the overlap, there is a gap between revoking the old value and every instance loading the new one, and requests fail in that gap.

Re-read without restarting

Load secrets so they can be refreshed in place. Common approaches:

• Watch the mounted file for changes.
• Cache the value with a TTL and re-read the file when it expires.
• Have the connection pool rebuild connections with fresh credentials.

The example that follows uses the TTL approach. Open connections keep working, and only new connections use the new value.

Run rotation in a fixed order

1. Create the new secret.
2. Distribute it to every consumer.
3. Verify each consumer authenticates with it.
4. Revoke the old one.

Prefer short-lived dynamic credentials

Use them where the backing system can issue them. A leaked credential then expires on its own, and nothing has to be revoked by hand.

Failure modes

• No overlap window. Rotating without one causes outages.
• Shared secrets. When several services share one secret, coordinating the overlap means tracking every consumer. This is a strong reason to issue one credential per service.
• Dynamic credential load and timing. Dynamic credentials add issuance load on the backend. They also bring clock and TTL edge cases, such as a batch job whose credential expires partway through.
• Unrehearsed rotation. Teams skip rotation when nobody has practiced it. Schedule a rotation drill so the first real rotation is not an emergency.

Copyimport threading
import time
from pathlib import Path

class RotatingSecret:
    """Re-reads a mounted secret file once its TTL expires."""

    def __init__(self, path, ttl_seconds=60, on_change=None):
        self._path = Path(path)
        self._ttl = ttl_seconds
        self._on_change = on_change  # called with the new value when it differs
        self._lock = threading.Lock()
        self._value = None
        self._loaded_at = 0.0

    def get(self):
        with self._lock:
            # Monotonic clock, so wall-clock jumps cannot stall or force refreshes.
            if self._value is None or time.monotonic() - self._loaded_at >= self._ttl:
                try:
                    new_value = self._path.read_text().strip()
                except OSError:
                    if self._value is None:
                        raise  # nothing cached yet, so fail loudly
                    new_value = self._value  # keep the last good value and retry next TTL
                self._loaded_at = time.monotonic()
                if new_value != self._value:
                    self._value = new_value
                    if self._on_change:
                        self._on_change(new_value)
            return self._value

pool = None

def rebuild_pool(password):
    # New connections use the new credential. The old one stays valid during
    # the overlap window, so in-flight work on the old pool can finish.
    global pool
    old, pool = pool, make_pool(user="app_orders", password=password)  # placeholder factory
    if old is not None:
        old.close()  # close after draining, per your driver's pool semantics

db_password = RotatingSecret("/var/run/secrets/db/password", ttl_seconds=60, on_change=rebuild_pool)

A Python loader caches a mounted secret file with a TTL, re-reads it when the TTL expires, keeps the last good value if a read fails, and rebuilds the connection pool only when the value changes.

Keeping plain configuration simple, and hygiene that prevents leaks

Non-sensitive per-deploy values, such as hostnames, feature flags, pool sizes and timeouts, belong in env vars or in versioned config files that go through the same review as code. They need no access policy or rotation. Routing them through a secrets backend makes changes slower and harder to see, and it hides the real secrets among dozens of harmless ones.

A settings manifest keeps the boundary honest

Over time the line between config and secret drifts. A small manifest checked into the repo prevents that. It lists every setting, marks it config or secret, names an owning team, and for secrets records the backend path. The YAML example below shows the shape. CI can fail a build when code reads a setting the manifest does not list. A reviewer who sees a new secret entry knows to ask about rotation and access.

Hygiene that prevents leaks

• Never log the config object or dump the environment. Debug output and crash reporters are common leak paths.
• Mask secret values in CI output, and do not echo them in scripts.
• Scan repositories and built images for committed credentials, both in pre-commit hooks and in CI.
• Never put secrets in Dockerfile build args or image layers. Layers persist and can be read by anyone who can pull the image.
• Write a revocation runbook before an incident: who revokes, where, and how dependent services recover.

These habits save time in three places. Reviews have a clear rule to check against. New engineers can read the manifest instead of reverse-engineering where each value comes from. During an incident, responders can see right away which credentials are exposed and who owns each one.

Copy# settings.yaml: every setting the service reads, checked into the repo.
# CI fails the build if code reads a setting that is not listed here.
# Changes to this file get the same review as code.
service: orders-api
settings:
  - name: DATABASE_HOST
    kind: config            # non-sensitive, can go in env vars or config files
    owner: platform-team
    example: orders-db.internal.example.com

  - name: DB_POOL_SIZE
    kind: config
    owner: orders-team
    example: "20"

  - name: FEATURE_NEW_CHECKOUT
    kind: config
    owner: checkout-team
    example: "false"

  - name: DATABASE_PASSWORD
    kind: secret            # never logged, never in images or build args
    owner: platform-team
    backend_path: kv/orders-api/prod/database   # where the value lives
    rotation: 90d           # reviewers ask about this for every new secret
    revocation_runbook: docs/runbooks/revoke-database-password.md

  - name: PAYMENT_PROVIDER_API_KEY
    kind: secret
    owner: checkout-team
    backend_path: kv/orders-api/prod/payments
    rotation: 30d
    revocation_runbook: docs/runbooks/revoke-payment-key.md

This settings manifest marks each value as config or secret and names the team that owns it. For secrets it also records the backend path, how often to rotate, and where the revocation runbook is, so reviewers, new engineers and incident responders can all check one file.

Migrating from everything-in-env-vars: cost, effort and alternatives

That manifest is also where migration starts. Here, a "phase" means a step you can stop after. Each phase leaves the system working, so the work can spread over several sprints.

Phase 2 is where most of the risk goes away, and the app doesn't have to change. Do phase 3 only for credentials whose leak would be expensive, such as database admin accounts, payment keys and signing keys.

The costs show up at different times. Changing loaders and writing policies is upfront work. After that, you have to run the backend, patch it and keep it highly available, because a backend outage can block deploys. In return, revoking a leaked credential takes a single action, rotation stops being a calendar reminder, and audits read one access log instead of reconstructing history.

Who should adopt a full backend, and who should wait

A full backend pays off for teams with several services, shared credentials, or audit and compliance requirements. A single small service, or a team that can't keep another critical system highly available, should wait. Until then, these lighter options are enough:

• Orchestrator-native secret objects, with encryption at rest turned on.
• Encrypted files committed to the repo, decrypted at deploy time.
• Platform-provided parameter stores from your hosting provider.

All three still keep config and secrets separate. Most of them also make it easier to move to a full backend later.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to secrets management backend, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.