GitHub Actions CI for Backends: A Reliable Pipeline Template

How GitHub Actions CI is structured: workflows, jobs, steps and runners

A GitHub Actions workflow is a YAML file in .github/workflows/ that runs when an event fires, such as a pull request being opened or a branch being pushed. A workflow contains one or more jobs. Each job runs on a runner, which is either a GitHub-hosted virtual machine or a self-hosted machine. A job is an ordered list of steps, and each step runs a shell command or calls a reusable action. Steps in the same job share a filesystem and run in sequence. Jobs are isolated from each other and run in parallel by default. The needs keyword makes one job wait for another. The GitHub Actions documentation covers these concepts in detail.

This model suits backend and platform teams that already host code on GitHub. CI lives next to the code, is reviewed in pull requests, and needs no separate server.

The main design choice is how to split the pipeline. Lint, unit tests and integration tests should be separate jobs connected by needs:

• Faster feedback. Lint finishes in seconds. If it fails, dependent jobs are skipped, so you don't spend minutes starting database containers for code that won't merge.
• Clearer failures. The pull request checks list shows which stage broke.
• Independent retries. You can re-run only the failed jobs instead of the whole pipeline.

Copy trigger (pull_request / push)
            |
          lint
            |
   +--------+--------+
   |        |        |
 unit     unit     unit      <- matrix: one job per version/OS
   |        |        |
   +--------+--------+
            |
   integration               <- service containers (DB, cache)
            |
        artifacts            <- test reports, build output

Copy  +-------------------------------------------+
  | trigger: pull_request / push              |
  +---------------------+---------------------+
                        |
                        v
              +-------------------+
              | job: lint         |   fails fast; skips
              | runs in seconds   |   dependent jobs
              +---------+---------+
                        | needs: lint
          +-------------+-------------+
          |             |             |
          v             v             v
    +-----------+ +-----------+ +-----------+
    | unit      | | unit      | | unit      |  matrix:
    | version A | | version B | | version C |  one job per
    +-----+-----+ +-----+-----+ +-----+-----+  version/OS
          |             |             |
          +-------------+-------------+
                        | needs: unit (waits for every matrix job)
                        v
  +-------------------------------------------+
  | job: integration                          |
  | runner ---> service containers: DB, cache |
  +---------------------+---------------------+
                        | needs: integration
                        v
              +-------------------+
              | artifacts         |   test reports, build output
              +-------------------+

Job dependency graph for the CI workflow: a failing lint job skips the downstream jobs, unit tests fan out across a matrix, and integration tests run against service containers only after every unit job passes.

Choosing triggers: pull_request, push, path filters and concurrency

Run the full pipeline on pull_request, because that is where it can block a bad merge. By default, a pull_request run checks out a merge commit of the PR head and the base branch. It tests the code as it would look after merging, not the branch on its own. Also trigger on push to main, but nothing else. That run confirms what actually landed, and it gives each commit on main its own result for deploy jobs to depend on. If you also trigger on push to every branch, each PR commit gets tested twice.

Path filters (paths, paths-ignore) skip runs when a change only touches docs or unrelated services. The risk is that GitHub does not report a workflow skipped by path filtering as passed. Its checks stay "Pending." If any of those checks are marked required in branch protection, a docs-only PR can never merge.

To avoid this, keep the workflow trigger unfiltered and filter inside the workflow instead:

• A first, cheap job compares the diff against the base branch and exposes outputs such as backend_changed.
• Expensive jobs use if: on those outputs. A job skipped by a condition reports success, unlike a skipped workflow.
• A final ci-summary job lists every other job under needs, runs with if: always(), and fails if any of them failed or was cancelled.

Make ci-summary the only required check. Then you can rename or add jobs without editing branch protection.

Finally, add a concurrency group keyed on the workflow and ref. Set cancel-in-progress to true only for pull_request events. A new push to a PR makes the earlier run pointless, so cancelling it saves runner time. On main, you want every commit's run to finish.

Copyname: ci
on:
  pull_request:            # no path filters, so required checks always report
  push:
    branches: [main]       # only main; PR branches are covered by pull_request
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  # Cancel superseded PR runs; let every main commit finish
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
  changes:                 # cheap job: decide what actually needs to run
    runs-on: ubuntu-latest
    outputs:
      backend_changed: ${{ steps.diff.outputs.backend_changed }}
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - id: diff
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            BASE="origin/${{ github.base_ref }}"
          else
            BASE="${{ github.event.before }}"
          fi
          if git diff --name-only "$BASE" HEAD | grep -qE '^(services/api/|go\.mod|go\.sum)'; then
            echo "backend_changed=true" >> "$GITHUB_OUTPUT"
          else
            echo "backend_changed=false" >> "$GITHUB_OUTPUT"
          fi
  test:                    # skipped via if: reports success, not pending
    needs: changes
    if: needs.changes.outputs.backend_changed == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make test
  ci-summary:              # the ONLY required check in branch protection
    needs: [changes, test]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - if: contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled')
        run: exit 1

Unfiltered triggers, an in-workflow diff job that gates expensive jobs with if:, PR-only cancel-in-progress, and an always-running ci-summary job to use as the single required check.

Caching dependencies without hiding breakage

For most backends the built-in option is enough. Setting cache: npm on actions/setup-node, or cache: pip on actions/setup-python, caches the package manager's download cache and derives the key from your lockfile. Use actions/cache directly when you need control over paths or keys. Typical cases are a Go module directory, or keys that include the OS and runtime version for matrix builds.

Build the primary key from hashFiles() over the lockfile, for example hashFiles('**/package-lock.json'), plus runner.os. Add restore-keys with the same prefix minus the hash. On a lockfile change, this restores the closest older cache, so the install only downloads the difference.

To avoid hiding breakage, cache downloads rather than installed trees like node_modules or a virtualenv. Keep the install step strict (npm ci, pip install -r against pinned requirements). That way a missing or conflicting dependency still fails the job instead of being masked by a restored directory.

Skip or narrow caching in these cases:

• Slow restores: The cache archive restores slower than a clean install. Compare step timings before you add the cache.
• Overly broad keys: A key without the lockfile hash keeps restoring stale dependencies.
• Cache pressure: Branches and matrix legs multiply entries against the repository's cache size limit. Least recently used entries get evicted, so your hit rate falls exactly when the pipeline is busiest.

Copyname: ci
on: [push, pull_request]

jobs:
  api-node:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # Built-in option: caches npm's download cache, keyed on package-lock.json
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      # Strict install: fails if package.json and the lockfile disagree
      - run: npm ci

  worker-python:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ['3.11', '3.12']
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      # Explicit cache: key includes OS and Python version so matrix legs don't collide
      - uses: actions/cache@v4
        with:
          path: ~/.cache/pip   # download cache only, never the virtualenv
          key: pip-${{ runner.os }}-py${{ matrix.python-version }}-${{ hashFiles('**/requirements.txt') }}
          # Partial hit on a requirements change: reuse the newest cache for this OS/Python
          restore-keys: |
            pip-${{ runner.os }}-py${{ matrix.python-version }}-
      # Pinned requirements still resolve from scratch, so conflicts fail the job
      - run: pip install -r requirements.txt

A workflow that uses setup-node's built-in npm cache for a simple job and actions/cache with a lockfile-hash key and restore-keys for a Python matrix, caching only download caches while keeping strict installs.

Scoping matrix builds and integration tests with service containers

A matrix earns its cost only when each leg tests something production depends on. Define strategy.matrix with the axes you actually ship against, such as two runtime versions and one operating system, then use exclude to drop combinations you never deploy and include to add one-off legs, like the next runtime version, without multiplying the whole grid.

Choose fail-fast on purpose

fail-fast defaults to true, so one failing leg cancels the rest. That saves minutes, but you lose the answer to "does this break everywhere or only on version N?" On pull requests, keep the matrix narrow (usually the production runtime plus the oldest supported one) and accept fail-fast. On pushes to main or a nightly schedule, run the wider matrix with fail-fast: false so every leg reports.

Service containers that wait for readiness

Declare PostgreSQL and Redis under the job's services key. Service containers require Linux runners. Pass Docker health-check flags through options: --health-cmd "pg_isready" for PostgreSQL and --health-cmd "redis-cli ping" for Redis, plus interval, timeout and retry values. The runner waits for each service to report healthy before running steps. This removes the race in which tests start while PostgreSQL is still initializing.

Treat green integration tests with caution. These containers use default configuration, empty datasets, no replication, no TLS and no network latency. A query that passes here can still time out against a large production table. Pin the image tag to the version you run in production to close at least one of those gaps.

Copyname: backend-ci
on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: "0 3 * * *"   # nightly run across the full matrix
jobs:
  integration:
    runs-on: ubuntu-latest   # service containers require a Linux runner
    strategy:
      # On PRs, one failure cancels the other legs to save minutes. On main and nightly runs, every leg reports.
      fail-fast: ${{ github.event_name == 'pull_request' }}
      matrix:
        python: ["3.11", "3.12"]   # oldest supported version + production version
        include:
          - python: "3.13"         # adds one leg without growing the grid
    services:
      postgres:
        image: postgres:16.4      # pin to the version you run in production
        env:
          POSTGRES_PASSWORD: ci-only-password   # throwaway CI value, not a real secret
        ports: ["5432:5432"]
        # Steps start only after this check passes, so tests never hit a database that is still starting up
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s --health-timeout 5s --health-retries 10
      redis:
        image: redis:7.2
        ports: ["6379:6379"]
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 5s --health-timeout 3s --health-retries 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python }}
      - run: pip install -r requirements.txt && pytest tests/integration
        env:
          DATABASE_URL: postgresql://postgres:ci-only-password@localhost:5432/postgres
          REDIS_URL: redis://localhost:6379/0

This workflow runs a small Python version matrix with fail-fast enabled only on pull requests, and gives each leg pinned PostgreSQL and Redis containers that must pass health checks before the integration tests start.

Hardening the pipeline: permissions, pinned actions, secrets and timeouts

Hardening the pipeline

Start with a top-level permissions block that grants contents: read and nothing else. Without it, GITHUB_TOKEN gets whatever default the repository or organization is configured with, which may include write access. Jobs that need more, such as commenting on a pull request, request it at the job level, so any extra scope is visible in review.

Pin every third-party action to a full 40-character commit SHA, with the release tag in a trailing comment for readers. A tag like v4 can be moved to different code at any time. A commit SHA cannot. The cost is that pinned actions never update on their own, so the pipeline can fall behind on security fixes. Assign an owner to updates and route them through Dependabot or a scheduled review, so each bump arrives as a reviewable pull request.

GitHub masks registered secrets in logs. It does not mask values derived from them, such as base64-encoded or concatenated forms. Do not echo those values, and register any derived value with ::add-mask::. Workflows triggered by pull_request from forks receive no secrets, which is the behavior you want. Avoid using pull_request_target to work around this, because it runs untrusted code with your secrets.

Set timeout-minutes on every job. The default is 360 minutes, so a hung integration test can hold a runner for six hours.

Finally, add actions/upload-artifact with if: failure() (or always()) to keep JUnit reports and service-container logs. Engineers can then debug a failure from the stored files instead of rerunning the job and hoping it fails the same way.

Maintaining the template across repositories

Once a second repository needs the same pipeline, copying YAML becomes the default, and each copy then drifts on its own. GitHub Actions provides two ways to share instead.

A reusable workflow declares on: workflow_call and is invoked at the job level with uses: org/ci-templates/.github/workflows/backend.yml@v2. It owns whole jobs, including runner selection, service containers, and the matrix, so it can carry the full template. A composite action bundles steps inside the caller's job. That makes it the right size for a shared "set up toolchain and restore cache" block, but it cannot define services or jobs.

To enforce the result, add the CI job as a required status check in branch protection or a ruleset. Jobs called through a reusable workflow report under a combined name in the form caller-job / called-job. Check the exact name in a real run before you mark it required.

Centralizing the template also centralizes risk. Treat the template repository as a product:

• Version it. Publish semver tags and move a major tag such as v2 only for compatible changes.
• Protect callers that need stability. Let sensitive callers pin to a commit SHA.
• Test it before release. Give the template repository its own CI, with a workflow that calls the template against a small fixture service on every pull request.

When to use it, when to avoid it, alternatives, and impact on cost and engineering time

Where the template fits

The template suits a backend service of moderate size: one deployable per repository, a test suite that needs a real database or queue, and several repositories that should behave the same way. It is overkill for a small library or script repository, where one job running lint and tests is easier to read than a reusable workflow. It is a poor fit for large monorepos, where you want build-graph-aware tooling that tests only the affected targets, because path filters are too coarse for that. It also does not fit jobs that need GPUs, unusual architectures or access to private networks. Those need dedicated self-hosted hardware.

Cost scaling

Illustrative estimate (assumed numbers, not a benchmark)

Formula: minutes = runs × (1 − cancelled) × legs × (hit × warm + (1 − hit) × cold)

Assumed inputs:

• 400 runs per month
• 25% of runs superseded by concurrency cancellation, with partial minutes ignored
• 3 matrix legs
• 8 minutes per leg with a warm cache, 12 minutes cold
• 80% cache hit rate

Result: 400 × 0.75 × 3 × 8.8 = 7,920 minutes.

With no caching and no cancellation the same workload is 400 × 3 × 12 = 14,400 minutes. Multiply either figure by your per-minute rate, which you can find in the current billing documentation linked from GitHub Actions docs. Each additional matrix leg increases the total in direct proportion, so add legs only when they catch real failures.

Engineering time

The time savings come from four places:

• Faster feedback. Cancellation and caching shorten the wait on each push.
• Clearer failure attribution. Separate jobs show whether lint, unit tests or integration tests failed.
• Fewer reruns. Timeouts and pinned dependencies reduce flaky retries.
• Less duplicated YAML. One reviewed template replaces many copies that drift apart over time.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to github actions ci, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.