How to Measure Engineering Time Saved by Tooling, Honestly

What honest measurement of time saved is, and why 'hours saved' numbers mislead

Honest measurement of time saved means comparing observed time before and after a tooling change, against a baseline you captured first. The common alternative is to guess a saving per use and multiply it by a usage count. That method is easy to produce and almost impossible to falsify. If "saves 10 minutes" is wrong, the error is multiplied by every invocation, and nobody checks whether those minutes existed in the first place.

Illustrative estimate (assumed numbers, not a benchmark): claimed hours = guessed minutes per use × uses ÷ 60. Assume 10 minutes × 3,000 uses ÷ 60 = 500 hours. If the real saving is 2 minutes, the claim overstates by 400 hours. Nothing in the calculation would reveal that.

A baseline-first approach separates three quantities. Each one behaves differently:

Time freed is not the same as time reallocated. A build that is five minutes faster does not become five minutes of feature work. Engineers fill the gap with context switches, reviews or breaks, and that is often a reasonable trade. Claim the reduction you measured, not a productivity gain you inferred.

This practice is for platform and backend engineers who build internal tooling and must justify headcount or infrastructure spend. A defensible small number holds up better in a budget review than a large one that falls apart under questioning.

When to measure, and when the honest answer is 'inconclusive'

Measurement pays off when it happens before the change, not after. The best time to set it up is ahead of a rollout of CI changes, environment provisioning or deploy tooling. At that point you can still record how long the current workflow takes under normal conditions. It is also worth the effort when a real decision depends on the answer: funding a team, renewing infrastructure spend, or deprecating a tool that people complain about but still use. If no decision will change based on the result, a lighter check is usually enough.

When to scale it down

Some situations call for less rigor, or for no number at all:

• The tool is already rolled out and there is no baseline. You can reconstruct one from logs or ask people what things used to be like, but both are weaker evidence. Reconstructed baselines inherit whatever the old logs failed to capture. Retrospective self-reports drift toward the story people already believe.
• The team is small. With a handful of engineers, one person's vacation or one unusual incident can move the average more than the tool does.
• The workflow is infrequent. A quarterly release or a monthly environment rebuild gives too few samples. Normal variance will swamp any real difference.
• Detailed tracking would damage trust. Per-person timing can feel like surveillance. Engineers then change how they work, or what they report, and the data stops reflecting reality.

In these cases, report the result as "inconclusive" and say why. That is more useful in a budget review than a confident figure built on a sample that cannot support it.

How it works: defining the unit of work and capturing a baseline

Start by choosing one workflow that an engineer repeats often enough to produce dozens of samples a week. Good candidates are push to green CI, clone to a working local environment, and merge to running in production. "Developer productivity" is not a workflow, and neither is "onboarding." You cannot timestamp either of them.

Then write the start and end events as precisely as a contract. "Push to green" could mean the first push on a branch, or it could mean every push. The end event could be the final required check passing, or it could be the moment someone notices. Pick one definition, write it down, and keep it fixed for both the baseline and the comparison period.

Most of the data already exists:

• CI job records for queued, started, and finished times.
• VCS events for push, review requested, approval, and merge.
• Deploy logs for rollout start and health check pass.
• Ticket transitions for in progress and done. Treat these as noisy, because people update tickets late.

Split each interval into segments, because tooling usually shrinks only one of them:

Push → [queue: waiting] → [build/test: machine] → [red: hands-on fix] → [rerun: machine] → green → [unnoticed: waiting]

Report the median and the p90, not the mean. Build and deploy times are long-tailed. A handful of hour-long flaky runs can pull the mean far away from what a typical engineer experiences. The median shows the normal case and the p90 shows the pain, and you can watch each one move independently.

Copy+-----------+  +-----------+  +-------------+  +---------------------+
| VCS events|  | CI jobs   |  | Deploy logs |  | Ticket transitions  |
| push, PR, |  | queued,   |  | rollout     |  | in progress, done   |
| approve,  |  | started,  |  | start,      |  | (noisy: late edits) |
| merge     |  | finished  |  | health pass |  |                     |
+-----+-----+  +-----+-----+  +------+------+  +----------+----------+
      |              |               |                    |
      +--------------+---------------+--------------------+
                                     v
                     +-------------------------------+
                     | Events per sample, under one  |
                     | fixed start/end definition    |
                     +---------------+---------------+
                                     v
   push    started    red      fixed        green           noticed
   |       |          |        |            |               |
   +=======+----------+########+------------+===============+
   | queue | build,   | fix    | rerun      | idle after    |
   | WAIT  | MACHINE  |HANDS-ON| MACHINE    | green  WAIT   |
   |       | tests    | config |            |               |
   +-------+----------+--------+------------+---------------+
                                     v
              +----------------------------------------------+
              | Per segment and total, across all samples:   |
              | report median (typical) and p90 (the tail),  |
              | not the mean, which flaky long runs skew     |
              +----------------------------------------------+
   Bar key:  = waiting    - machine    # hands-on

Existing CI, VCS, deploy and ticket timestamps feed one fixed push-to-noticed-green interval, which is split into waiting, machine and hands-on segments and summarized per segment by median and p90.

Implementing it: queries, sampling and comparison groups

Querying the event table

The query below takes a table of CI or deploy events, each with a start and end timestamp, and groups them by week and pipeline. For each group it returns the run count, the median duration, and the p90 duration. Filter out cancelled runs and keep retries as separate rows. Retries are part of the wait an engineer actually sits through. Keep the run count next to each percentile, because a p90 computed from a dozen runs moves too much from week to week to tell you anything.

Sampling hands-on time

Event logs show machine time. They don't show the twenty minutes someone spent reading a failed build log. To capture that, use a short survey sent every week or two ("roughly how long did your last release take you, hands-on?"), or ask a few volunteers to log one task type for a fixed window, such as two weeks. People misremember durations, so treat these answers as a range that bounds your estimate, and check them against the event data. Don't rely on self-reports alone.

Comparison groups

Build times also change for reasons unrelated to your tool: new hardware, a code freeze, a quiet holiday month. A staggered rollout handles this, because teams that don't have the tool yet act as the control group for teams that do. If you can't stagger, choose comparison repos with similar size and change volume.

Report only at the team or pipeline level. Tracking individuals makes people change how they work, and the figures stop measuring anything.

Copy-- Weekly median and p90 duration per pipeline from CI/deploy events.
-- Assumed table: ci_runs(run_id, pipeline, status, started_at, finished_at)
-- Uses PostgreSQL syntax (percentile_cont ... WITHIN GROUP, date_trunc).
WITH runs AS (
  SELECT
    pipeline,
    date_trunc('week', started_at) AS week,
    -- Duration in minutes. Retries stay as separate rows because they are
    -- part of the wait an engineer actually sits through.
    EXTRACT(EPOCH FROM (finished_at - started_at)) / 60.0 AS duration_min
  FROM ci_runs
  WHERE status <> 'cancelled'          -- cancelled runs are not real waits
    AND finished_at IS NOT NULL        -- skip runs still in progress
    AND started_at >= now() - INTERVAL '26 weeks'
)
SELECT
  pipeline,
  week,
  COUNT(*) AS run_count,               -- always report next to percentiles
  ROUND(CAST(percentile_cont(0.5) WITHIN GROUP (ORDER BY duration_min)
             AS numeric), 1) AS median_min,
  ROUND(CAST(percentile_cont(0.9) WITHIN GROUP (ORDER BY duration_min)
             AS numeric), 1) AS p90_min,
  -- Flag weeks with too few runs for a stable p90 (threshold is a judgment call).
  COUNT(*) < 30 AS low_sample
FROM runs
GROUP BY pipeline, week
ORDER BY pipeline, week;

A PostgreSQL query that groups non-cancelled CI or deploy runs by pipeline and week, then returns the run count, median and p90 duration, and a flag for weeks with too few runs to trust the p90.

Impact on development cost: a worked estimate with labeled assumptions

Illustrative estimate (assumed numbers, not a benchmark). Every input below is an assumption. Replace each one with a value from your own baseline.

• Minutes saved per run: 4 (assumed)
• Blocking runs per engineer per day: 6 (assumed)
• Engineers: 40 (assumed)
• Blocking fraction, meaning the share of runs where the engineer actually waits: 0.5 (assumed)

Formula: minutes saved × runs × engineers × blocking fraction → 4 × 6 × 40 × 0.5 = 480 engineer-minutes/day, which is 8 engineer-hours.

The table below changes one input at a time and holds the others at their expected values.

The combined row spans more than 30×. That spread is the honest answer about how much the estimate depends on its inputs.

Next, subtract costs. Use one amortization period, for example 12 months. Count build hours, ongoing maintenance, migration of every repository, onboarding, and on-call and support load in engineer-hours. Count infrastructure in money. If these costs take up half of the gross hours, report the net figure.

Don't multiply the minutes by salary. Saved time comes in two- to four-minute pieces, and people rarely spend those on output that ships. A per-minute salary figure treats every saved minute as delivered work, so it overstates the value.

Induced demand works against you as well. When the pipeline gets faster, engineers push more often, so run count and compute cost both rise. Measure runs per engineer after rollout. Don't assume it stays at the baseline.

Avoiding vanity metrics and checking that time was actually saved

Adoption and invocation counts tell you that people use the tool. They don't tell you the tool is worth using. A dashboard that multiplies invocations by "4 minutes saved per run" turns a usage count into a value claim without testing it. The constant was guessed once and never checked, and the total rises with usage even when each run saves nothing. Report invocations as a usage metric and nothing more.

Pair any time-saved figure with outcomes you can verify independently:

• Lead time for changes: if engineers save time, merged work should reach production sooner.
• Deploy frequency: saved time that turns into output shows up as more shipped changes.
• Change failure rate: this tells you whether the speed came at the expense of correctness.
• Queue length (open reviews, pending tickets): this shows whether work moved downstream instead of disappearing.

The last two matter because optimizing the measured step often pushes cost into steps nobody is measuring. Faster code generation can produce larger diffs, which slow reviewers. A quicker deploy path can increase rollbacks and on-call pages. If review latency or incident hours rise while authoring time falls, the saving is partly an accounting transfer between teams.

Measure again after the novelty period, once usage has settled and early enthusiasts no longer dominate the sample. One of three things will have happened. The saving held, it became throughput (visible in deploy frequency or lead time), or it was absorbed into meetings, context switching or slack time. Absorption is not a failure, but report it as absorption, not as hours returned.

Alternatives and how to report the result

Each way of measuring supports a different level of confidence, so pick the method based on the decision the number has to support.

The staggered rollout is usually the only option that separates the tool's effect from everything else that changed that quarter. That is why it is worth the delay when the budget decision is large. Surveys and dashboards are better used to explain a measured effect than to produce one.

Reporting template

• Estimate: low, expected and high hours per engineer per month, with the expected value stated as a range rather than a point.
• Range drivers: the two or three inputs that move the result most, such as task frequency, baseline duration and absorption rate.
• Evidence that would narrow it: for example, a second rollout cohort, a longer baseline window or timing data from a task type that is currently self-reported.
• Qualitative benefits: listed next to the numbers, not in an appendix, with a one-line source for each (interview, onboarding retro, on-call notes).
• Confidence statement: one sentence naming the method behind the estimate and its main weakness.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to measure engineering time saved, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.