Microservice Batch API Design: Replacing Chatty Per-Item Calls

What chatty microservices and n+1 service calls look like

A chatty interaction is one where a service makes a separate network call for each item it needs. A typical case: an order service fetches 200 orders, then calls the catalog service once per line item to get product details. This is the n+1 pattern applied across a network boundary. One call returns a list, and N more calls fill it in.

How to detect it

Distributed traces show the pattern clearly. Look for a parent span with dozens or hundreds of near-identical child spans to the same downstream endpoint, differing only in an ID parameter. You can confirm it with metrics. Track outbound calls per inbound request and plot that against result size. If the call count rises linearly as pages get bigger, you have per-item fan-out.

Why it costs so much

Each call pays a fixed overhead: serialization, connection acquisition, possibly a TLS handshake, network transit, and server-side request handling. That overhead is the same whether the payload covers one item or two hundred.

Illustrative estimate (assumed inputs): 200 items, 5 ms per call, run sequentially, gives about 1,000 ms of waiting. Assume one batch call for all 200 IDs takes 20 ms. That is roughly 50x less time spent on the round trip. Running the calls in parallel reduces wall time, but it still opens 200 requests' worth of connections and adds 200 units of load on the downstream service.

A batch API fixes this by accepting many IDs or operations in one request. Backend and platform engineers who own service-to-service calls carry the latency and capacity costs of chatty calls, so they are the ones who should design it.

CopyCHATTY: n+1 calls across a network boundary

 +----------------+  1. GET /orders?page=1   +----------------+
 |                |------------------------->|                |
 |  Order service |<-- 200 orders -----------|  Order store   |
 |                |                          +----------------+
 |                |  2. GET /products/{id}   +----------------+
 |                |=== x200 calls ==========>|                |
 |                |<== x200 responses =======| Catalog service|
 +----------------+                          +----------------+
   Trace:   1 parent span -> 200 near-identical child spans
   Metric:  outbound calls per request grow with page size
   Illustrative estimate: 200 calls x 5 ms (sequential) ~= 1,000 ms

BATCH: one request carries all IDs

 +----------------+  1. GET /orders?page=1   +----------------+
 |                |------------------------->|                |
 |  Order service |<-- 200 orders -----------|  Order store   |
 |                |                          +----------------+
 |                |  2. POST /products:batch +----------------+
 |                |--- ids=[200 IDs] ------->|                |
 |                |<-- 200 products ---------| Catalog service|
 +----------------+                          +----------------+
   Trace:   1 parent span -> 1 child span
   Metric:  outbound calls per request stay flat as pages grow
   Illustrative estimate: 1 batch call ~= 20 ms (assumed)

Per-item lookups fan out into 200 child calls to the catalog service, while a batch API sends all IDs in one request; the timings are illustrative estimates from assumed inputs, not measurements.

Designing the batch contract: reads, writes and request shapes

A batch endpoint is a contract, and most of its design work goes into deciding what one item looks like and how the server reports on each item.

Reads versus writes

For reads, the ID already identifies each result. Return results keyed by ID instead of relying on position, because the server may deduplicate or reorder. For writes, a create has no server ID yet. Each operation therefore carries a client-supplied correlation key, and the response echoes that key so the caller can match outcomes to inputs.

Example schema. The request is { "operations": [ { "key": "c-1", "op": "update", "id": "ord_42", "fields": { "status": "shipped" } } ] }. The response is { "results": [ { "key": "c-1", "status": "ok", "id": "ord_42" } ] }.

Bounding batch size

Unbounded lists turn one request into an unpredictable amount of memory, lock time and downstream fan-out. Document a maximum item count and a maximum payload size. Reject oversized requests up front with a clear error that states the limit, and do not process any part of them. Clients then split large inputs into chunks below the limit, which keeps each request's latency predictable and keeps retries cheap.

Copy// Batch write contract: each operation carries a client-supplied key.
type Operation = { key: string; op: "create" | "update" | "delete"; id?: string; fields?: Record<string, unknown> };
type Result = { key: string; status: "ok" | "error"; id?: string; error?: string };

// Documented limits, published in the API reference.
const MAX_ITEMS = 100;
const MAX_BYTES = 256 * 1024;

// Server: reject oversized requests before processing any operation.
function validateBatch(rawBody: string): { status: number; body: unknown } | null {
  if (Buffer.byteLength(rawBody, "utf8") > MAX_BYTES) {
    return { status: 413, body: { error: "payload_too_large", maxBytes: MAX_BYTES } };
  }
  const { operations } = JSON.parse(rawBody) as { operations: Operation[] };
  if (operations.length > MAX_ITEMS) {
    return { status: 400, body: { error: "too_many_items", maxItems: MAX_ITEMS, received: operations.length } };
  }
  return null; // within limits; proceed to per-item processing
}

// Client: split large inputs into chunks below the item limit.
async function sendInChunks(ops: Operation[], send: (chunk: Operation[]) => Promise<Result[]>) {
  const byKey = new Map<string, Result>();
  for (let i = 0; i < ops.length; i += MAX_ITEMS) {
    const results = await send(ops.slice(i, i + MAX_ITEMS));
    // Match outcomes by echoed key, never by array position.
    for (const r of results) byKey.set(r.key, r);
  }
  return byKey;
}

A TypeScript batch write contract with per-operation correlation keys, server-side rejection of requests over documented size limits, and client-side chunking that matches results by key.

Partial failure responses and atomic versus best-effort semantics

Once a request passes validation, the response must report on every item. Return one result entry per input, keyed by the correlation key the client supplied (a client-generated ID or the item's natural key) rather than by array position. Positional matching breaks as soon as the server deduplicates, reorders or parallelises work. Each entry carries a status. Failed entries also carry a stable, machine-readable error code such as NOT_FOUND or VERSION_CONFLICT, plus a message for humans. Clients branch on the code and log the message.

Add an explicit top-level outcome: succeeded, partial or rejected (the request was invalid and nothing ran). Do not leave clients to infer this from one HTTP code. A common bug is a client that sees 200 and assumes every item worked, so failures are silently dropped. To prevent this, use a distinct status for mixed results, such as 207 Multi-Status, and document that clients must read the outcome field.

Atomic versus best-effort

State the mode in the contract with a required request field or a separate endpoint per mode, never as an implicit default. Atomic mode over large batches holds locks longer and increases contention. If the items span services, atomic mode turns a simple endpoint into a coordination problem.

Copy// Contract for a best-effort batch response (mode is declared in the request).
type ItemResult =
  | { key: string; status: "ok" }
  | { key: string; status: "failed"; code: "NOT_FOUND" | "VERSION_CONFLICT" | "INVALID_ITEM"; message: string };

interface BatchResponse {
  mode: "atomic" | "best_effort";      // echoed back so the client knows which rules applied
  outcome: "succeeded" | "partial" | "rejected";
  results: ItemResult[];                // one entry per input, keyed by client-supplied key
}

// Client: decide from `outcome`, never from the HTTP code alone.
async function updateBatch(items: { key: string; price: number }[]): Promise<string[]> {
  const res = await fetch("", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ mode: "best_effort", items }),
  });
  // 200 = all succeeded, 207 = mixed, 400 = rejected; the body is still authoritative.
  const body = (await res.json()) as BatchResponse;

  if (body.outcome === "rejected") {
    throw new Error("Batch rejected; nothing was applied");
  }

  // Match by key, not array index: the server may dedupe or reorder.
  const retryable: string[] = [];
  for (const r of body.results) {
    if (r.status === "ok") continue;
    console.warn(`item ${r.key} failed: ${r.code} - ${r.message}`);
    if (r.code === "VERSION_CONFLICT") retryable.push(r.key); // re-read, then resend only these
    // NOT_FOUND and INVALID_ITEM are permanent: surface them, do not retry.
  }
  return retryable;
}

A typed best-effort batch response with a top-level outcome and per-item error codes, and a client that reads the outcome rather than trusting the HTTP status, matches results by correlation key, and collects only the retryable failed keys.

Idempotency and safe retries for bulk endpoints

A single idempotency key on the whole request is not enough. If the client retries only the failed items, the retry is a different payload, so a request-level key either rejects it as a mismatch or misses duplicates. Use two scopes:

• Request key: identifies the batch submission. It lets the server return the stored response when the identical batch is resent, and gives clients a handle for status lookups.
• Item key: generated by the client per item and kept stable across every retry. The server deduplicates on this key, so an item that already committed is never applied twice, whichever batch carries it.

Store each item key with its result in the same transaction as the write itself. If the key is recorded separately, a crash between the two steps opens a window where the side effect exists but the dedup record does not. If a key arrives with a different payload than the one stored, reject that item instead of silently returning the old result.

To retry, resend only items that failed with retryable errors such as conflicts or upstream unavailability, using their original item keys. Validation failures will fail again, so do not retry them.

A timeout is different. The client loses visibility into every item, not just some. The server may have committed half the batch before the connection dropped. There are two safe recovery paths:

• Resend the entire batch with the same keys. Committed items return their stored results, and the rest execute.
• Query a status endpoint by request key or item keys before deciding what to resend.

Both paths depend on a server-side retention window that is longer than the client's total retry horizon.

Copy-- PostgreSQL. Item keys live in the same database as the business write.
CREATE TABLE batch_item_results (
  item_key     text PRIMARY KEY,           -- client-generated, stable across retries
  request_key  text NOT NULL,              -- batch submission handle for status lookups
  payload_hash text NOT NULL,              -- detects reuse of a key with a different payload
  status       text NOT NULL,              -- 'committed' or 'rejected'
  result       jsonb,
  created_at   timestamptz NOT NULL DEFAULT now()  -- purge only after the client's full retry horizon
);
CREATE INDEX ON batch_item_results (request_key);

-- Per item: claim the key, apply the side effect, and record the result atomically.
BEGIN;
INSERT INTO batch_item_results (item_key, request_key, payload_hash, status)
VALUES ('itm_01HZX4K9QW', 'req_01HZX4K2MB', 'sha256:9f2c1a', 'committed')
ON CONFLICT (item_key) DO NOTHING
RETURNING item_key;
-- No row returned: the key already exists. ROLLBACK, then read the stored row.
-- Same payload_hash: return its stored result. Different hash: reject this item.

-- Row returned: this item is new. Write it in the same transaction.
INSERT INTO orders (customer_id, sku, quantity)
VALUES (4821, 'SKU-1042', 3)
RETURNING id;

UPDATE batch_item_results
SET result = jsonb_build_object('order_id', 99173)  -- id returned by the insert above
WHERE item_key = 'itm_01HZX4K9QW';
COMMIT;  -- the key and the side effect commit together or not at all

-- After a client timeout, recover per-item status before deciding what to resend.
SELECT item_key, status, result
FROM batch_item_results
WHERE request_key = 'req_01HZX4K2MB'
   OR item_key = ANY (ARRAY['itm_01HZX4K9QW', 'itm_01HZX4KA3R']);
-- Items with no row never committed. Resend them with their original item keys.

PostgreSQL sketch that claims each item key in the same transaction as its write, rejects reuse of a key with a different payload, and lets a client that timed out look up per-item status by request key or item keys.

Implementing bulk endpoints on the server and the client

Server side

A bulk endpoint should issue set-based queries, such as one WHERE id IN (...) per table, instead of looping over the single-item lookup. Otherwise you have moved the N+1 problem from the network into the database. Set-based queries return rows in arbitrary order, so the contract must state the order. Either reassemble results to match the input order, or return a map keyed by ID and say that order is not guaranteed. Deduplicate IDs before querying, and document whether duplicates appear once or once per occurrence in the response. Report missing IDs explicitly as per-item "not found" entries. Silently omitting them forces clients to diff lists.

Run authorization per item, but in bulk. Load ownership or ACL data for the whole set in one query, then filter. Return "not found" for items the caller cannot see, so the response does not reveal that they exist.

Route the single-item endpoint through the bulk path with a list of one. That way validation, authorization and error mapping cannot diverge between the two.

Client side

A data-loader style aggregator collects lookups for a few milliseconds, deduplicates them and sends one batch request. Split large sets into chunks under the server's limit, and run chunks with bounded concurrency so a single caller cannot saturate the service.

These patterns have costs. The collection window adds latency on purpose. A single trace span also covers many logical requests, so you need to tag spans with item counts and log per-item errors against the IDs that caused them.

When to use batch APIs, when to avoid them, and the alternatives

A batch endpoint fits when one caller needs many items from the same owner in one request cycle, the IDs are known up front, and the round-trip count is what slows it down. It is the wrong fix when the cost is in the work done per item, not in the network hops.

Watch tail latency. A batch response returns only when its slowest item does. If the server fans out to shards, one slow shard delays results that were ready milliseconds earlier. Set a per-item deadline and return a partial result rather than waiting.

Large batches also put load on resources. A 10,000-element IN list can produce a poor query plan or exceed driver parameter limits. Buffering the full request and response raises memory use for every concurrent caller. Enforce a maximum batch size, cap payload bytes and rate-limit by item count, not by request count. Without these limits, one client can do the work of thousands of requests while counting as one.

Batching also loses HTTP caching. A per-item GET caches cleanly by URL. A batch of arbitrary IDs almost never repeats exactly. You may still need a per-item cache on the server that the batch handler reads from.

Impact on development cost and engineering time

A batch endpoint is not free. Every resource that gains a bulk variant roughly doubles its API surface: another route, another schema, more documentation, and a second set of contract tests. Partial failure handling has to be tested on purpose, with mixed success and error payloads, duplicate IDs and oversized requests. Clients need code that parses per-item results and retries only the failed items. The server needs limit enforcement on item count, payload size and per-tenant quotas. That work lands on teams before any latency improves.

The savings appear later and recur. Fewer fan-out loops means fewer tail-latency incidents and fewer timeout cascades during traffic spikes. Connection pools, sidecars and downstream databases see fewer, larger requests instead of bursts of small ones. Debugging is easier when the contract states what a batch guarantees: one request ID, one log entry with per-item outcomes, and no need to reassemble a trace from hundreds of spans.

Keeping the cost from multiplying

Most of the ongoing cost comes from every team building batch support in its own way. Three choices prevent that:

• One shared code path. The single-item endpoint calls the batch handler with a list of one. Validation, authorization and persistence logic exist once, so a fix applies to both.
• A standard per-item error format. If every service reports item index, error code and retryability the same way, client parsing and dashboards are written once.
• A reusable client aggregator. A shared library that chunks requests to the server's limits, merges results and retries failed items means product teams use batching without re-implementing it.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservice batch api design, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.