How is CloudWatch Logs cost calculated?

The bill has three main parts, each priced per GB:

  • Ingestion: every GB sent to a log group, charged once on arrival.
  • Storage: GB-months of archived data, billed on the compressed size and charged every month the data is kept.
  • Logs Insights queries: GB of data scanned, so broad time ranges and large log groups cost more per query.

A retention policy only changes storage. Ingestion is the same whether you keep logs for a day or forever. That is why teams that only set retention often see a smaller drop in the bill than they expected.

Illustrative estimate (assumed numbers, not a benchmark). Assume 100 GB/day ingested, a 30-day month, a stored size of 25% of raw (compression), an ingestion rate I = 1 unit/GB and a storage rate S = 0.06 unit/GB-month. These rates are symbolic, not AWS prices.

monthly cost = (daily GB × 30 × I) + (stored compressed GB × S) + (GB scanned × query rate)

Each month adds 750 GB of stored data. With "Never expire" the stored total grows by that amount every month. With 30-day retention it stays at about 750 GB.

Month Ingestion (both) Storage, Never expire Storage, 30-day
1 3,000 45 45
12 3,000 540 45
36 3,000 1,620 45

Ingestion dominates early and scales linearly with volume. Storage only becomes a large share when retention has been unbounded for years. Recalculate this with current pricing for your region.

Which log groups are driving your CloudWatch Logs bill?

Before touching retention or log classes, find out which log groups actually generate the spend. In most accounts a small number of groups produce most of the ingested volume, so changes elsewhere save little.

Rank groups by ingested bytes

The AWS/Logs namespace publishes an IncomingBytes metric with a LogGroupName dimension. Pull it with the Sum statistic over the last 30 days, one data point per day, then sort the log groups by total. Daily granularity matters because it separates a steady producer from a group that spiked once during an incident. A steady producer needs a permanent fix. A spike may need nothing. The companion IncomingLogEvents metric helps distinguish groups with a high event count from groups with a few very large events. Those two patterns call for different emission fixes.

For storage, each log group reports its current stored bytes in the log group description. Sorting groups by that value shows where an unbounded retention setting has been accumulating data for a long time.

Cross-check against billing

Metrics show volume, but billing shows cost. In Cost Explorer, filter to the CloudWatch service and group by usage type. Ingestion, archived storage, and Logs Insights scanning appear under separate usage types, typically containing DataProcessing-Bytes, TimedStorage-ByteHrs, and DataScanned-Bytes. Vended logs appear separately. This tells you whether the problem is mostly ingestion, storage, or query cost, and that determines which of the following sections applies to you.

The result should be a short list of log groups, each labeled with its main cost driver. Spend engineering time only on that list.

# Rank log groups by 30-day ingested bytes and show stored bytes alongside.
import boto3
from datetime import datetime, timedelta, timezone

logs, cw = boto3.client("logs"), boto3.client("cloudwatch")
end = datetime.now(timezone.utc)
start = end - timedelta(days=30)

groups = [g for p in logs.get_paginator("describe_log_groups").paginate() for g in p["logGroups"]]
stored = {g["logGroupName"]: g.get("storedBytes", 0) for g in groups}
total, peak = {}, {}

# GetMetricData accepts up to 500 queries per request, so batch the groups.
for i in range(0, len(groups), 500):
    batch = groups[i:i + 500]
    queries = [{
        "Id": f"g{j}",  # ids must start with a lowercase letter
        "MetricStat": {
            "Metric": {"Namespace": "AWS/Logs", "MetricName": "IncomingBytes",
                       "Dimensions": [{"Name": "LogGroupName", "Value": g["logGroupName"]}]},
            "Period": 86400,  # one data point per day: steady producer vs one-off spike
            "Stat": "Sum"}} for j, g in enumerate(batch)]
    for page in cw.get_paginator("get_metric_data").paginate(
            MetricDataQueries=queries, StartTime=start, EndTime=end):
        for r in page["MetricDataResults"]:
            name = batch[int(r["Id"][1:])]["logGroupName"]
            total[name] = total.get(name, 0) + sum(r["Values"])
            peak[name] = max([peak.get(name, 0)] + r["Values"])

GB = 1024 ** 3
print(f"{'log group':60} {'ingest GB':>10} {'peak day %':>10} {'stored GB':>10}")
for name, b in sorted(total.items(), key=lambda kv: kv[1], reverse=True)[:20]:
    # A high peak-day share suggests an incident spike rather than a steady producer.
    share = 100 * peak[name] / b if b else 0
    print(f"{name:60} {b / GB:10.2f} {share:10.1f} {stored[name] / GB:10.2f}")

A boto3 script that sums the daily IncomingBytes metric per log group over 30 days, flags spiky groups by their peak-day share, and prints the top 20 next to each group's stored bytes.

How do you set log retention and avoid the 'Never expire' default?

New log groups default to Never expire, and that includes the ones services create for you. A Lambda function writes to /aws/lambda/<function-name> on its first invocation, and other services auto-create groups the same way. Nobody chose a retention period for these groups, so their storage grows indefinitely.

To fix an existing group, run aws logs put-retention-policy --log-group-name <name> --retention-in-days <n>. The value must be one of the fixed periods CloudWatch accepts. Cleaning up afterward is reactive, though, and it misses every group created after the cleanup. A better approach is to declare each log group in your infrastructure-as-code with an explicit retention value, and to pre-create service log groups before the service does. If the Lambda log group already exists when the function first runs, the function writes to your group with your retention instead of an unmanaged one.

Choose each period based on who needs the data and when:

Driver Question to ask Typical outcome
Incident debugging How far back do engineers actually look during an incident? Short (days to weeks)
Audit and compliance What does policy or regulation require, and does it require these specific logs? Long, often archived elsewhere
Query patterns Do dashboards or reports query older data? Match the longest real query window

Shortening retention deletes older data permanently. If a slow-burning incident or a late-discovered security breach turns out to have started before the new cutoff, the evidence will already be gone. Before you change retention in bulk, get agreement from security and compliance owners, and record the reasoning for each period.

AWSTemplateFormatVersion: '2010-09-09'
Description: Pre-create a Lambda log group with explicit retention

Parameters:
  FunctionRoleArn:
    Type: String
    Description: ARN of an existing execution role for the function

Resources:
  # Declare the group Lambda would otherwise auto-create with 'Never expire'.
  # The name must match /aws/lambda/<function-name> exactly.
  OrdersApiLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /aws/lambda/orders-api
      # 30 days covers the incident debugging window agreed with on-call.
      # Must be one of the fixed values CloudWatch Logs accepts.
      RetentionInDays: 30

  OrdersApiFunction:
    Type: AWS::Lambda::Function
    # Create the log group first so the first invocation writes into
    # this managed group instead of creating an unmanaged one.
    DependsOn: OrdersApiLogGroup
    Properties:
      FunctionName: orders-api
      Runtime: python3.12
      Handler: index.handler
      Role: !Ref FunctionRoleArn
      Code:
        ZipFile: |
          def handler(event, context):
              print("order received")
              return {"statusCode": 200}

A CloudFormation template that declares the Lambda function's log group with a 30-day retention and makes the function depend on it, so the group never falls back to the 'Never expire' default.

What is the Infrequent Access log class and what do you give up?

Infrequent Access is a log class chosen per log group. It charges a lower per-GB ingestion price than Standard in exchange for fewer features. Because ingestion is usually the largest line on the bill, the class affects cost more than retention does for high-volume, rarely read logs. Do not rely on remembered numbers. Check the current CloudWatch pricing page for each region you run in, because rates differ by region.

Infrequent Access log groups still store events, honor retention policies, and can be queried with CloudWatch Logs Insights. You can still investigate an incident after the fact. What you lose are the features that act on events as they arrive.

Feature support has changed over time and can differ by region. Treat this table as a checklist to confirm against current documentation, not as a reference:

Capability Standard Infrequent Access (verify)
Storage and retention policies Yes Yes
Logs Insights queries Yes Yes
Metric filters Yes Check; historically not supported
Subscription filters (streaming to other services) Yes Check; historically not supported
Live Tail Yes Check; historically not supported
Other real-time or analysis features (e.g. anomaly detection, embedded metric extraction) Yes Check each one

The log class is set when the log group is created. You cannot switch an existing group in place, so moving to Infrequent Access means creating a new group and repointing producers to it.

Rule of thumb: if an alarm, metric, or streaming pipeline depends on a log group, keep it Standard. Missing metric filters or subscriptions do not cause an error. The alarm stays quiet and the downstream system stops receiving data, which is the failure you least want to find during an incident.

When should you use Infrequent Access, and when should you avoid it?

The groups that benefit most from Infrequent Access are large, quiet ones: verbose debug traces, request dumps, or audit-style records that you keep because you might need to search them, not because anything reads them continuously. If a group has no metric filters, no alarms, and no subscription filters, and someone queries it only a few times a quarter, the lower ingestion price is close to pure savings. The higher the volume, the larger the savings, so start with the groups that topped your bill in the earlier analysis.

Avoid Infrequent Access for any group that feeds metric filters, alarms, subscription filters, or a downstream pipeline. Also avoid it when you are unsure how the group will be used later. That uncertainty matters because the log class is set when the group is created and cannot be changed afterward. To move a group between classes, you create a new log group and repoint every producer at it. The new name then has to fit your naming conventions, your IaC modules need a class parameter, and IAM policies scoped to the old ARN need updating. A wrong guess costs a migration, not a setting change.

Splitting Standard and Infrequent Access groups also has ongoing operational costs:

  • You have more log groups to manage.
  • Routing logic moves into agents or logging libraries.
  • An engineer can search the wrong group during an incident. A query against the Standard group returns nothing, and that empty result looks like evidence. Document the split in runbooks and dashboards so responders know which group holds which data.
#!/usr/bin/env bash
# Pre-migration check: only move a log group to Infrequent Access if nothing
# reads it continuously. The class cannot be changed in place, so this
# creates a NEW group that producers must be repointed to.
set -euo pipefail

OLD_GROUP="/app/orders/debug"
NEW_GROUP="/app/orders/debug-ia"   # must still fit your naming convention
RETENTION_DAYS=30

# Metric filters (and any alarms built on them) are not supported by IA.
mf=$(aws logs describe-metric-filters --log-group-name "$OLD_GROUP" \
  --query 'length(metricFilters)' --output text)

# Subscription filters feed downstream pipelines; IA is the wrong fit here too.
sf=$(aws logs describe-subscription-filters --log-group-name "$OLD_GROUP" \
  --query 'length(subscriptionFilters)' --output text)

if [ "$mf" -ne 0 ] || [ "$sf" -ne 0 ]; then
  echo "$OLD_GROUP has $mf metric filter(s), $sf subscription filter(s): keep Standard." >&2
  exit 1
fi

# Log class is fixed at creation time.
aws logs create-log-group \
  --log-group-name "$NEW_GROUP" \
  --log-group-class INFREQUENT_ACCESS

aws logs put-retention-policy \
  --log-group-name "$NEW_GROUP" \
  --retention-in-days "$RETENTION_DAYS"

# Confirm the class, then repoint agents/libraries, update IAM policies scoped
# to the old ARN, and record the split in the incident runbook.
aws logs describe-log-groups --log-group-name-prefix "$NEW_GROUP" \
  --query 'logGroups[].[logGroupName,logGroupClass,retentionInDays]' --output table

A shell check that refuses to migrate a log group with metric or subscription filters, then creates a separate Infrequent Access group with retention, since the class cannot be changed on an existing group.

How can you reduce log ingestion at the source?

Log class and retention change what you pay per gigabyte after the data arrives. Emitting fewer bytes lowers ingestion cost, which is usually the largest line item, under either class.

Drop debug output in production

Debug and trace statements are written for a developer stepping through code, not for an operator reading thousands of requests. Set the production default to INFO or WARN. Keep the level configurable at runtime, for example through an environment variable or a config reload, so you can raise it for one service during an investigation without a redeploy.

Sample the success path

A health check or a successful cache hit logged on every request produces a large volume of nearly identical lines. Log a fixed fraction of successes and always log errors, slow requests, and anything above a latency threshold. The example below routes records through a filter that passes everything at WARN and above and keeps only a small percentage of INFO records from designated high-volume handlers.

Trim payloads and vended logs

Full request and response bodies inflate every line and often contain data you should not store. Log identifiers, sizes, and status codes instead.

For VPC flow logs, capture only the traffic type you analyze (REJECT alone is often enough for security review). Use a custom format to drop fields you never query. Consider whether the destination needs to be CloudWatch Logs at all.

The trade-off

Rare failures are often explained by the ordinary events that came before them. A blanket level change or aggressive sampling can delete that context. Sample by event type, never by severity. Propagate a request ID so that when one request fails, its full trace is kept.

import logging
import os
import random

# Level comes from the environment so one service can be raised to DEBUG without a redeploy.
LOG_LEVEL = os.environ.get("LOG_LEVEL", "INFO").upper()
SAMPLE_RATE = float(os.environ.get("LOG_SAMPLE_RATE", "0.01"))  # keep 1% of sampled successes
SLOW_MS = 500  # always keep requests slower than this
# Sample by event type: only these high-volume, low-value loggers are thinned.
SAMPLED_LOGGERS = {"app.healthcheck", "app.cache"}


class SuccessPathSampler(logging.Filter):
    def filter(self, record):
        if record.levelno >= logging.WARNING:
            return True  # never sample warnings or errors
        if getattr(record, "duration_ms", 0) >= SLOW_MS:
            return True  # slow requests are kept even on the success path
        if record.name in SAMPLED_LOGGERS:
            return random.random() < SAMPLE_RATE
        return True  # everything else at INFO passes unchanged


handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s"))
handler.addFilter(SuccessPathSampler())  # handler filters also see records from child loggers

root = logging.getLogger()
root.setLevel(LOG_LEVEL)
root.addHandler(handler)

log = logging.getLogger("app.cache")
# Log identifiers, sizes and status, not the payload itself.
log.info("cache hit key_id=%s bytes=%d", "user-1234", 2048, extra={"duration_ms": 3})
log.warning("cache miss storm key_prefix=%s", "user-")  # always emitted

A Python logging setup that reads its level from an environment variable, always keeps warnings, errors and slow requests, and samples only INFO records from designated high-volume loggers.

What are the alternatives for long-term retention and query cost control?

Once emission is under control, the remaining question is where logs should live after they stop being operationally useful. Many teams keep everything in CloudWatch for a year or more because it is the simplest option, but that means paying CloudWatch storage rates for data that is rarely read.

Concern Keep in CloudWatch Export to S3
Storage cost Higher per GB Lower, especially with colder S3 storage classes
Query path Logs Insights, same console and IAM Separate engine and schema setup
Freshness Near real time Export tasks are batch jobs, not a stream
Lifecycle Retention setting per log group S3 lifecycle rules

The usual pattern is short retention in CloudWatch for incident response, plus scheduled exports to S3 for audit or compliance. If you need continuous delivery to S3, use a subscription filter feeding a streaming pipeline rather than export tasks. Budget engineering time for the query tooling on the S3 side, because it does not come for free.

Query habits also matter. Logs Insights bills on data scanned, so a dashboard that refreshes every minute across a 30-day window and a dozen log groups can cost more than the storage it reads. This applies to Infrequent Access groups too. If you moved a group to Infrequent Access because it is rarely read, frequent broad queries against it can erase the savings. Narrow the time range first, select only the log groups you need, and filter early in the query.

Prioritized checklist

  1. Measure. Rank log groups by ingested and stored bytes before changing anything.
  2. Set retention. Replace every "Never expire" setting with a deliberate value.
  3. Choose the log class. Move groups to Infrequent Access only when they are not used for real-time features and queries against them are rare and narrowly scoped.
  4. Reduce emission. Cut verbose and duplicate logging at the source, keep request IDs, and sample by event type.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to cloudwatch logs cost, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.