How is CloudWatch Logs cost calculated, and why does ingestion usually dominate?
CloudWatch Logs bills on several independent dimensions. Rates vary by region and change over time, so check the current AWS CloudWatch pricing page for your region before modelling anything.
| Dimension | Billed on | What drives it |
|---|---|---|
| Ingestion | GB ingested (rate varies by log class) | Every byte your services emit |
| Archived storage | GB-month stored, measured after compression | Retention period × volume |
| Logs Insights | GB scanned per query | Query time range and how many log groups each query covers |
| Vended logs (e.g. VPC Flow Logs) | Separate, often tiered, delivery rate | AWS-generated log volume |
| Subscription filters | Charges from the destination service | Streaming logs to Kinesis, Firehose or Lambda |
Ingestion usually dominates because you pay its rate on every byte, once, at full uncompressed size. Storage is charged on compressed data at a much lower per-GB rate.
Illustrative estimate (assumed numbers, not a benchmark). These inputs are assumed and are not real rates:
- 1,000 GB ingested per month
- $0.50/GB ingestion
- $0.03/GB-month storage
- 5:1 compression
- 30-day retention
Formulas, with the example worked through:
- Ingestion: GB_ingested × ingest_rate = 1,000 × 0.50 = $500
- Storage (steady state): (GB_ingested ÷ compression_ratio) × months_retained × storage_rate = (1,000 ÷ 5) × 1 × 0.03 = $6
Under these assumptions, even 12-month retention would add only about $72/month. Ingestion would still be the bulk of the bill.
This matters most for two kinds of team:
- High-traffic services that log per request. Volume scales linearly with traffic.
- Lambda-heavy accounts. Every invocation writes START, END and REPORT lines plus any debug output from your function.
For both, cutting volume at the source saves far more than tightening retention. Set retention policies anyway, but start by reducing volume.
How do you find which log groups are driving the bill?
Before you change anything, find out where the bytes actually come from. The bill tells you how much you spend. It does not tell you which service is responsible.
Start with Cost Explorer
In Cost Explorer, filter by the CloudWatch service and group by usage type. Ingestion appears as usage types containing DataProcessing-Bytes, prefixed by a region code. Storage and Logs Insights scans appear as separate usage types. This view confirms whether ingestion is your largest line item and which regions it comes from. Cost Explorer stops there, because it does not break spend down by log group.
Rank log groups by IncomingBytes
For per-group attribution, use the IncomingBytes metric in the AWS/Logs namespace, which has a LogGroupName dimension. Sum it over a representative window, such as the last 14 or 30 days, and sort in descending order. In most accounts, a few log groups account for most of the volume. Because log group names usually follow conventions like /aws/lambda/<function> or a service prefix, you can map those top groups to the services and teams that produce them.
Tag and assign owners
Tag every log group with at least a team or owner tag and a service tag. A ranked list without owners turns into a spreadsheet that nobody acts on. With owners, each heavy log group has a person who can decide whether its debug output is still needed.
Keep the ranking current
A snapshot of IncomingBytes goes stale as traffic grows, new services launch, or someone enables verbose logging in production. Set an AWS Budgets alert on CloudWatch spend so you hear about a jump before the invoice arrives. Re-run the ranking on a schedule, such as monthly, because a cost review done once becomes outdated quickly.
# Rank log groups by IncomingBytes over the last 30 days (one region).
# Run once per region that shows DataProcessing-Bytes in Cost Explorer.
import boto3
from datetime import datetime, timedelta, timezone
DAYS = 30
REGION = "us-east-1" # placeholder: set per region
logs = boto3.client("logs", region_name=REGION)
cw = boto3.client("cloudwatch", region_name=REGION)
# Collect every log group name in the region.
names = [g["logGroupName"]
for page in logs.get_paginator("describe_log_groups").paginate()
for g in page["logGroups"]]
end = datetime.now(timezone.utc)
start = end - timedelta(days=DAYS)
totals = {}
# GetMetricData accepts up to 500 queries per request, so batch the groups.
for i in range(0, len(names), 500):
batch = names[i:i + 500]
queries = [{
"Id": f"q{j}",
"MetricStat": {
"Metric": {"Namespace": "AWS/Logs", "MetricName": "IncomingBytes",
"Dimensions": [{"Name": "LogGroupName", "Value": n}]},
"Period": DAYS * 86400, # one bucket covering the whole window
"Stat": "Sum"}} for j, n in enumerate(batch)]
pages = cw.get_paginator("get_metric_data").paginate(
MetricDataQueries=queries, StartTime=start, EndTime=end)
for page in pages:
for r in page["MetricDataResults"]:
name = batch[int(r["Id"][1:])]
totals[name] = totals.get(name, 0) + sum(r["Values"])
# Print the heaviest groups; map them to owners via naming and tags.
for name, b in sorted(totals.items(), key=lambda kv: kv[1], reverse=True)[:20]:
print(f"{b / 1024**3:10.2f} GiB {name}")
A boto3 script that sums the AWS/Logs IncomingBytes metric per LogGroupName over 30 days and lists the 20 log groups ingesting the most data, so you can re-run it on a schedule to keep the ranking current.
How do you reduce log volume at the source without losing incident data?
The biggest lever is the number of bytes your services emit, because every line is billed at ingestion before retention or log class matters.
Set production log levels deliberately. Run at INFO or WARN, and make DEBUG something you enable per service or per request when needed, not a permanent default.
Sample the success path, never the failures. Routine 200 responses are repetitive, so keep a fraction of them. Always log errors, timeouts, and requests slower than your latency threshold. Make that decision at the end of the request, once status and duration are known.
Drop full request and response payloads. Log identifiers, sizes, status codes, and a trace ID instead. Bodies are large, rarely read, and often contain personal data you shouldn't be storing anyway.
Remove duplicate logging. If both the application and a proxy sidecar log every request, you pay for the same event twice. Choose one source for access logs.
Keep JSON compact. Use short, consistent keys, omit null fields, and skip pretty-printing.
Illustrative estimate (assumed inputs): 200 requests/second × 500 bytes per line × 2,592,000 seconds in a 30-day month ≈ 259 GB/month. Suppose you keep 10% of successful requests and all failures, and 2% of requests fail. Then 0.98 × 0.10 + 0.02 ≈ 12% of the original volume is retained, or about 31 GB/month.
Over-aggressive cuts can remove the lines you need during an incident. Before reducing volume, check the last few postmortems to see which fields were actually used. The payoff goes beyond the bill: smaller, consistent logs make queries faster and easier to read when you are under pressure.
import json
import random
import sys
SUCCESS_SAMPLE_RATE = 0.10 # keep 10% of routine successful requests
SLOW_MS = 1000 # always keep requests slower than this threshold
def should_log(status, duration_ms):
# Decide at the end of the request, once status and duration are known.
if status >= 400 or duration_ms >= SLOW_MS:
return True # errors, timeouts (e.g. 504) and slow requests are always kept
return random.random() < SUCCESS_SAMPLE_RATE
def log_request(trace_id, method, route, status, duration_ms,
req_bytes, resp_bytes, user_id=None, err=None):
if not should_log(status, duration_ms):
return
record = {
"ts_trace": trace_id, # correlate with traces instead of logging bodies
"m": method,
"r": route, # route template, not the full URL with query params
"st": status,
"ms": round(duration_ms),
"in": req_bytes, # payload sizes only, never the payloads themselves
"out": resp_bytes,
"uid": user_id,
"err": err,
# Record the rate so dashboards can scale sampled counts back up.
"rate": 1.0 if status >= 400 or duration_ms >= SLOW_MS else SUCCESS_SAMPLE_RATE,
}
# Omit null fields and use compact separators: no spaces, no pretty-printing.
compact = {k: v for k, v in record.items() if v is not None}
sys.stdout.write(json.dumps(compact, separators=(",", ":")) + "\n")
# Example calls from request middleware:
log_request("4bf92f3577b34da6", "GET", "/orders/{id}", 200, 42.7, 0, 1834)
log_request("a3ce929d0e0e4736", "POST", "/orders", 504, 30012.0, 912, 0,
user_id="u-1029", err="upstream timeout")
A stdlib-only Python request logger that makes the sampling decision after the request finishes, always keeps errors and slow requests, keeps 10% of routine successes, logs sizes and a trace ID instead of payloads, and writes compact JSON with null fields omitted.
When should you use the Infrequent Access log class, and when should you avoid it?
Infrequent Access (IA) is a log class with a lower per-GB ingestion price than Standard, in exchange for fewer features. Since ingestion is usually the largest line item, moving a high-volume group to IA saves more than any retention change. The catch is that the class is set when the log group is created and cannot be changed later.
IA fits logs that people read occasionally and that nothing watches automatically. Examples include audit trails, access logs kept for compliance, and verbose debug output you query during an investigation. You can still search these with Logs Insights. You just can't build real-time automation on them.
| Capability | Standard | Infrequent Access |
|---|---|---|
| Ingestion price | Higher | Lower |
| Logs Insights queries | Yes | Yes |
| Metric filters | Yes | Historically no |
| Live Tail | Yes | Historically no |
| Change class after creation | No | No |
Keep a group on Standard if it has metric filters, if anyone tails it with Live Tail during deploys, or if an alarm depends on it. In those cases the savings do not justify losing the signal.
Because the class is fixed, migrating means replacing the group:
- Create a new IA log group.
- Repoint the producer, such as the Lambda function, ECS task definition, or agent config.
- Update your infrastructure-as-code so the next apply does not recreate the old group.
The main risk is a silent alarm. A metric filter left on the old group stops receiving data. Depending on how the alarm treats missing data, it may sit in INSUFFICIENT_DATA or OK and never fire. Before migrating, list every filter and subscription attached to the group. AWS has expanded IA support over time, so check its current feature-support list rather than relying on this table.
Standard, Infrequent Access, or S3: what are the alternatives for each log group's purpose?
Pick the destination by asking who reads the log, how quickly they need it, and how long it must exist.
Routing by purpose Producers (services, Lambda, containers) → Standard log group, short retention → on-call search, metric filters, alarms Audit / compliance sources → Infrequent Access log group → occasional Logs Insights lookups Direct-to-S3 sources (e.g. VPC Flow Logs) → S3 bucket → lifecycle rules → archive storage class → Athena-style queries
| Purpose | Destination | Main trade-off |
|---|---|---|
| Operational debugging | Standard, short retention | Highest ingestion price, but full feature support |
| Audit, rarely queried | Infrequent Access | Cheaper ingestion; fewer features, and the log class can't be changed later |
| Long-term archive | S3 with lifecycle rules | Cheapest storage; no native search |
S3 is cheapest per gigabyte retained, but you give up the console search experience. Ad hoc queries become Athena-style scans, which are billed by data scanned. Without date-based partitioning and a compressed columnar or line-delimited format, every query reads the whole bucket. You also take on bucket policies, lifecycle transitions and object-count overhead yourself.
Delivering directly from the source to S3 avoids CloudWatch ingestion completely. A subscription filter does not. The data is ingested into CloudWatch first. You then pay for the transport (Firehose delivery or Lambda invocations) and for the destination. If the CloudWatch copy also keeps long retention, you are paying for storage twice.
When a subscription filter feeds S3, set the source log group's retention to a few days. That leaves enough history to recover from a delivery failure without duplicating the archive.
PRODUCERS DESTINATION READERS
+---------------+ +------------------------+
| Audit / |---->| Infrequent Access |---> occasional Logs
| compliance | | log group | Insights lookups
+---------------+ +------------------------+
+---------------+ +------------------------+
| Services, |---->| Standard log group |---> on-call search,
| Lambda, | | short retention | metric filters,
| containers | +-----------+------------+ alarms
+---------------+ |
| subscription filter: ingested first,
v then Firehose/Lambda transport cost
+------------------------+
| Firehose or Lambda |
+-----------+------------+
|
+---------------+ v
| Direct-to-S3 | +------------------------+
| sources, e.g. |---->| S3 bucket |
| VPC Flow Logs | | date-partitioned, |---> Athena-style queries
| (no CW ingest)| | compressed | (billed per data
+---------------+ +-----------+------------+ scanned)
|
| lifecycle rules
v
+------------------------+
| Archive storage class |
+------------------------+
Routing logs by purpose: operational logs go to Standard with short retention, audit logs go to Infrequent Access, and archives go to S3 either directly from the source or through a subscription filter, which incurs CloudWatch ingestion first.
How do you enforce a log retention policy across every log group?
New log groups default to "Never expire," and many are created by something other than your IaC. Lambda creates /aws/lambda/<function-name> on first invocation, and other services and agents do the same. Without enforcement, those groups keep accumulating storage indefinitely.
Audit first. aws logs describe-log-groups omits the retentionInDays field for groups set to never expire. Filtering for that absent field gives you the list of unbounded groups. Paginate through every region you use, because log groups are regional.
Fix with aws logs put-retention-policy. It takes --log-group-name and --retention-in-days. Only a fixed set of day values is accepted (1, 7, 14, 30, 90, and so on), so map your policy to valid values.
Prevent recurrence in two layers:
- Infrastructure-as-code. Declare each Lambda's log group explicitly, with retention set, before the function first runs. If the function has already been invoked, the group already exists, and you have to import it into your IaC state rather than create it.
- Scheduled remediation. A periodic job finds groups without retention and applies a default. This catches groups created outside IaC.
Safeguards for the remediation job:
- Run it in dry-run mode first so it reports changes without making them.
- Honor an exclusion tag, such as a
retention-exemptkey, and a prefix allowlist. - Never shorten a retention period that is already set. Only fill in missing ones.
- Get written sign-off on retention periods from whoever owns the audit or regulatory obligations. Engineering should not pick these numbers alone.
Retention only reduces storage. It does nothing for ingestion, which is usually the larger share of the bill, so expect modest savings unless old data has piled up for years.
#!/usr/bin/env bash
# Fill in retention only for log groups that have none ("Never expire").
# Existing retention settings are never touched, so nothing gets shortened.
# Usage: DRY_RUN=false ./enforce-retention.sh (defaults to dry run)
set -euo pipefail
DRY_RUN="${DRY_RUN:-true}"
DEFAULT_DAYS=30 # Must be a value CloudWatch accepts; confirm with compliance owner
REGIONS="us-east-1 eu-west-1" # Log groups are regional: list every region you use
SKIP_PREFIXES="/audit/ /security/" # Prefixes managed elsewhere (e.g. regulated retention)
EXEMPT_TAG="retention-exempt" # Tag key that opts a group out of remediation
for region in $REGIONS; do
# The CLI paginates automatically. A missing retentionInDays field matches null.
groups=$(aws logs describe-log-groups --region "$region" \
--query 'logGroups[?retentionInDays==null].logGroupName' --output text | tr '\t' '\n')
for lg in $groups; do
skip=false
for p in $SKIP_PREFIXES; do [[ "$lg" == "$p"* ]] && skip=true; done
if $skip; then echo "[$region] skip (prefix): $lg"; continue; fi
# Check for the exemption tag. Outputs None when the tag is absent.
exempt=$(aws logs list-tags-log-group --region "$region" --log-group-name "$lg" \
--query "tags.\"$EXEMPT_TAG\"" --output text)
if [[ "$exempt" != "None" ]]; then echo "[$region] skip (tagged): $lg"; continue; fi
if [[ "$DRY_RUN" == "true" ]]; then
echo "[$region] DRY RUN would set ${DEFAULT_DAYS}d: $lg"
else
aws logs put-retention-policy --region "$region" \
--log-group-name "$lg" --retention-in-days "$DEFAULT_DAYS"
echo "[$region] set ${DEFAULT_DAYS}d: $lg"
fi
done
done
A scheduled remediation script that finds log groups with no retention in each region and skips exempt prefixes and tagged groups. It reports changes in dry-run mode by default and only fills in missing retention. It never shortens retention that is already set.
How do you control Logs Insights query costs and metrics derived from logs?
Logs Insights bills per gigabyte of log data scanned, and the scanned set is determined by the log groups you select and the time range you choose. The two highest-impact habits are querying only the log groups that can contain the answer and starting with the narrowest time window, widening only if needed. Putting a filter early in the query makes it faster and its results smaller, but it does not reduce the bytes scanned, so it is no substitute for a tight scope. Dashboards are the hidden cost. A Logs Insights widget re-runs its query each time the dashboard loads or refreshes, so a 24-hour query across many log groups on a wall display is billed again on every refresh.
Logging a line only so a metric filter can count it is an expensive way to produce a number. You pay ingestion on every line, and the counter still becomes a custom metric.
| Option | You pay for | Development cost |
|---|---|---|
| Metric filter | Log ingestion, plus the custom metric | Lowest; no code change, but the filter breaks silently if the log format changes |
| Embedded metric format | Log ingestion of structured JSON, plus the custom metric | Moderate; requires structured log output, and the data stays queryable as logs |
| Direct metric API | API requests, plus the custom metric; no log ingestion | Highest; requires an SDK or agent, plus batching to keep request counts down |
Illustrative estimate (assumed numbers, not a benchmark): monthly ingestion cost = lines per month × bytes per line ÷ 1,073,741,824 × ingestion price per GB. Assume 500 million lines of 200 bytes each. That is about 93 GB per month ingested only to increment a counter. Batched direct metric calls avoid that ingestion cost entirely.
A repeatable CloudWatch Logs cost-review checklist
Cost work that only happens after a surprising invoice tends to repeat the same investigation each time. A short review on a fixed schedule, monthly for most teams and weekly during large migrations, catches regressions while they are still small and keeps the effort to a predictable block of time.
The checklist
- Rank log groups by ingestion. Sort by the
IncomingBytesmetric over the last 30 days and compare the result with the previous review. A group that jumped in the ranking usually points to a recent deploy, a new debug flag, or a retry loop. Investigate it before it becomes the new baseline. - Flag groups with no retention. Any group set to "Never expire" keeps accumulating storage charges indefinitely. Treat each one as a defect unless someone has documented a reason for it.
- Check each group's class against its actual use. If an Infrequent Access group now feeds alarms or live tailing, it is in the wrong class. If a Standard group has had no queries or consumers in months, it is a candidate for Infrequent Access.
- Audit subscription filters and saved queries. Remove filters whose destination no longer exists or that nobody reads. Retire dashboard widgets that rerun broad Logs Insights scans on every refresh.
- Assign an owner to every high-volume group. Record a team name in a tag. A group without an owner keeps growing because nobody is responsible for reducing it.
- Set budget alerts. Create a budget scoped to CloudWatch with a forecast-based alert. That way, drift between reviews gets flagged early instead of being discovered at month end.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to cloudwatch logs cost, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.