What are AWS Budgets and AWS Cost Anomaly Detection, and how do they work?
AWS Budgets compares actual or forecasted spend against a fixed threshold you choose, such as "alert at 80% of $5,000 this month." AWS Cost Anomaly Detection has no fixed number. It learns a spending baseline for each dimension you monitor, such as a service, linked account, cost category or cost allocation tag. It then flags spend that deviates from that pattern, even if the total is still far below any budget.
Both tools read the same Cost and Usage billing data. AWS processes that data in batches several times a day, not as a stream. So every billing-based alert describes money that has already been spent. Budgets and anomaly detection cover different failure shapes, but neither is fast.
| AWS Budgets | Cost Anomaly Detection | |
|---|---|---|
| Trigger type | Fixed threshold on actual or forecasted spend | Statistical deviation from a learned baseline |
| Catches | Slow overruns, month-end surprises | Sudden spikes that stay under the budget line |
| Setup inputs | Amount, period, scope filters, threshold percentages | Monitor dimension, alert threshold (absolute or percentage), notification frequency |
| Latency | Hours after usage | Hours after usage, plus the detection run |
| Typical noise | Static amounts that drift out of date as the business grows | Planned launches, migrations, new services with little history |
The layered detection path
Resource usage → CloudWatch usage metrics and alarms (minutes) → billing pipeline, Budgets and Cost Anomaly Detection (hours) → SNS topic → chat channel or pager → named owner with a runbook
The CloudWatch layer reacts to usage before the billing data catches up. The billing layer confirms the spend in dollars. An alert only leads to action when it reaches a named owner who has a runbook to follow.
+-----------------------------+
| Resource usage |
+--------------+--------------+
|
+-----------------+-----------------+
| |
v v
+-----------------------------+ +--------------------------------+
| FAST LAYER (minutes) | | SLOW LAYER (hours) |
| CloudWatch usage metrics | | Billing pipeline, batched |
| and CloudWatch alarms | | several times a day |
| Reacts to usage before | | +-----------+ +--------------+ |
| billing data catches up | | | Budgets | | Cost Anomaly | |
+--------------+--------------+ | | (fixed | | Detection | |
| | | threshold)| | (baseline) | |
| | +-----------+ +--------------+ |
| +----------------+---------------+
| |
+-----------------+-----------------+
|
v
+-----------------+
| SNS topic |
+--------+--------+
|
v
+------------+------------+ +----------------------+
| Chat channel or pager |--->| Named owner + runbook|
+-------------------------+ +----------------------+
Layered cost detection: CloudWatch alarms react to usage within minutes, Budgets and Cost Anomaly Detection confirm spend in dollars hours later from batched billing data, and both feed one SNS topic that routes to chat or paging and a named owner with a runbook.
Why do billing alerts arrive too late to stop cost spikes?
Billing alerts evaluate cost data, and AWS does not produce cost data in real time. Usage is metered, aggregated, and priced in batches before it appears in Cost Explorer, Budgets, or Cost Anomaly Detection. That can take hours. Anomaly detection also needs enough of the new pattern to separate it from normal variation, which adds more delay. By the time a billing alert fires, the spike has been running for a while.
The cost of an incident depends on how long it runs before someone stops it:
Cost of incident = spend rate × (detection lag + time to fix)
Illustrative estimate (assumed numbers, not a benchmark):
- A misconfigured job adds $40/hour.
- The first billing alert arrives about 12 hours after the job starts.
- The on-call engineer needs 2 hours to find and fix it.
- $40 × (12 + 2) = $560
Now assume a CloudWatch alarm on the underlying usage metric, such as invocation count, instance count, or bytes transferred, fires within 15 minutes:
- $40 × (0.25 + 2) = $90
The fix took the same 2 hours in both cases. Detection lag accounts for nearly all of the difference. Late detection also costs engineering time. After 12 hours, the job may have written data, scaled dependent resources, or overlapped with a deploy, so the investigation is harder.
Treat billing alerts as a backstop. They catch what usage alarms miss and confirm the impact in dollars. Usage alarms are the first line of defense.
Who needs account structure and tagging before alerts are useful?
Anyone who wants an alert to reach the team that caused the spend needs this groundwork first. Budgets and anomaly monitors can only scope to dimensions the billing data already carries: linked account, cost category, or an activated cost allocation tag. If two teams share an account and nothing distinguishes their resources, a spike can only be routed to whoever owns the whole bill.
Checklist
- Split by linked account where you can. Account boundaries are the most reliable attribution because they don't depend on anyone remembering to tag.
- Activate tags in the Billing console. Applying a tag to a resource is not enough. It must also be activated as a cost allocation tag. Activation is not retroactive, so spend before that date stays unattributed.
- Enforce tags with policy or IaC checks. Use tag policies, SCP conditions that deny untagged creates, or a pipeline check that rejects missing keys. If enforcement only happens in review, tags will drift.
- Define cost categories that map accounts and tags to teams. Then point budgets and monitors at those categories.
- Expect an unattributed bucket. Untagged spend appears as a "no tag" line. Give that line its own owner and budget instead of ignoring it.
Known attribution gaps
Some costs resist clean attribution. Shared NAT gateways and data transfer serve many teams. Support charges, credits, and refunds apply at the account or organization level. One-off Savings Plans, Reserved Instance, or Marketplace purchases can show up as large spikes. Decide in advance who owns each of these.
How do you set up budgets and cost anomaly detection? A checklist
Budgets
- One account- or organization-level monthly cost budget. Set actual-spend notifications at 50, 80, and 100 percent, and a forecasted notification at 100 percent. The forecasted alert gives the earliest warning. The 50 and 80 percent actual alerts confirm whether the forecast was right.
- Per-team or per-environment budgets filtered by cost allocation tags or linked accounts. A production overrun and a forgotten sandbox need different owners, so give them separate budgets.
- Route notifications to an SNS topic, not only to email. SNS can forward to chat or paging tools, where a named person sees the alert. The topic's access policy must allow
budgets.amazonaws.comto publish.
Cost Anomaly Detection
- Scope monitors deliberately. Use a service-level monitor for broad coverage. Add monitors scoped to linked accounts, cost categories, or tags where team ownership matters.
- Set impact thresholds on each subscription. Use an absolute dollar threshold, a percentage threshold, or both. This keeps a $3 blip on a $5 service from paging anyone.
- Choose the alert frequency for each audience. Immediate alerts go through SNS and suit on-call engineers. Daily or weekly email summaries suit managers and finance.
- Expect a learning period. New accounts and newly tagged spend need some history before the model has a baseline. During that window, rely on budgets.
- Allow
costalerts.amazonaws.comto publish to the same SNS topic.
Manage all of it as code
Define budgets, monitors, subscriptions, and topics in Terraform or CloudFormation. Thresholds then get reviewed in pull requests, and new accounts start with the same guardrails as existing ones.
# CloudFormation template. Deploy in us-east-1, where Budgets and Cost Anomaly Detection deliver to SNS.
Resources:
CostAlertsTopic:
Type: AWS::SNS::Topic # subscribe chat/paging integrations to this topic
CostAlertsTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
Topics: [!Ref CostAlertsTopic]
PolicyDocument:
Statement:
- Effect: Allow # both services must be allowed to publish
Principal: { Service: [budgets.amazonaws.com, costalerts.amazonaws.com] }
Action: sns:Publish
Resource: !Ref CostAlertsTopic
MonthlyCostBudget:
Type: AWS::Budgets::Budget
Properties:
Budget: { BudgetType: COST, TimeUnit: MONTHLY, BudgetLimit: { Amount: 10000, Unit: USD } }
NotificationsWithSubscribers: # actual 50/80/100% plus forecasted 100%
- Notification: { NotificationType: ACTUAL, ComparisonOperator: GREATER_THAN, Threshold: 50 }
Subscribers: [{ SubscriptionType: SNS, Address: !Ref CostAlertsTopic }]
- Notification: { NotificationType: ACTUAL, ComparisonOperator: GREATER_THAN, Threshold: 80 }
Subscribers: [{ SubscriptionType: SNS, Address: !Ref CostAlertsTopic }]
- Notification: { NotificationType: ACTUAL, ComparisonOperator: GREATER_THAN, Threshold: 100 }
Subscribers: [{ SubscriptionType: SNS, Address: !Ref CostAlertsTopic }]
- Notification: { NotificationType: FORECASTED, ComparisonOperator: GREATER_THAN, Threshold: 100 }
Subscribers: [{ SubscriptionType: SNS, Address: !Ref CostAlertsTopic }]
ServiceMonitor:
Type: AWS::CE::AnomalyMonitor # broad, per-service coverage
Properties: { MonitorName: all-services, MonitorType: DIMENSIONAL, MonitorDimension: SERVICE }
OnCallAnomalySubscription:
Type: AWS::CE::AnomalySubscription
Properties:
SubscriptionName: oncall-immediate
Frequency: IMMEDIATE # IMMEDIATE requires an SNS subscriber; DAILY/WEEKLY use email
MonitorArnList: [!GetAtt ServiceMonitor.MonitorArn]
Subscribers: [{ Type: SNS, Address: !Ref CostAlertsTopic }]
# Only alert when total impact is at least $100, filtering small-dollar noise
ThresholdExpression: '{"Dimensions":{"Key":"ANOMALY_TOTAL_IMPACT_ABSOLUTE","MatchOptions":["GREATER_THAN_OR_EQUAL"],"Values":["100"]}}'
A CloudFormation template that defines an SNS topic both services can publish to, a monthly cost budget with actual 50/80/100 percent and forecasted 100 percent alerts, a service-level anomaly monitor, and an immediate subscription with a $100 absolute impact threshold.
What are the alternatives to billing alerts for catching runaway spend early?
Billing data lags, but usage metrics arrive in CloudWatch within minutes. An alarm on the usage that drives the bill fires while the problem is still small. Billing alerts then become the backstop, not the first signal.
| Cost driver | Usage metric to watch | Why it spikes |
|---|---|---|
| Lambda | Invocations, Duration (AWS/Lambda) |
Retry loops, recursive triggers |
| NAT gateway | BytesOutToDestination (AWS/NATGateway) |
Traffic to S3 or ECR routed through NAT instead of VPC endpoints |
| Inter-AZ transfer | Instance or load balancer network bytes, as a proxy | Chatty services split across AZs |
| CloudWatch Logs | IncomingBytes per log group (AWS/Logs) |
Debug logging left on in production |
| S3 requests | AllRequests (request metrics must be enabled per bucket) |
Hot listing or polling loops |
| EC2 and GPU | vCPU counts in the AWS/Usage namespace |
Forgotten experiments, leaked instances |
| Auto Scaling | GroupInServiceInstances (group metrics must be enabled) |
Scaling on a bad signal |
A typical alarm watches Lambda Invocations for one function. It uses the Sum statistic over a 300-second period and fires after three consecutive breaching periods. The alarm action publishes to the SNS topic the team already routes to on-call. Requiring three consecutive breaches filters out single bursts but still fires within about 15 minutes.
Guardrails that cap blast radius
Alarms tell you about a problem. Guardrails limit how large it can get before anyone reads the alarm.
- Service quotas: keep quotas low in non-production accounts so a runaway process hits a hard ceiling.
- Auto Scaling max capacity: set a maximum the group cannot scale past.
- Lambda reserved concurrency: cap how many copies of a function can run at once.
- Log retention and sampling: set retention on every log group and sample high-volume logs.
- S3 lifecycle rules: expire or transition old objects automatically.
- Organization policies (SCPs): in sandbox accounts, deny unused regions and expensive instance families.
Each alarm costs money and needs its threshold retuned as traffic grows. Cover only your top three to five spike risks.
#!/usr/bin/env bash
set -euo pipefail
# Placeholder values: replace with your function, region, and on-call topic.
FUNCTION_NAME="orders-processor"
REGION="us-east-1"
ONCALL_TOPIC_ARN="arn:aws:sns:us-east-1:111122223333:oncall-alerts"
# Usage-level alarm: fires when invocations stay above normal for
# 3 consecutive 5-minute periods (about 15 minutes). This catches retry
# loops or recursive triggers long before billing data shows the cost.
# Set the threshold from the function's observed peak plus headroom.
aws cloudwatch put-metric-alarm \
--region "$REGION" \
--alarm-name "${FUNCTION_NAME}-invocations-spike" \
--alarm-description "Lambda invocations far above baseline; possible runaway loop" \
--namespace "AWS/Lambda" \
--metric-name "Invocations" \
--dimensions Name=FunctionName,Value="$FUNCTION_NAME" \
--statistic Sum \
--period 300 \
--evaluation-periods 3 \
--datapoints-to-alarm 3 \
--threshold 50000 \
--comparison-operator GreaterThanThreshold \
--treat-missing-data notBreaching \
--alarm-actions "$ONCALL_TOPIC_ARN"
# Guardrail: cap concurrent executions so a runaway loop hits a ceiling
# even before anyone reads the alarm. Size this to normal peak concurrency.
aws lambda put-function-concurrency \
--region "$REGION" \
--function-name "$FUNCTION_NAME" \
--reserved-concurrent-executions 50
An AWS CLI script that creates a CloudWatch alarm on a Lambda function's Invocations (Sum over 300 seconds, three consecutive breaches, notifying an on-call SNS topic) and sets reserved concurrency as a guardrail that limits blast radius.
When should you avoid automated budget actions and other alert pitfalls?
Keep automated budget actions out of production
Budget actions can attach an IAM policy, apply a service control policy, or stop EC2 and RDS instances when a threshold is crossed. These are blunt tools. A deny policy does not know which caller is the runaway batch job and which is your checkout service. A stop action does not drain connections first. An outage triggered by a cost control can easily cost more than the overspend it was meant to prevent. Reserve these actions for sandboxes, dev accounts and other non-critical environments. Before you rely on one, trigger it on purpose, confirm it fires, and confirm you can reverse it quickly.
Tune for the trade-offs
-
Noise vs. sensitivity: Tight thresholds page people for routine variance, and engineers soon learn to ignore them. Loose thresholds miss real spikes.
-
Percentage-only and absolute-only thresholds both fail:
Illustrative estimate (assumed numbers, not a benchmark) Assume a service that costs $20/day with a 50% threshold. It alerts on a $10 increase, which is noise. Now assume a $5,000/day service with a $500 absolute threshold. It stays silent until spend has risen 10%. Combine both conditions: alert when
impact ≥ $X AND impact ≥ Y% of expected spend. -
Learning periods: Anomaly detection needs spend history before its baseline means anything. New accounts and services are under-covered at first.
-
Gradual growth: A slow climb becomes the new baseline and never registers as an anomaly. Fixed budgets catch this kind of drift.
-
Planned spikes: Launches, load tests and month-end jobs look like anomalies. Announce them to alert owners in advance.
How do alert ownership and runbooks save engineering time and cost?
Most runaway-spend incidents are not detection failures. The alert fired, but it went to a shared inbox or channel where everyone assumed someone else would look. Give every budget and anomaly monitor three attributes before you enable it:
- A named owner. Pick a team or an on-call rotation, not "platform" in general.
- A severity. This decides whether the alert pages someone or waits for business hours.
- A first-hour runbook:
- In Cost Explorer, group the spend by service, then by usage type, to see what is actually growing.
- Find the resource through its tags. If tags are missing, use CloudTrail to find who or what created or changed it.
- Decide whether to scale it down, stop it, or accept the spend and record why.
Decide ahead of time which accounts allow automated responses and which require a human. Sandbox and dev accounts are good candidates for automatic stops. Production should go to a person, because cutting capacity there can cost more than the spike itself.
Hold a monthly review that answers three questions:
- Which alerts fired?
- Which of those were noise?
- Which incidents were caught late or missed entirely?
Tune thresholds from those answers instead of guessing. When every alert has a known owner and an agreed first step, triage starts immediately rather than with a search for who owns the account. Alerts that never lead to action get removed, which reduces fatigue. Dev environments benefit the most. Idle clusters and forgotten test fleets stop running for weeks unnoticed, because someone is accountable for them and has the authority to shut them down.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to aws cost anomaly detection, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Consumers learn about a breaking change at review time rather than from production errors.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.