How do Loki and Elasticsearch store and index logs differently?
Loki and Elasticsearch both accept the same log line, but they make opposite decisions about what to index.
Loki: index the labels, compress the content
Loki groups log lines into streams. A stream is defined by its unique set of labels, such as {app="checkout", env="prod", cluster="eu-1"}. The index records only which label sets exist and which chunks hold their data for a given time range. The log lines are buffered per stream, compressed into chunks, and flushed, usually to object storage.
Loki never looks inside a line at write time. The index stays small because its size grows with the number of streams, not with the volume of text. Object storage is cheap and durable, so keeping large volumes of logs costs little. The cost appears at query time. Loki narrows the search by labels, then fetches and decompresses chunks and scans every line in them by brute force.
Elasticsearch: index the content up front
Elasticsearch stores each log line as a JSON document in an index, which is split into shards. Each shard is a Lucene index made of immutable segments on the data nodes' local disks.
At ingest, analyzed text fields are tokenized into an inverted index that maps each term to the documents containing it. Elasticsearch also keeps the original _source and columnar doc values for sorting and aggregations.
Elasticsearch does that work up front, so a term lookup goes straight to matching documents without a scan. You pay for it in ingest CPU, segment merging, replica copies and fast disks that are often several times the raw log size.
In short, Loki pays for search when you query, and Elasticsearch pays when you write and store.
same log line arrives
|
+-----------------+----------------------+
v v
+--------------------------------+ +----------------------------------+
| LOKI: index labels only | | ELASTICSEARCH: index content |
| stream = {app,env,cluster} | | line -> JSON doc -> index/shard |
+----------------+---------------+ +-----------------+----------------+
| |
v v
+--------------------------------+ +----------------------------------+
| buffer lines per stream, | | analyze + tokenize text fields |
| compress into chunks | | inverted index: term -> docs |
| never reads line content | | + _source + doc values |
+-------+----------------+-------+ +-----------------+----------------+
| | |
v v v
+--------------+ +---------------+ +----------------------------------+
| index: label | | chunks in | | Lucene segments on local disk |
| sets -> refs | | object storage| | merges + replicas, often several |
| (small) | | (cheap) | | times raw log size |
+------+-------+ +-------+-------+ +-----------------+----------------+
| | |
+--------+--------+ v
v +----------------------------------+
+--------------------------------+ | QUERY: term lookup -> matching |
| QUERY: match labels, fetch + | | docs directly, no scan |
| decompress chunks, scan lines | +----------------------------------+
+--------------------------------+
cost paid at query time cost paid at ingest and storage
Loki indexes only stream labels, writes compressed chunks to object storage, and scans them at query time. Elasticsearch tokenizes content into an inverted index in Lucene segments on local disk, so lookups need no scan but ingest and storage cost more.
How does a LogQL query execute compared with an Elasticsearch search?
Loki resolves a query in two stages. The stream selector, such as {service="checkout", env="prod"}, is matched against the label index to find stream IDs. Loki then fetches every chunk for those streams in the requested time range, decompresses them, and runs line filters (|= "timeout") and parsers (| json, | logfmt) over each line. The index has no information about line contents, so a rare search term costs about as much as a common one. Cost depends on how many bytes the selector and time range cover. Queriers can split work across time ranges and shards, which reduces latency but not the bytes scanned.
Elasticsearch looks up the term in each shard's inverted index and reads only the matching postings. This makes needle-in-a-haystack searches fast. A trace ID search across every service stays cheap because the term matches a handful of documents. Loki has to scan every stream the selector covers. Aggregations read doc values on mapped fields, not raw text.
| Task | LogQL | Elasticsearch (Lucene syntax) |
|---|---|---|
| Errors in one service | {service="checkout"} |= "timeout" | json | status >= 500 |
service:checkout AND message:timeout AND status:>=500 |
| Trace ID, all services | {env="prod"} |= "4bf92f3577b34da6" |
trace_id:"4bf92f3577b34da6" |
| Errors by status | sum by (status) (count_over_time({service="checkout"} | json [5m])) |
terms aggregation on status inside a date_histogram |
LogQL metric queries are computed by parsing raw lines at query time. A dashboard that refreshes every 30 seconds over a 24-hour window reprocesses the same chunks on every refresh. For panels that stay up permanently, emit real counters from the application. Use LogQL metrics for investigation and occasional ad hoc queries.
# Same trace ID lookup against both backends. Hosts and index names are placeholders.
LOKI=
ES=
TRACE=4bf92f3577b34da6
# Loki: the selector {env="prod"} picks every prod stream. The |= filter then
# runs over every decompressed line in those chunks for the whole hour, so
# bytes scanned do not depend on how rare the trace ID is.
curl -sG "$LOKI/loki/api/v1/query_range" \
--data-urlencode "query={env=\"prod\"} |= \"$TRACE\"" \
--data-urlencode "start=$(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%SZ)" \
--data-urlencode "end=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
--data-urlencode 'limit=100'
# Elasticsearch: a term query on a keyword field reads the postings list for
# that one term in each shard, so a rare value matches only a few documents.
curl -s -X POST "$ES/logs-*/_search" -H 'Content-Type: application/json' -d "{
\"query\": { \"term\": { \"trace_id\": \"$TRACE\" } }
}"
# Errors by status. Loki parses every checkout line with | json at query time,
# and a dashboard repeats that work on each refresh.
curl -sG "$LOKI/loki/api/v1/query" \
--data-urlencode 'query=sum by (status) (count_over_time({service="checkout"} | json [5m]))'
# Elasticsearch reads doc values for the mapped status and @timestamp fields
# and does not touch the raw message text.
curl -s -X POST "$ES/logs-*/_search" -H 'Content-Type: application/json' -d '{
"size": 0,
"query": { "term": { "service": "checkout" } },
"aggs": {
"per_5m": {
"date_histogram": { "field": "@timestamp", "fixed_interval": "5m" },
"aggs": { "by_status": { "terms": { "field": "status" } } }
}
}
}'
Equivalent trace ID searches and status aggregations sent to the Loki HTTP API and the Elasticsearch _search API: Loki scans every line in the selected streams, while Elasticsearch looks up the term in its inverted index and reads doc values.
How do you implement log indexing without cardinality or mapping blowups?
Loki: keep labels low-cardinality
Loki creates one stream per unique label combination, and each stream has its own chunks and index entries. Put request_id or user_id in a label and every request opens a new stream. Ingesters then hold millions of tiny, mostly empty chunks in memory, flush them as small objects, and the index grows faster than the logs. Queries suffer as well, because a selector has to resolve and open far more streams to find the same data.
Limit labels to values that describe where a log came from, such as cluster, namespace, service, env, and maybe level. Something like {service="checkout", env="prod", level="error"} is a reasonable ceiling. High-cardinality identifiers belong in the log line, where you can filter them with | json | request_id="...", or in structured metadata, which is attached to each entry without creating new streams.
Elasticsearch: define mappings before data arrives
Dynamic mapping turns every new JSON key into a field. Services that log arbitrary keys, such as header maps or per-customer attributes, eventually reach the index's total-fields limit, and Elasticsearch starts rejecting documents. Type conflicts cause similar failures. If one service sends status as a number and another sends "ok", the second document fails to index, or the field is silently dropped if ignore_malformed is set.
Create an index template before the first document is written. Set dynamic to strict or false. Map identifiers as keyword so exact matches and aggregations work, and reserve text for message bodies. Send unbounded maps to a flattened field so they don't add new fields to the mapping.
Both mistakes are cheap to avoid at the start and expensive to correct later. Fixing them means changing shippers, parsers, and dashboards across every team that produces logs, and in Elasticsearch it also means reindexing existing data.
# Create the template BEFORE any service writes to app-logs-*.
# Changing a mapping later means reindexing and reworking pipelines.
curl -X PUT "" \
-H 'Content-Type: application/json' -d '
{
"index_patterns": ["app-logs-*"],
"priority": 200,
"template": {
"settings": {
"index.mapping.total_fields.limit": 200
},
"mappings": {
"dynamic": "false",
"properties": {
"@timestamp": { "type": "date" },
"service": { "type": "keyword" },
"env": { "type": "keyword" },
"level": { "type": "keyword" },
"request_id": { "type": "keyword" },
"user_id": { "type": "keyword" },
"status": { "type": "keyword" },
"message": { "type": "text" },
"attributes": { "type": "flattened" }
}
}
}
}'
# dynamic=false: unknown keys stay in _source but add no new fields.
# Use "strict" instead to reject documents with unmapped keys.
# status is keyword so both 200 and "ok" index without a type conflict.
# attributes (headers, per-customer data) is one flattened field,
# however many keys it contains.
An Elasticsearch composable index template that fixes field types up front, maps identifiers as keyword and message bodies as text, and stores unbounded key maps in a single flattened field so the mapping cannot keep growing.
What does log retention cost in Loki vs Elasticsearch?
Retention cost is roughly daily ingest × on-disk multiplier × copies × days. Elasticsearch and Loki differ most in the multiplier and in the kind of storage it lands on.
Illustrative estimate (assumed numbers, not a benchmark)
Assumed inputs: 100 GB/day of raw logs, 30-day retention.
- Elasticsearch: Compression is largely offset by the inverted index and doc values, so assume on-disk size ≈ 1.0× raw. Assume 1 replica, which means 2 copies. Add 25% free-space headroom so nodes stay below disk watermarks.
- Loki: Assume chunks compress 10:1 (0.1× raw) and the label index adds 1% of raw. Assume a single logical copy, because the object store handles durability.
Formula: stored GB = daily GB × (data ratio + index ratio) × copies × days × headroom
| Elasticsearch | Loki | |
|---|---|---|
| Data + index ratio | 1.0 | 0.1 + 0.01 |
| Copies | 2 | 1 |
| Headroom | 1.25 | 1.0 |
| Storage type | Block (SSD) | Object |
| 30-day footprint | 100 × 1.0 × 2 × 30 × 1.25 = 7,500 GB | 100 × 0.11 × 30 = 330 GB |
The two footprints differ by more than 20×. Block storage also usually costs more per GB than object storage, so the gap in the bill is larger than the gap in bytes.
That multiplier is what forces the tradeoff. Suppose an Elasticsearch cluster is capped at 2,000 GB. Its retention is 2,000 ÷ (100 × 2 × 1.25) = 8 days. Keeping 30 days means buying nearly four times the disk. Teams then shorten retention, drop replicas, or move older indices to cheaper tiers. Loki keeps more history for the money, but every query against that history has to scan chunks.
These ratios depend heavily on log shape. Repetitive text compresses far better than high-entropy JSON, and mapping choices change index size. Before you budget, ingest a representative day of your own logs into each system and measure the results.
How does log retention work in each system?
Loki: the compactor deletes by time
Loki stores chunks in object storage and never edits them. The compactor handles retention. With retention_enabled: true on the compactor, it reads the index, marks chunks older than the configured period, and deletes them after a delay. You set a default with retention_period in limits_config. You can override it per tenant, and per stream with retention_stream, for example {selector: '{namespace="debug"}', period: 72h}. Debug logs can then expire after a few days while audit streams stay much longer. If the compactor is not running, nothing is deleted and your storage keeps growing.
Elasticsearch: lifecycle rules on time-based indices
Elasticsearch writes to time-based indices behind a data stream or alias. An index lifecycle management (ILM) policy rolls over to a new index when a condition is met, such as max_primary_shard_size: 50gb or max_age: 1d. It can move older indices to warm or cold nodes and remove them in a delete phase at min_age: 30d. Dropping a whole index is cheap. Deleting individual documents is expensive, so retention has to line up with index boundaries.
What each system costs to run
Elasticsearch needs ongoing tuning. Shards that are too large recover slowly. Many tiny shards, often caused by daily indices for low-volume sources, use up heap and slow down cluster state updates. JVM heap needs sizing per node. When a node fails, the cluster copies its replica shards to other nodes, which uses disk I/O and network bandwidth while you are already short a node.
Loki depends on object storage for durability, so losing a node does not trigger that kind of re-replication. In both systems, each additional tier or replica copy adds more things to monitor and more ways for retention to fail.
# loki.yaml: compactor-driven retention
compactor:
working_directory: /loki/compactor
compaction_interval: 10m
retention_enabled: true # without this, chunks are never deleted
retention_delete_delay: 2h # grace period between marking and deleting chunks
delete_request_store: s3 # object store that holds delete requests
limits_config:
retention_period: 744h # default for all tenants (31 days)
retention_stream:
# Debug logs expire quickly; a higher priority wins when selectors overlap
- selector: '{namespace="debug"}'
priority: 1
period: 72h
---
# runtime-config.yaml: per-tenant overrides, reloaded without a restart
overrides:
"tenant-audit":
retention_period: 8760h # keep this tenant's audit logs for one year
retention_stream:
- selector: '{app="healthcheck"}'
priority: 1
period: 24h # drop noisy probe logs after one day
Loki retention configuration with a global default, a per-stream override for debug logs, and a per-tenant override in the runtime config. Expired chunks are deleted only if the compactor runs with retention enabled.
When should you use Loki, and when should you avoid it?
Loki works best when the question already names a service and a time window. During incident triage, an engineer typically knows that the checkout service started failing around 14:05. A stream selector on service and env narrows the query to a few streams. Loki then decompresses only the chunks for those streams in that window, so the line filter runs over a small slice of data. Teams that already label Prometheus metrics with the same keys get a second benefit: they can jump from a latency graph to the matching logs without translating between naming schemes.
Avoid Loki when engineers expect to search for an arbitrary string across weeks of data or across every service. There is no inverted index over log content, so a broad selector forces Loki to fetch and scan every matching chunk from object storage. Adding queriers spreads that scan out, but the total amount of data read stays the same.
Elasticsearch is the better fit for broad full-text search, for security investigations that pivot on IPs or user IDs across all sources, and for ad-hoc aggregations over many fields. The index answers these lookups without scanning raw logs. The cost is a cluster you have to size, upgrade and rebalance, and index storage that makes long retention expensive.
| Query pattern | Better fit |
|---|---|
| Errors from one service in the last hour | Loki |
| Correlating logs with metrics by label | Loki |
| Find a token across all services over 30 days | Elasticsearch |
| Security pivots on IPs or user IDs | Elasticsearch |
| Ad-hoc aggregations over many fields | Elasticsearch |
The main alternative is a hybrid setup: Loki for all logs, with Elasticsearch holding only audit or security streams. Running both systems side by side roughly doubles the work of maintaining ingestion pipelines, schemas and on-call runbooks.
Which log backend fits your team? A decision checklist
Before committing to either system, answer these questions:
| Question | Points toward Loki | Points toward Elasticsearch |
|---|---|---|
| Dominant query pattern | Filter by service, environment, or pod, then grep a time window | Free-text or field search across many services with no known starting label |
| Ingest volume | High and growing, where index size drives cost | Moderate, or high with budget for hot-tier nodes |
| Retention requirements | Long retention for occasional lookups | Long retention that must stay fast to search |
| Operational capacity | Small platform team, no time for shard sizing or mapping fixes | Team that can tune index templates, lifecycle policies, and cluster health |
| Metrics correlation | Metrics already use the same label model, so dashboards can jump from a metric to its logs | Logs are analyzed mostly on their own or through structured aggregations |
Each answer changes where engineering time goes. Choosing Loki moves cost into label design up front. After that, little ongoing index maintenance remains, so no one rebalances shards or fixes mapping conflicts after a deploy. Choosing Elasticsearch moves cost into cluster operations. In return, an engineer chasing a request ID across a dozen services gets an answer from one query instead of scanning stream after stream.
Don't settle the choice with published benchmarks. Their hardware, log shapes, and queries won't match yours. Take one representative day of your own logs and pick the ten queries your on-call engineers actually ran during recent incidents. Run both systems with comparable resources. Record four numbers for each:
- Query latency
- Bytes stored after compression and indexing
- Ingest CPU
- Hours spent on setup and tuning
Those four numbers tell you more than any general comparison.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to loki vs elasticsearch logs, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.