What is edge log filtering with Fluent Bit, and why does it reduce observability spend?

Edge log filtering means dropping, trimming or rerouting records inside the Fluent Bit agent, on the node where they are produced, before any output plugin ships them. The decision happens at the source. A discarded record never uses network bandwidth, never enters the backend's ingest pipeline, and is never indexed or stored.

Backend-side filtering cannot do this. Most observability backends meter ingested bytes, and cross-zone or internet egress is usually billed per byte as well. A drop rule applied after ingestion can reduce storage and retention costs, but by then the record has already been transferred and counted. Only the agent can avoid both charges.

Log volume is rarely spread evenly. A few predictable sources usually dominate:

  • Health checks and readiness probes that log a line on every hit.
  • Debug and trace output left enabled in production.
  • Repeated errors, such as the same stack trace emitted thousands of times during an incident.
  • Oversized payloads, such as full request bodies, serialized objects or base64 blobs embedded in a message.

Illustrative estimate (assumed numbers, not a benchmark): daily savings = daily volume × fraction removed × (ingest price per GB + transfer price per GB). Assume 500 GB/day, 40% of it from probes and debug logs, and a combined $0.60/GB: 500 × 0.40 × 0.60 = $120/day.

This checklist is for backend and platform engineers who own the agent configuration. It pays off most on high-volume clusters where a few sources produce most of the bytes.

Who should use Fluent Bit edge filtering?

Edge filtering fits teams that already run Fluent Bit, or plan to, as the node-level log agent and that pay per byte for ingest or transfer. In practice that means:

  • Platform and SRE teams that run Fluent Bit as a DaemonSet on Kubernetes or as an agent on VMs, and that control its configuration centrally.
  • Backend engineers whose services produce high-volume, low-value output such as probe access logs, verbose INFO logging or debug output that was never turned off.
  • Teams with a known noise profile. You should be able to name the sources you want to cut, or be willing to measure first as described in Step 1.
  • Organizations with multiple destinations, where some logs belong in an indexed backend and others only need a cheap archive.

It is a poor fit if you cannot change the agent configuration yourself, if nearly all of your log volume is compliance or audit data that must be retained in full, or if your volume is small enough that the cost of writing and testing rules is higher than the savings. The last section of this article covers these cases in more detail.

How does edge filtering work in Fluent Bit?

Fluent Bit treats every log line as a record that carries a tag, a timestamp and a set of key-value fields. The configuration declares a pipeline of inputs, parsers, filters and outputs. Filters are the part that does the cost work. Each filter selects records by matching their tag and then does one of three things:

  • Drop the record, for example grep with an Exclude rule, throttle, or a Lua script that returns -1.
  • Shrink the record by removing keys or metadata it does not need, for example with modify or record_modifier.
  • Reroute the record by changing its tag with rewrite_tag, so that a different, cheaper output matches it.

Because all of this happens inside the agent before records are buffered and flushed, anything dropped or shrunk is never sent over the network or billed by the backend. The next section explains how records move through the filter chain and why order matters.

How does a Fluent Bit filter chain process records?

Every record in Fluent Bit moves through the same stages. An input (for example, tail reading container log files) creates the record and assigns it a tag. An optional parser turns the raw line into structured fields. Filters then modify, enrich or drop the record. Finally, outputs ship it. Filters and outputs each declare a Match pattern, which can include wildcards such as kube.*. A record only passes through the filters and outputs whose pattern matches its tag. Tags are therefore your routing mechanism. If tags are sloppy, a filter will either miss traffic or catch traffic it shouldn't.

Filters run in the order you declare them, and each one sees the output of the previous one. This makes ordering a cost decision:

Where cheap drops belong

  • First: grep exclusions on the raw log field, such as health-check paths or DEBUG markers. These are a single regex test per record, and every record dropped here skips all later work.
  • Then: parsing, which costs more because it decodes JSON or runs regexes that capture fields.
  • Last: the kubernetes filter. It looks up pod metadata, caches it, and adds labels and annotations that inflate every record. Rules that depend on namespace or label have to come after it. Everything else should come before it.

Buffers and backpressure

Filters run as records are ingested, before data is written into buffer chunks. A dropped record therefore never uses memory or filesystem buffer space. A slow filter, especially a heavy Lua script, slows ingestion itself. When a slow output causes chunks to pile up and the input reaches its memory limit (Mem_Buf_Limit on tail), Fluent Bit pauses that input. Early, cheap drops lower both CPU per record and the volume that gets buffered. Both effects make it less likely that the agent stalls during an output outage.

+---------------------------+
| INPUT: tail               |  assigns tag, e.g. kube.*
| Mem_Buf_Limit             |<-----------------------------+
+---------------------------+   pause input when limit hit |
              |                                            |
              v                                            |
+-------------------------------------------------+        |
| FILTERS  Match kube.*  (run in declared order)  |        |
|                                                 |        |
|  1. grep: exclude raw log        cheap --> DROP |        |
|               |                                 |        |
|               v                                 |        |
|  2. parser: decode JSON / regex   more cost     |        |
|               |                                 |        |
|               v                                 |        |
|  3. kubernetes: pod metadata      heavy, grows  |        |
|               |                   each record   |        |
|               v                                 |        |
|  4. grep: namespace/label rules        --> DROP |        |
+-------------------------------------------------+        |
              |                                            |
              v                                            |
+-------------------------------------------------+        |
| BUFFER CHUNKS (memory / filesystem)             |--------+
+-------------------------------------------------+ backlog
              |
              v
+-------------------------------------------------+
| OUTPUT  Match kube.*   (slow output -> backlog) |
+-------------------------------------------------+

Records flow from a tagged tail input through filters in declared order, with cheap grep drops before parsing and Kubernetes enrichment, so dropped records never reach the buffer, and a backlog behind a slow output can pause the input at Mem_Buf_Limit.

Step 1: How do you measure log volume before filtering?

Filter against numbers, not guesses. Before adding a single rule, find out which sources produce the volume and record a baseline that every later step is compared against.

Enable the agent's built-in HTTP server (HTTP_Server On in the [SERVICE] section) and read /api/v1/metrics/prometheus. Each plugin instance reports its own counters:

  • fluentbit_input_records_total and fluentbit_input_bytes_total show what enters.
  • fluentbit_filter_drop_records_total shows what each filter removes. It should read zero at baseline.
  • fluentbit_output_proc_records_total and fluentbit_output_proc_bytes_total show what is actually shipped.

Set an explicit Alias on each plugin. Without one, instances are labelled tail.0 or grep.1, and those labels are hard to track across config changes. Scrape these counters over a full traffic cycle, at least a day, so nightly batch jobs and peak hours both appear.

The counters give you totals, not causes. To find the causes, sample shipped records and group them two ways:

  • By namespace, container or tag.
  • By message template. Replace numbers, UUIDs and timestamps with placeholders so that thousands of distinct lines collapse into a handful of patterns.

Rank each group by bytes and by record count. A chatty health check ranks high on count, while a stack-trace dump ranks high on bytes. Each one needs a different filter.

Illustrative estimate (assumed inputs): 40 nodes each ship 2,000 records/s at 400 bytes per record. That is 32 MB/s, roughly 2.8 TB/day. If template analysis shows that readiness-probe access logs make up 30% of records, dropping them removes about 0.8 TB/day. Record these figures. Step 7 checks the actual drop counters against them.

[SERVICE]
    Flush        5
    Log_Level    info
    # Expose the built-in monitoring endpoint
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name   tail
    # A stable alias keeps metric labels the same across config changes
    Alias  in_container_logs
    Tag    kube.*
    Path   /var/log/containers/*.log

[FILTER]
    Name   kubernetes
    # Adds namespace and pod metadata so records can be grouped by source.
    # Its drop counter should read zero at baseline.
    Alias  enrich_k8s
    Match  kube.*

[OUTPUT]
    Name   forward
    # Shipped records and bytes are reported under this alias
    Alias  out_aggregator
    Match  *
    Host   log-aggregator.example.internal
    Port   24224

A baseline Fluent Bit config that enables the HTTP monitoring endpoint and gives each plugin a stable Alias, so the per-plugin input, filter and output counters can be scraped and compared against later filtering steps.

Steps 2–5: The ordered filter checklist

Apply the filters in this order. Each step shrinks the input for the next one, so the cheapest and broadest cuts go first.

Step Filter What it removes Why it sits here
2 grep with Exclude Health checks, readiness probes, known-noise paths A regex match is cheap and drops whole records early
3 Level enforcement DEBUG/TRACE in production It needs a parsed level field
4 modify / record_modifier Unneeded keys and metadata It shrinks the records you keep
5 throttle / Lua Bursts and excess INFO It acts on what remains

Step 2. Anchor every Exclude regex, for example ^GET /healthz. An unanchored healthz also drops any error message that mentions the endpoint.

Step 3. Parse JSON logs so that level becomes a key. Then use rewrite_tag to give namespaces or pods that carry a debug-override annotation a separate tag. Apply the DEBUG/TRACE grep only to the default tag. Because the override is a tag rule, you can enable debug logging for one service without changing the global rules.

Step 4. In the kubernetes filter, set Labels Off and Annotations Off unless your queries use them. Use Remove or Allowlist_key to delete duplicated fields, such as a raw log field that has already been merged.

Step 5. throttle limits the record rate per tag over a time window. It does not deduplicate messages, and it does not report what it dropped, so record the drop counts from Fluent Bit's metrics. The Lua example keeps 1 in N INFO records and always passes WARN and ERROR. It adds a sample_rate field so dashboards can scale the INFO counts back up. Review this script and test it against real records before you deploy it, because a bug in it can drop or corrupt every record that passes through.

pipeline:
  filters:
    - name: lua
      match: kube.default.*   # default tag only; debug-override tags are not sampled
      call: sample_info
      code: |
        local N = 10          -- keep 1 in N INFO records
        local seen = 0        -- one counter for this filter instance, shared by every matched tag

        function sample_info(tag, timestamp, record)
          local level = record["level"]
          -- Pass records with no parsed level untouched; never drop what you can't classify
          if level == nil then
            return 0, timestamp, record
          end
          level = string.upper(tostring(level))
          -- WARN, ERROR and anything else that is not INFO always pass
          if level ~= "INFO" then
            return 0, timestamp, record
          end
          seen = seen + 1
          if seen % N ~= 0 then
            return -1, timestamp, record   -- -1 drops the record
          end
          -- Dashboards multiply INFO counts by sample_rate to estimate the true volume
          record["sample_rate"] = N
          return 2, timestamp, record      -- 2 means the record changed and the timestamp is kept
        end

A Fluent Bit YAML lua filter with an inline script that keeps one INFO record in ten, passes WARN, ERROR and records without a level unchanged, and tags each kept INFO record with sample_rate so dashboards can scale counts back up.

Step 6: How do you route low-value logs instead of deleting them?

Deleting a record is irreversible. Rerouting it keeps your options open. For logs you are unsure about, such as DEBUG output from a service you rarely touch, health-check access lines, or INFO records the sampler would otherwise discard, use the rewrite_tag filter to give them a new tag. Then match that tag to a cheaper destination, such as an object storage bucket with a lifecycle policy, or a compressed archive the backend does not index.

A rewrite_tag rule has four parts: a record key, a regex, the new tag, and a keep flag. Set keep to false so the record leaves the expensive path instead of being duplicated into both destinations. The re-tagged record is re-emitted at the start of the pipeline through an internal emitter. Because of that, follow these practices:

  • Avoid loops. Make sure the new tag (for example archive.*) does not match the filter's own Match pattern.
  • Buffer the emitter. Give the emitter a memory buffer limit, or filesystem storage, so a burst of low-value logs cannot push out higher-priority data.
  • Keep the archive searchable. Index the archive tier by date and service. During an incident you can then rehydrate or query one narrow slice instead of wishing you had kept everything.

Exclude compliance, audit and security logs from every cost-driven rule. Give them their own input or a distinct tag prefix early in the pipeline. Then confirm that no sampling, level, or rewrite filter's Match pattern can reach them. Retention requirements for these streams come from policy, not from your observability budget.

[SERVICE]
    Parsers_File  parsers.conf

[INPUT]
    Name    tail
    Path    /var/log/app/*.log
    Tag     app.*
    Parser  json

[INPUT]
    Name    tail
    Path    /var/log/audit/*.log
    Tag     audit.*
    Parser  json

[FILTER]
    Name                  rewrite_tag
    Match                 app.*
    Rule                  $level ^(DEBUG|debug)$  archive.$TAG  false
    Rule                  $path  ^/healthz$       archive.$TAG  false
    Emitter_Name          re_emit_archive
    # Cap the emitter so a burst of low-value logs cannot crowd out other data
    Emitter_Mem_Buf_Limit 10M

[OUTPUT]
    Name   http
    Match  app.*
    Host   logs-backend.internal.example
    Port   443
    tls    On
    Format json

[OUTPUT]
    Name   file
    Match  archive.*
    Path   /var/archive/logs

[OUTPUT]
    Name   http
    Match  audit.*
    Host   audit-store.internal.example
    Port   443
    tls    On
    Format json

A Fluent Bit classic config that uses rewrite_tag with keep=false to move DEBUG and health-check records from the main backend to a cheaper archive output, while audit logs use a separate tag prefix that no cost-driven rule matches.

Step 7: How do you verify, monitor and roll back Fluent Bit filters safely?

Filters fail silently. A regex that is slightly too broad still produces valid output, just less of it. Verification has to be quantitative.

Compare records in and out per filter

Enable Fluent Bit's built-in HTTP server and scrape its per-plugin metrics. Give every filter an Alias so its counters read as drop_healthchecks rather than an auto-numbered instance name. For each filter, track records dropped against records entering the pipeline. This shows which rule is responsible for a drop, which a single input-versus-output total cannot.

Alert on sharp changes in drop ratios

Alert on change, not on an absolute threshold. A filter that normally drops 40% of a stream and suddenly drops 95% usually means one of two things: a deploy changed the log format, or a rule now matches far more than intended. A drop to 0% means the format changed and the rule no longer matches anything. Either case should page someone before the logs are needed.

Test rules before deploying

Build a fixture file of real log lines that includes edge cases: multiline stack traces, mixed casing in level fields, and malformed JSON. Run the candidate config against it with a tail or dummy input and a stdout output. Diff the result against the expected output in CI.

Version control and rollback

Above each rule, add a one-line comment stating what it drops and why. Roll back by reverting the commit, not by editing configs live. Clusters that deploy from the same repository cannot drift apart unnoticed. During an incident, git log shows who removed which logs and why, so nobody has to reconstruct that decision under pressure. Reviews then focus on the stated rationale instead of reverse-engineering a regex.

How can edge filtering save engineering time?

The direct saving is money, but a well-run filter setup also gives engineers time back:

  • Less noise to search through. When probe access lines and stray DEBUG output never reach the backend, queries during an incident return fewer irrelevant matches. Engineers spend less time writing exclusions into every search and scrolling past repeated lines.
  • One central change instead of many code changes. A single grep or rewrite_tag rule in the agent configuration can silence a noisy source across every service on the cluster. Fixing the same noise in each application means a code change, review and deploy per team. Use the filter as the immediate fix and schedule the application fix for chronic offenders.
  • Fewer pipeline stalls to debug. Dropping records early reduces the volume that is buffered, which makes Mem_Buf_Limit pauses less likely during output slowdowns. That means fewer "missing logs" investigations caused by a backed-up agent.
  • Faster, safer reviews. A fixture-based CI test, a stated rationale above each rule and per-filter drop alerts make rule changes quick to review and quick to roll back. Nobody has to reverse-engineer why a log line disappeared.
  • Debug on demand without config churn. The debug-override tag from Step 3 lets one team turn on verbose logging for one service without asking the platform team to edit global rules and redeploy the agent.

Illustrative estimate (assumed inputs, not a measurement): assume a team of 10 engineers each spends 20 minutes a week filtering noise out of log searches, and that removing probe and debug noise halves that time. That saves 10 × 10 minutes = 100 minutes per week, or roughly 80 engineer-hours a year. Weigh it against the assumed one-time cost of writing, testing and rolling out the rules, plus the ongoing upkeep of the fixture file and drop-ratio alerts. Substitute your own figures. The point is to compare both sides, not to rely on these numbers.

When should you avoid edge filtering, and what are the alternatives?

Edge filtering is the wrong first move when you can't yet say which logs are worthless. A dropped record cannot be recovered, so a filter written on a guess becomes a gap you find during an incident. Each technique has its own failure mode:

  • Sampling breaks per-request completeness. You keep one line of a trace and lose the rest, and error counts derived from sampled logs are skewed unless every consumer applies the sample rate.
  • Level-based drops assume applications log at the correct level. A service that logs a payment failure at DEBUG loses it silently.
  • Throttling caps volume during exactly the moments volume matters. A spike from 1,000 to 50,000 errors per minute looks flat at the cap, which hides the size of an outage.
  • Filters run on every node. Regex and Lua filters cost CPU and memory on each node, and that cost grows with node count.
Approach Development cost Transfer savings Ingest savings
Edge filtering (Fluent Bit) Moderate: config, tests, rollout per node Yes Yes
Backend-side filtering Low: one central rule set None: data still crosses the network Partial, depending on where the backend meters
Broad sampling Low Large Large, at the cost of completeness and accurate counts
Retention cuts Very low None None: only reduces storage
Fixing logging in the application High: code changes across teams Largest, and durable Largest, and removes serialization cost too

Use edge filtering for known, stable noise. Fix chronic offenders in the application, where the savings don't depend on a filter staying correct.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to fluent bit filter logs reduce cost, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.