How to Reduce Data Costs with OpenTelemetry and Bindplane

Originally written by Paul Stefanski, updated by Dylan Myers.

Data costs fill a large column in many organizations' accounting sheets. Data pipeline setup and management is a significant time sink for DevOps, IT, and SRE. Setting up telemetry pipelines to reduce unwanted data often takes even more time, which could better be spent creating value rather than reducing costs. This post will show you how to quickly set up your data pipeline to filter unnecessary telemetry data.

The pipeline below is the one we will work with throughout. A single log source fans out to two destinations, a production Splunk instance and a cheap file-based cold store. Both branches currently receive the same 5.5 MB/m, because nothing is being filtered yet.

Getting Started with OpenTelemetry and Bindplane

Bindplane is a telemetry pipeline management platform built to work natively with OpenTelemetry. If you already have collectors running, there is a guide for migrating an existing OpenTelemetry collector to Bindplane management. If you are starting from scratch, getting started with Bindplane walks you through installing your first collector.

Bindplane runs on Linux, macOS, and Windows. It is vendor-agnostic, so you can pair any supported source with any supported destination. New integrations land regularly.

Inspect and Filter in the Same Place

Processors run on data after it is received and before it is sent to a destination. That placement is what makes them useful for cost control, since data dropped in the pipeline never reaches a billable destination at all.

Click the processor node on the pipeline branch you want to change. Here we are clicking the node in front of splunk-prod, because Splunk is the expensive destination and cold storage is the cheap one.

The panes on either side are a live before-and-after view. The left pane shows what arrives at this point in the pipeline. The right pane shows a preview of what will leave after the processors are applied and rolled out. It updates as you edit, so you check each filter against real records instead of guessing.

Read the left pane before you touch anything. This service is overwhelmingly debug and info, with only occasional warn and error. That is the shape of a log stream where a large reduction is available for very little risk.

The Pipeline Intelligence panel at the top proposes processors based on what Bindplane observes in this stream. It has spotted the same debug-heavy pattern and is offering a one-click Filter Debug Logs processor, with an estimated reduction attached to it. That estimate is recalculated as Bindplane samples more of the stream, so do not be surprised when it moves between visits. Taking the suggestion is a perfectly good shortcut. We are going to build the processors by hand instead, so it is clear what each one does.

To pick a specific field to filter on, expand any record in the incoming pane. You get the parsed severity, the full body, and every attribute.

Note the fields that cost money on every single event without helping anyone debug anything: session_token, container_id, k8s.pod.uid, thread_id, and build_sha. We come back to those later.

Click Add Processor to see the catalog. Searching narrows it down, and the filtering processors all live under Filter And Reduce.

Filter by Severity

The single highest-leverage change for most log pipelines is dropping everything below a minimum severity. Choose Filter by Severity and set Severity to WARN, which keeps warn, error, and fatal while dropping trace, debug, and info.

Watch the preview. The left pane holds 112 recent logs, and the right pane holds 12. Every record that survives is a warn or an error, which is exactly what an on-call engineer actually pages on.

Severity filtering is blunt on purpose, and it is the right first move when a service is debug-heavy in production. If you need the informational logs for anything, send them to your cheap destination rather than dropping them outright.

Filter by Condition

Severity is not always the axis you care about. Filter by Condition takes any condition you can express over the record. That makes it the right tool when the noise is a specific kind of event rather than a specific level.

Health-check traffic is the classic example. Kubernetes liveness and readiness probes hit an endpoint every few seconds per pod. They succeed essentially always, and nobody has ever debugged an incident from them. Set Action to exclude and add a statement per endpoint.

One detail is easy to get wrong. Statements inside a single condition group are combined with AND. Putting both endpoints in one group asks for a record whose http.target equals two different values at once, which never matches anything. Use Add Condition to give each endpoint its own group and join them with OR, as shown above. The preview confirms it, dropping 107 logs to 93.

The same processor handles metrics and traces. To drop whole metric families you are not using, Filter by Metric Name takes the metric names directly.

Delete Fields

Filtering decides which records you keep. Field deletion decides how big the kept ones are. High-cardinality fields like request IDs, session tokens, and container UIDs add bytes to every event that reaches your destination. Many of them are never queried.

Remember those fields we spotted when we expanded a record. Delete Fields removes them by name. Here we drop session_token, container_id, k8s.pod.uid, thread_id, and build_sha from the body.

The record count is identical on both sides, at 107 logs, which is the point. Nothing was dropped. Compare the bodies though: on the left every record opens with build_sha, and on the right the removed fields are simply gone. You keep every event and pay for less of it.

Sample Logs

Sometimes you want less of a stream without losing the category entirely. HTTP access logs are the usual case. Thousands of near-identical successful requests per second still have statistical value in aggregate, even if no single one matters.

Sample Logs forwards a random fraction and drops the rest. A Drop Ratio of 0.75 keeps one record in four.

Two cautions. Never sample error or fatal records, because losing three quarters of your errors defeats the purpose of collecting them. And pick sampling over severity filtering when you want the shape of a stream preserved. The two overlap: a WARN severity filter has already dropped every info access log this sampler would have thinned. Stacking both on the same branch leaves the sampler with nothing to do.

The Result

Combine two or three of these techniques on your highest-volume sources. That is typically enough to reach a 20 percent or greater reduction in total ingestion cost.

Our production branch now runs a severity filter and field deletion together. Those two are complementary, since one reduces how many records go out and the other reduces how large the survivors are.

Cold storage still receives the full 5.4 MB/m, because we deliberately left that branch alone and it is cheap. The Splunk branch is down to 429 KB/m, and we did not lose a single warning or error along the way. This particular service was unusually debug-heavy, which is why it lands so far past that mark. Your own reduction depends on how noisy your streams are, and the live preview tells you before you commit.

Filtering, trimming, and sampling are all straightforward ways to cut data costs while keeping the value in your telemetry. These processors live in the pipeline rather than in your instrumentation. You can roll all of this out to a fleet without asking a single application team to change code or redeploy.

To learn more about Bindplane, visit bindplane.com or come talk to us in the community Slack.