OpenTelemetry Collector: Production Best Practices

It's tempting to treat the OpenTelemetry Collector as glue — a shim between your apps and your backend. In production it's the opposite: it's the single pipe every trace, metric, and log flows through, and when it falls over you go blind everywhere at once.

Deploying it well means engineering it with the same rigor you'd give any tier-one service.

The pattern that scales is two layers. A lightweight agent collector runs close to each workload — as a DaemonSet or sidecar — doing cheap local work: receiving telemetry, adding resource attributes, and forwarding on. A separate gateway collector, deployed as a horizontally-scaled service, handles the expensive work: tail-based sampling, sensitive-data scrubbing, and fan-out to backends.

This split keeps per-host overhead tiny while centralizing the policies you want to change without redeploying every node.

Two processors are non-negotiable. The memory limiter caps how much the collector will consume and sheds load before the OS kills the process, which prevents a telemetry spike from cascading into a collector crash. The batch processor groups data before export, dramatically reducing the request volume and CPU cost of talking to your backend.

Order matters: put the memory limiter first in the pipeline so it can protect everything downstream of it.

The collector exports its own internal metrics — queue lengths, dropped spans, export failures, memory pressure. Scrape them and alert on them. A silently dropping collector is the worst failure mode in observability, because your dashboards look fine right up until you realize they've been lying for an hour.

Roll the collector out with the same care as any critical service — staged deploys, health checks, and enough replicas to lose one without losing coverage.