WIUNIX ACADEMY
OpenTelemetry Collector queue full: bound memory, retries and recovery
Measure accepted, queued and failed telemetry; size a bounded exporter path and rehearse a downstream outage without treating the Collector as a durable message broker.
Make this guide yoursYOUR SYSTEM. YOUR STEPS.
Where are you working?
Choose your OS and version. The prerequisites, commands and troubleshooting follow your environment.
USE YOUR OWN VALUES
Configure this tutorial
Set the example values for your environment; related explanations and copy panels update together. Never enter passwords, private keys or tokens. Values stay in this tab and are not sent to the server or share links. Check actual DNS, certificates and network policy before running commands.
Different IP addresses, domains or ports? Open the settings below, change the values and select “Apply to tutorial”, then copy the updated commands.
Set values for my environment6 editable values
THE OUTCOME
What will you have at the end?
Diagnose exporter backpressure and review a bounded traces pipeline with explicit loss, restart and capacity assumptions.
Supported operating systems and versions
Documented candidate configuration and diagnostic method; no Collector was started/restarted, no endpoint was disrupted and no delivery/load guarantee is claimed.
Before you begin
- Existing supported otelcol-contrib binary and its exact-version component documentation
- Authorized access to internal metrics and backend delivery evidence
- Separate approved test endpoint/generator for any outage rehearsal; no private payloads
LET’S BUILD
Prepare the selected operator host
These are operator-host packages only. Review the transaction before approving; do not install tools inside application containers. The cluster, server and application binaries must already exist with owner-approved versions. Use Bash for the following panels. Package installation changes this selected host but starts no new application server.
Collector core/contrib/vendor components and queue sizer fields vary by version. The existing binary must include the shown components and pass its own validate command before a separate rehearsal; OS selection does not select or upgrade Collector.
Selected operator host; approved package transaction
sudo dnf install ca-certificates bash coreutils openssl iproute
if ! command -v curl >/dev/null 2>&1; then sudo dnf install curl; fi
cat /etc/os-release
command -v bash curl timeout
'/usr/local/bin/otelcol-contrib' --version
'/usr/local/bin/otelcol-contrib' components
openssl version
ss --versionThe OS/version matches the selection, required tools resolve, and existing application/client versions satisfy the stated compatibility boundary. Stop if a binary is missing or incompatible.
Preserve host security and existing paths
Keep SELinux enforcing and firewalld rules in place. These client-side diagnostics require outbound access to the approved API/TLS/SFTP destination; they need no new inbound host port. A failure in a confined systemd service may differ from an interactive shell. Review AVC denials, file labels and service permissions with the policy owner rather than disabling SELinux, broad allow rules or disabling the firewall. Trust-chain/crypto-policy failures require a compatible endpoint, not weakened system crypto. Existing custom data/config paths need their established labels; never run a recursive relabel blindly.
Selected operator host; read-only security inspection
if command -v getenforce >/dev/null 2>&1; then getenforce; fi
if command -v firewall-cmd >/dev/null 2>&1; then firewall-cmd --state; fiDefine the reliability boundary
A Collector accepts telemetry from a receiver, processes it and hands it to an exporter. Queue admission, a successful export attempt and backend queryability are different events. Retry cannot recover data rejected before queue admission, and a memory queue does not survive restart. Permanent errors can bypass retry.
The sample is a candidate traces-only configuration for an existing otelcol-contrib binary, not a drop-in replacement for production pipelines. It receives locally over gRPC and exports to a reviewed TLS endpoint. Preserve existing authentication, privacy filters and other signals when integrating through the owner.
LET’S BUILD
Pin the actual binary and components
Run as the unprivileged operator on the Collector host. Record version and installed components; core, contrib, vendor and custom distributions differ. Match documentation to that exact version and inspect the existing service/configuration paths privately. Do not upgrade a running Collector merely to make a sample parse.
Operator terminal; authorized lab context
'/usr/local/bin/otelcol-contrib' --version
'/usr/local/bin/otelcol-contrib' components
'/usr/local/bin/otelcol-contrib' validate --helpInstalled version, otlp receiver/exporter, memory_limiter and batch availability are known.
LET’S BUILD
Read rates and queue occupancy together
Use the existing protected internal telemetry endpoint; this sample assumes a loopback Prometheus reader already exists at the selected port. Internal metric names, units and _total suffixes depend on the version/reader. Inspect actual output rather than copying a dashboard label. Bound this request and keep resource attributes/endpoint labels private.
Operator terminal; authorized lab context
set -o pipefail
curl --noproxy '*' --fail --silent --show-error --max-time 5 'http://127.0.0.1:8888/metrics' | grep -E '^otelcol_(receiver_(accepted|refused)|exporter_(queue|enqueue_failed|send_failed|sent)|processor_refused|process_memory)'Capture accepted/refused rates, queue size/capacity, enqueue failures, send failures and memory over the same interval.
Calculate an outage budget before enlarging the queue
Queue size must be interpreted in its configured sizer units. With requests, 1000 means requests/batches, not spans or bytes; a batch-size setting is not a total-memory bound. Approximate backlog growth as incoming request rate minus sustainable export rate. Required capacity for an outage is that positive rate multiplied by outage seconds, plus measured burst headroom.
Measure representative serialized/request memory and include batch processors, receiver buffers and runtime overhead. A queue of 2000 requests at 20 requests/second covers roughly100seconds with no drain; this is arithmetic, not a measured guarantee. Infinite retry can turn a network incident into disk/memory exhaustion.
The exporter max_elapsed_time limits retry work for an actively attempted request; it is not a TTL attached to every item when it enters the queue. Queue wait age and active retry duration differ.