Wiunix
Talk to an Engineer
← Start of tutorial

WIUNIX ACADEMY

OpenTelemetry Collector queue full: bound memory, retries and recovery

Page 2 of 3. Review the earlier steps and prerequisites before continuing.

YOUR SYSTEM. YOUR STEPS.

Where are you working?

Choose your OS and version. The prerequisites, commands and troubleshooting follow your environment.

Rocky Linux 10Reviewed against official documentation ·

USE YOUR OWN VALUES

Configure this tutorial

Set the example values for your environment; related explanations and copy panels update together. Never enter passwords, private keys or tokens. Values stay in this tab and are not sent to the server or share links. Check actual DNS, certificates and network policy before running commands.

Different IP addresses, domains or ports? Open the settings below, change the values and select “Apply to tutorial”, then copy the updated commands.

Set values for my environment6 editable values
04

LET’S BUILD

Create a separate candidate file

Use a new work directory and file. The next YAML is configuration data. The example limit and batch/queue values are deliberately finite teaching values; size them below actual process/container limits using measurements. No service is restarted by editing this file.

bash

Operator terminal; authorized lab context

umask 077
mkdir '/tmp/academy-otel' && cd '/tmp/academy-otel' || exit 1
(set -C; : > candidate.yaml) || exit 1
${EDITOR:-vi} candidate.yaml
How do I know it worked?

A separate private candidate exists; the active configuration is unchanged.

05

LET’S BUILD

Review a bounded traces pipeline

memory_limiter is first so upstream can receive refusal before other processors allocate more memory. batch sends bounded batches; the exporter queue is finite with explicit consumers and retry horizon. The receiver binds loopback only. The remote endpoint must have a trusted certificate for the selected domain; no insecure TLS option or embedded credential is provided.

yaml · /tmp/academy-otel/candidate.yaml

File contents; save at the displayed filename

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:14317
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 384
    spike_limit_mib: 96
  batch:
    timeout: 1s
    send_batch_size: 256
    send_batch_max_size: 512
exporters:
  otlp/backend:
    endpoint: otel.example.com:4317
    tls:
      insecure: false
    timeout: 5s
    sending_queue:
      enabled: true
      sizer: requests
      queue_size: 1000
      num_consumers: 4
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 10s
      max_elapsed_time: 2m
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/backend]
How do I know it worked?

The candidate defines one local traces receiver and one TLS exporter with finite queue/retries; it has no persistence claim.

06

LET’S BUILD

Validate with the exact installed distribution

A successful validation checks component/configuration compatibility, not remote TLS, delivery, capacity or service health. If a field is unsupported, consult that exact version’s exporterhelper documentation and revise the candidate; do not remove limits blindly. Check port availability before a separately approved isolated rehearsal. Never launch this candidate alongside a process that owns the same listener.

bash

Operator terminal; authorized lab context

'/usr/local/bin/otelcol-contrib' validate --config='/tmp/academy-otel/candidate.yaml'
ss -lntp '( sport = :14317 or sport = :8888 )'
timeout 15s openssl s_client -connect 'otel.example.com:4317' -servername 'otel.example.com' -verify_hostname 'otel.example.com' -verify_return_error </dev/null
How do I know it worked?

Configuration validation passes; existing listeners and TLS trust are understood before any runtime change.

Distinguish retryable failure from rejected data

send_failed increasing can include retried attempts; it is not a direct count of permanently lost spans. enqueue_failed indicates rejection before the exporter retry queue. receiver_refused or memory_limiter refusal must propagate to a sender that can back off; SDK/exporter retry behavior and its own finite queue still matter.

Authentication/invalid-payload failures may be permanent. A full backend disk cannot be solved by enlarging an upstream queue forever. Increase concurrency only when the downstream can accept it; otherwise retry amplification makes the incident worse. Inspect status codes, durations and rate deltas with the backend owner.

Rehearse outage and drain with synthetic IDs

In a separate approved test pipeline, send a small known set of synthetic trace IDs using the existing generator. Interrupt only that test destination through its owning controls, then restore it. Record accepted/refused/enqueue/sent deltas and query the exact IDs at the backend. Compare a short outage with one beyond the active retry horizon, but observe actual loss rather than assuming every queued item expires at that time; queue wait and retry start differ. Never disrupt a shared production exporter for evidence.

Treat persistent queues as a separate design

file_storage can protect queued data across some restarts, but requires a version-reviewed contrib extension, dedicated persistent writable storage, capacity/permissions, lifecycle recovery and restart/outage acceptance on the exact build. It is not an exactly-once broker. Upstream issue15677 reports clean-shutdown loss on0.157.x; that report is a reason to test your version, not proof all releases share the bug. Do not delete a WAL/PVC to make a stuck queue look healthy.

Explore all tutorials ↗