Skip to Content
SourcesFluentd

Fluentd

Fluentd  is a CNCF-graduated open-source data collector that aggregates logs and events from many sources into a unified pipeline.

Why isn’t direct Fluentd to GlassFlow supported today?

GlassFlow’s OTLP receiver requires the x-glassflow-pipeline-id HTTP header on every request. Fluentd itself does not have a maintained output plugin that can satisfy this requirement:

  • The canonical OTel output for Fluentd is the community-maintained fluent-plugin-opentelemetry gem. Its <http> config block accepts only endpoint, proxy, error_response_as_unrecoverable, retryable_response_codes, compress, and timeouts. The underlying http_output_handler.rb hardcodes the request headers and exposes no override hook. There is no headers { } block.
  • The bundled out_http plugin can set custom headers but cannot emit OTLP-protobuf bodies; it has no codec for OTLP encoding.

Forward Fluentd events to a local OpenTelemetry Collector via the Fluentd Forward protocol, and configure the Collector’s OTLP exporter to send to GlassFlow with the required header. This pattern is validated end-to-end against the GlassFlow OTLP receiver.

OpenTelemetry Collector config

# otel-collector.yaml
receivers:
  fluent_forward:
    endpoint: 0.0.0.0:8006
 
processors:
  batch:
    timeout: 1s
 
exporters:
  otlp_http/glassflow:
    endpoint: http://<glassflow-otlp-receiver>:4318
    compression: none
    headers:
      x-glassflow-pipeline-id: <your-pipeline-id>
 
service:
  pipelines:
    logs:
      receivers: [fluent_forward]
      processors: [batch]
      exporters: [otlp_http/glassflow]

Disable gzip compression on the otlp_http exporter (compression: none). The GlassFlow OTLP receiver does not decompress gzip-encoded payloads and will return 400 Bad Request if you leave the exporter’s default gzip compression enabled.

Fluentd output

On the Fluentd side, send to the Collector with the forward output:

# fluent.conf
<match **>
  @type forward
  <server>
    name otel-collector
    host otel-collector
    port 8006
  </server>
  <buffer>
    flush_interval 1s
  </buffer>
</match>

Each Fluentd record’s keys land as queryable OTel attributes on the corresponding LogRecord. For example, a Fluentd record {"level": "INFO", "service": "checkout", "message": "order received"} arrives in ClickHouse as:

ColumnValue
bodyorder received
attributes{'level': 'INFO', 'service': 'checkout', 'fluent.tag': '<your-fluentd-tag>'}

The fluent_forward receiver uses the record’s message field as the OTel body and promotes the remaining record fields into attributes, plus a fluent.tag attribute carrying the Fluentd tag the event was matched on.

For the GlassFlow pipeline that writes these records to ClickHouse, see OTLP Examples and the OTLP Schema Reference. For full OTel Collector configuration options including gRPC, TLS, and back-pressure handling, see Sending data to GlassFlow OTLP.

Last updated on
Fluent BitGoogle Cloud Storage