Skip to Content
SourcesAWS CloudWatch

AWS CloudWatch

AWS CloudWatch  is AWS’s native monitoring service. Every managed AWS service emits its logs and metrics to CloudWatch by default, which makes it the canonical source of telemetry for AWS workloads.

Why isn’t direct CloudWatch to GlassFlow supported today?

CloudWatch has no first-party way to push data to a third-party OTLP endpoint with a custom HTTP header. AWS Marketplace integrations cover a fixed set of partner vendors, and the CloudWatch Logs subscription path through Kinesis Data Firehose’s HTTP endpoint cannot inject per-record headers without a Lambda transform. Either route is strictly more work than running the standard OpenTelemetry Collector’s pull-based receiver, so GlassFlow’s recommended path is the Collector relay below.

Run an OpenTelemetry Collector  with the awscloudwatchreceiver (alpha; covers both logs and metrics) and forward to the GlassFlow OTLP receiver via otlp_http. The Collector polls CloudWatch over standard AWS APIs using the SDK credential chain (env vars, shared credentials file, or EC2 IMDS).

OpenTelemetry Collector config

# otel-collector.yaml
receivers:
  awscloudwatch/logs:
    region: <aws-region>
    logs:
      poll_interval: 1m
      groups:
        named:
          /your/log-group-name:
            names: [your-log-stream]
 
  awscloudwatch/metrics:
    region: <aws-region>
    metrics:
      collection_interval: 1m
      period: 60s
      delay: 10m
      queries:
        - namespace: AWS/EC2
          metric_name: CPUUtilization
          dimensions:
            InstanceId: i-0123456789abcdef0
 
processors:
  batch:
    timeout: 1s
 
exporters:
  otlp_http/glassflow_logs:
    endpoint: http://<glassflow-otlp-receiver>:4318
    compression: none
    headers:
      x-glassflow-pipeline-id: <your-logs-pipeline-id>
 
  otlp_http/glassflow_metrics:
    endpoint: http://<glassflow-otlp-receiver>:4318
    compression: none
    headers:
      x-glassflow-pipeline-id: <your-metrics-pipeline-id>
 
service:
  pipelines:
    logs:
      receivers: [awscloudwatch/logs]
      processors: [batch]
      exporters: [otlp_http/glassflow_logs]
    metrics:
      receivers: [awscloudwatch/metrics]
      processors: [batch]
      exporters: [otlp_http/glassflow_metrics]

Disable gzip compression on the otlp_http exporter (compression: none). The GlassFlow OTLP receiver does not decompress gzip-encoded payloads and will return 400 Bad Request if you leave the exporter’s default gzip compression enabled.

One OpenTelemetry Collector instance can drive both signals in parallel. Logs and metrics need their own GlassFlow pipelines (otlp.logs source and otlp.metrics source respectively), so route each signal to its own exporter with the matching x-glassflow-pipeline-id.

Minimum IAM permissions

The credentials used by the Collector require read-only access to the specific CloudWatch APIs called by the receiver. While a wildcard is fine for initial validation, you should restrict the ‘Resource’ scope to specific log group ARNs and metric namespaces in production.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams",
        "logs:FilterLogEvents",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricData"
      ],
      "Resource": "*"
    }
  ]
}

What lands in ClickHouse

A CloudWatch log event {"event_id":"a1","level":"info","message":"hello"} on log group /your/log-group-name, stream your-log-stream, in region eu-central-1 arrives as:

ColumnValue
body{"event_id":"a1","level":"info","message":"hello"}
resource_attributes{'aws.region': 'eu-central-1', 'cloudwatch.log.group.name': '/your/log-group-name', 'cloudwatch.log.stream': 'your-log-stream'}
attributes{'id': '<cloudwatch-event-id>'}

A CloudWatch metric in namespace AWS/EC2, name CPUUtilization, with dimension InstanceId=i-... arrives as:

ColumnValue
metric_nameamazonaws.com/AWS/EC2/CPUUtilization
resource_attributes{'cloud.provider': 'aws', 'cloud.region': 'eu-central-1'}
attributes{'Namespace': 'AWS/EC2', 'MetricName': 'CPUUtilization', 'Dimensions': '{...}'}

The receiver prepends amazonaws.com/<Namespace>/ to the metric name per the CloudWatch Metric Streams OpenTelemetry 1.0 format .

Numeric metric values land in different schema fields depending on the receiver’s stats configuration:

  • No stats (default). Each datapoint is an OTel Summary built from Sum, SampleCount, Minimum, Maximum. Map the GlassFlow OTLP fields sum, count, min, max in your sink to capture the full statistical picture.
  • Explicit stats: [Average] (or any single stat). Each datapoint is a Gauge with one value. Map the GlassFlow OTLP field value_double in your sink.

For full OTLP source field reference, see OTLP Schema Reference. For the GlassFlow pipeline shape that writes these records to ClickHouse, see OTLP Examples.

Last updated on
Apache CassandraConfluent Cloud