Skip to Content
SourcesConfluent Cloud

Confluent Cloud

Confluent Cloud is a managed Kafka platform that simplifies operating and scaling streaming data infrastructure. It adds enterprise features like governance, schema registry, connectors, and cloud-native deployment on top of Apache Kafka®.

Using Confluent Cloud with GlassFlow

Confluent Cloud is Kafka API-compatible. Connect with the GlassFlow Kafka source using SASL_SSL + PLAIN and a Confluent API key/secret as the credentials:

{
  "type": "kafka",
  "source_id": "events",
  "connection_params": {
    "brokers": ["pkc-xxxxx.region.aws.confluent.cloud:9092"],
    "protocol": "SASL_SSL",
    "mechanism": "PLAIN",
    "username": "<CONFLUENT_API_KEY>",
    "password": "<CONFLUENT_API_SECRET>"
  },
  "topic": "events",
  "consumer_group_initial_offset": "earliest",
  "schema_fields": [
    {"name": "event_id", "type": "string"},
    {"name": "timestamp", "type": "datetime"}
  ]
}

The API key and secret are created in the Confluent Cloud console under Cluster → API Keys. Confluent Cloud requires TLS, so always use SASL_SSL rather than SASL_PLAINTEXT.

Schema Registry Enterprise

GlassFlow can read schemas from a Confluent-compatible Schema Registry in the Enterprise Edition. Add a schema_registry object to the source and GlassFlow fetches the schema referenced by the ID in each message’s Confluent envelope. This works for JSON, Avro, and Protobuf sources. See Data Formats for the full configuration.

On the Open Source Edition, define your fields explicitly with schema_fields on the source, as shown above.

Last updated on
AWS CloudWatchDelta Lake