Supported Data Formats

GlassFlow currently supports JSON as the primary data format for Kafka topics.

JSON Format

GlassFlow exclusively supports JSON data format for Kafka topics. This means all messages consumed from Kafka topics must be valid JSON objects.

Example JSON Message:


{
  "event_id": "evt_12345",
  "user_id": "user_789",
  "timestamp": "2024-01-15T10:30:00Z",
  "event_type": "purchase",
  "amount": 99.99
}

Nested JSON Support

GlassFlow provides comprehensive support for nested JSON structures through dot notation field access. This allows you to extract data from deeply nested objects.

Example event with nested fields:


{
  "user": {
    "id": "user_123",
    "profile": {
      "name": "John Doe",
      "email": "[email protected]"
    },
    "created_at": "2024-01-15T10:30:00Z"
  }
}

Field Configuration:


{
  "name": "user.profile.name",
  "type": "string"
}

Full pipeline configuration example


{
  "pipeline_id": "kafka-to-clickhouse-pipeline",
  "source": {
    "type": "kafka",
    "connection_params": {
      "brokers": ["kafka-broker-0:9092"],
      "protocol": "SASL_SSL",
      "mechanism": "PLAINTEXT",
      "username": "<user>",
      "password": "<password>",
    },
    "topics": [
      {
        "name": "users",
        "schema": {
          "type": "json",
          "fields": [
            {
              "name": "user.id",
              "type": "string"
            },
            {
              "name": "user.profile.name",
              "type": "string"
            },
            {
              "name": "user.profile.email",
              "type": "string"
            },
            {
              "name": "created_at",
              "type": "datetime"
            }
          ]
        }
      }
    ]
  },
  "sink": {
    "type": "clickhouse",
    "host": "clickhouse-server",
    "port": "9000",
    "database": "default",
    "username": "default",
    "password": "c2VjcmV0",
    "table": "users",
    "table_mapping": [
      {
        "source_id": "users",
        "field_name": "user.id",
        "column_name": "user_id",
        "column_type": "UUID"
      },
      {
        "source_id": "users",
        "field_name": "user.profile.name",
        "column_name": "user_name",
        "column_type": "string"
      },
      {
        "source_id": "users",
        "field_name": "user.profile.email",
        "column_name": "user_email",
        "column_type": "string"
      },
      {
        "source_id": "users",
        "field_name": "created_at",
        "column_name": "created_at",
        "column_type": "DateTime"
      }
    ]
  }
}

Supported Data Type mappings

GlassFlow supports the following data type mappings to ClickHouse:

Kafka Type	Supported ClickHouse Type Mappings
string	String, FixedString, DataTime, DataTime64, UUID, Enum8, Enum16, LowCardinality(String), LowCardinality(FixedString), LowCardinality(DateTime)
int8	Int8, LowCardinality(Int8)
int16	Int16, LowCardinality(Int16)
int32	Int32, LowCardinality(Int32)
int64	Int64, DateTime, DateTime64, LowCardinality(Int64)
float32	Float32, LowCardinality(Float32)
float64	Float64, DateTime, DateTime64, LowCardinality(Float64), LowCardinality(DateTime)
bool	Bool
bytes	String
array	Array(String), Array(Int8), Array(Int16), Array(Int32), Array(Int64), String