Pipeline

GlassFlow pipelines are the core of the platform, enabling real-time data processing from Kafka to ClickHouse.

What is a Pipeline?

A pipeline in GlassFlow is a configuration that defines:

• Source: Kafka topics to consume from
• Transformations: Data processing steps (deduplication, joins, etc.)
• Sink: ClickHouse tables to write to
• Monitoring: Health checks and metrics

Creating Pipelines

GlassFlow provides two ways to create pipelines:

Web UI

Visual wizard for creating pipelines through the web interface.

Python SDK

Programmatic pipeline creation using the Python SDK.