Transformations

GlassFlow supports several transformations that can be applied to your data as it flows through the pipeline. Transformations allow you to process, filter, and combine data before it reaches the final destination in ClickHouse.

Available Transformations

• Deduplication - Remove duplicate events from your data stream based on a unique identifier field
• Join - Combine data from multiple Kafka topics based on join keys and time windows
• Filter - Selectively process events based on configurable expressions
• Stateless Transformation - Reshape event payloads on the fly using expression-based mappings

Transformation Order

Transformations are applied in the following order within a pipeline:

1. Filter - Applied at the ingestor stage, before any other processing
2. Deduplication - Applied after ingestion, before joining
3. Stateless Transformation - Applied after deduplication, before joining
4. Join - Applied after stateless transformations, before sinking to ClickHouse