Transformations

GlassFlow supports several transformations that can be applied to your data as it flows through the pipeline. Transformations allow you to process, filter, and combine data before it reaches the final destination in ClickHouse.

Available Transformations

• Deduplication — Remove duplicate events from your data stream based on a unique identifier field.
• Join — Combine data from multiple Kafka topics based on join keys and time windows.
• Filter — Drop events that match a configurable expression. Events that do not match the expression pass through the pipeline.
• Stateless Transformation — Reshape event payloads on the fly using expression-based mappings.

Transformation Order

Transformations are applied in the following order within a pipeline:

1. Filter — Applied in the Transform stage, alongside deduplication and stateless transformations. Events that match the filter expression are dropped before deduplication or stateless transforms run.
2. Deduplication — Applied in the Transform stage, after filtering.
3. Stateless Transformation — Applied in the Transform stage, after deduplication.
4. Join — Applied after the Transform stage, before sinking to ClickHouse.