Skip to Content
ConfigurationTransformationsFilter

Filter

The Filter transformation allows you to selectively drop events based on configurable expressions. Events that match the filter expression are dropped and not processed further. Events that do not match the filter expression continue through the rest of the pipeline.

How It Works

Filtering in GlassFlow runs in the Transform stage (Stage 3), the same stage as deduplication and stateless transformations. Events are read from NATS JetStream and evaluated against the filter expression before deduplication or stateless transforms run. The filter uses expression-based evaluation to determine whether an event should be processed.

Internal Process

  1. Event Reception: Events are read from NATS JetStream (after the Ingestor stage)
  2. Expression Evaluation: Each event is evaluated against the configured filter expression
  3. Filtering Decision:
    • If the expression evaluates to true, the event is dropped and not processed further
    • If the expression evaluates to false, the event passes through to the rest of the pipeline
  4. Processing: Only events where the expression evaluated to false continue through the pipeline (deduplication, stateless transformations, join, sink)

Expression Language

GlassFlow uses the expr  expression language for filter expressions. This provides a simple, safe way to evaluate conditions on your event data.

Key Features:

  • Field-based evaluation using event field names
  • Support for common comparison operators (==, !=, >, <, >=, <=)
  • Logical operators (and, or, not)
  • Type-safe evaluation based on field types

Configuration

Filter is configured at the pipeline level. The expression field defines the drop condition: events for which the expression evaluates to true are discarded, and events for which it evaluates to false pass through.

Expression Syntax

Filter expressions use field names from your event schema and support the following operations:

Comparison Operators

  • == - Equality
  • != - Inequality
  • > - Greater than
  • < - Less than
  • >= - Greater than or equal
  • <= - Less than or equal

Logical Operators

  • and - Logical AND
  • or - Logical OR
  • not - Logical NOT

Examples

The filter expression defines the drop condition. When the expression evaluates to true, the event is dropped. When it evaluates to false, the event passes through. Write your expression to match the events you want to exclude, not the events you want to keep.

Drop events where status equals 'active' (string comparison):

Drop events where age is greater than 18 (numeric comparison):

Drop events where is_premium is true (boolean field):

Drop events matching multiple conditions with AND:

Drop events matching any condition with OR:

Drop events matching a complex expression with parentheses:

Drop events using nested field access:

Best Practices

Expression Design

  • Write drop conditions: The expression defines what to discard, not what to keep. If you want to keep events where status == 'active', write the expression as status != 'active' to drop everything else.
  • Keep expressions simple: Complex expressions can be harder to maintain and debug
  • Use parentheses: Explicitly group conditions with parentheses for clarity
  • Test expressions: Validate filter expressions before deploying to production

Field Names

  • Use exact field names: Field names in expressions must match exactly with your event schema
  • Case sensitivity: Field names are case-sensitive
  • Nested fields: Use dot notation for nested fields (e.g., user.age)

Type Safety

  • Match field types: Ensure comparison values match the field types in your schema
  • String literals: Use single quotes for string literals in expressions
  • Numeric values: Use numeric literals without quotes for numeric comparisons
  • Boolean values: Use true or false (lowercase) for boolean comparisons

Example Configuration

Here’s a complete example of a pipeline with filtering enabled. In this example, events where age > 18 and status == 'active' are dropped. Events that do not meet both conditions pass through to ClickHouse.

{
  "version": "v2",
  "pipeline_id": "filtered-pipeline",
  "name": "Filtered Events Pipeline",
  "source": {
    "type": "kafka",
    "connection_params": {
      ...
    },
    "topics": [
      {
        "name": "user-events",
        "consumer_group_initial_offset": "latest",
        "replicas": 1
      }
    ]
  },
  "filter": {
    "enabled": true,
    "expression": "age > 18 and status == 'active'"
  },
  "sink": {
    "type": "clickhouse",
    ...
  }
}
Last updated on
DeduplicationJoin