Load Test Results

Last Updated: May 22, 2025

Overview

This document presents the results from a load testing run of the ETL pipeline with GlassFlow. The test was conducted to evaluate the performance and reliability of the system under various configurations.

Test Data

The complete test results are available in the etl-clickhouse-loadtest  repository. The results include detailed metrics for each test variant, including throughput, latency, and processing times.

Test Data Format

The tests were conducted using user event data with the following JSON structure:

{
    "event_id": "$uuid4",
    "user_id": "$uuid4",
    "name": "$name",
    "email": "$email",
    "created_at": "$datetime(%Y-%m-%d %H:%M:%S)"
}

Test Scope

These load tests focused specifically on the deduplication functionality of GlassFlow. The tests did not include temporal joins. Each test run:

• Generated events with the above schema
• Applied deduplication based on event_id
• Measured performance metrics for the deduplication process

Discussion

We encourage discussion and feedback on these results. Please join the conversation in our GitHub Discussions  where you can:

• Share your observations
• Ask questions about the test methodology
• Discuss potential optimizations
• Compare results with your own testing

Test results

GlassFlow Pipeline Parameters

Test Results

Result parameters

Performance Graph

Key Findings

Stability

• The system was able to handle the load without any issues, even upto 55k RPS.
• The system was able to deduplicate the data without any issues

GlassFlow Processing RPS

• GlassFlow processing RPS was stable even under high load
• The processing RPS seems to peak at 9000 records per second in the current setup
• The processing RPS is limited by resources of the machine running GlassFlow (currently in docker container)

Lag

• Lag was directly proportional to the Ingestion Kafka RPS and the amount of data sent at the given RPS.
• For a given Ingestion Kafka RPS, lag was increasing with the amount of data sent (expected)
• Since GlassFlow RPS was maxed at around 9000 records per second, lag was increasing with the increased Ingestion Kafka RPS

Resource Utilization

• CPU utilization was efficient across all test variants
• Memory usage remained stable during extended test runs

Next Steps

1. Further Testing

   • Longer duration tests
   • Higher throughput scenarios
   • Tests with different events data

2. Batch Size Optimization

   • Testing with different batch sizes to compare performance and lag

3. Monitoring Improvements

   • Enhanced metrics collection
   • Advanced analysis of the results

4. Hardware setup

   • Testing on different hardware
   • Tests with remote clickhouse and kafka to check performance with network delay