Load Test Setup Guide
The complete load testing code is available in the clickhouse-etl-loadtest repository.
Test Environment
Hardware Specifications
The load tests were conducted on a MacBook Pro with the following specifications:
Specification | Details |
---|---|
Model Name | MacBook Pro |
Model Identifier | Mac14,5 |
Model Number | MPHG3D/A |
Chip | Apple M2 Max |
Total Number of Cores | 12 (8 performance and 4 efficiency) |
Memory | 32 GB |
Software Stack
The test environment uses Docker containers for all components:
- Kafka: Message broker for event streaming
- ClickHouse: Database for storing and querying the processed events
- GlassFlow: Data pipeline tool for processing and transforming events
Test Architecture
Component Overview
Test Flow
-
Each test run:
- Creates a new GlassFlow pipeline
- Configures the pipeline with test parameters
- Sends test data to Kafka
- Monitors the data flow through the pipeline
- Verifies results in ClickHouse
- Reports metrics for the run
-
Success Criteria:
- Test is considered successful when all expected data is available in ClickHouse
- Metrics are collected and reported for each run
Performance Considerations
Kafka Ingestion Rate (Records Per Second)
- The tool attempts to achieve the requested RPS
- The parameter controlling the Kafka ingestion rate is
num_processes
- The actual achieved RPS may vary based on:
- System resources
- Network conditions
- Pipeline configuration
- The results table shows the actual RPS achieved for each run
Lag Measurement
- Lag is defined as the time difference between:
- When the last event is sent to Kafka
- When that event becomes available in ClickHouse
- Lag is influenced by:
- Target RPS
- Total number of events
- Pipeline configuration
- System performance
GlassFlow Processing RPS (Records Per Second)
- The results shows the average RPS processed by GlassFlow
Configuration Parameters
Test Parameters
The load test can be configured using the following parameters in load_test_params.json
:
Parameter | Required/Optional | Description | Example Range/Values | Default |
---|---|---|---|---|
num_processes | Required | Number of parallel processes | 1-N (step: 1) | - |
total_records | Required | Total number of records to generate | 500,000-5,000,000 (step: 500,000) | - |
duplication_rate | Optional | Rate of duplicate records | 0.1 (10% duplicates) | 0.1 |
deduplication_window | Optional | Time window for deduplication | [“1h”, “4h”] | “8h” |
max_batch_size | Optional | Max batch size for the sink | [5000] | 5000 |
max_delay_time | Optional | Max delay time for the sink | [”10s”] | ”10s” |
Example Configuration
Example configuration:
{
"parameters": {
"num_processes": {
"min": 1,
"max": 4,
"step": 1,
"description": "Number of parallel processes to run"
},
"total_records": {
"min": 5000000,
"max": 10000000,
"step": 5000000,
"description": "Total number of records to generate"
}
},
"max_combinations": 1
}
ClickHouse Sink Parameters
These parameters affect the performance of the ClickHouse sink component in GlassFlow:
-
Batch Size
- Controls how many records are processed in a single batch
- Higher batch sizes generally provide better performance under high load
- Should be configured based on:
- Expected load
- Available memory
- Latency requirements
-
Delay Time
- Maximum time to wait before processing a batch
- Affects the balance between latency and throughput
- Should be tuned based on:
- Real-time processing requirements
- System resources
- Expected load patterns
Running the Tests
Prerequisites
- Docker installed and running
- Python 3.x installed
- Required Python packages installed
Test Execution
# Run a load test with specific configuration
python main.py --test-id load_test_1 --config load_test_params.json
# Analyze the results
python results.py --results-file results/load_test_1.csv
Test Results File Format
The following metrics are collected and displayed for each test run:
Metric | Description | Unit |
---|---|---|
duration_sec | Total time taken for the test | seconds |
result_num_records | Number of records processed | count |
result_time_taken_publish_ms | Time taken to publish records to Kafka | milliseconds |
result_time_taken_ms | Time taken to process records through the pipeline | milliseconds |
result_kafka_ingestion_rps | Records per second sent to Kafka | records/second |
result_avg_latency_ms | Average latency per record | milliseconds |
result_success | Whether the test completed successfully | boolean |
result_lag_ms | Lag between data generation and processing | milliseconds |
result_glassflow_rps | Records per second processed by GlassFlow | records/second |
Example Results
{
"Parameters": {
"Variant ID": "load_bd0fdf39",
"Max Batch Size": 5000,
"Duplication Rate": 0.1,
"Deduplication Window": "8h",
"Max Delay Time": "10s"
},
"Results": {
"Success": "True",
"Number of Records": "20.0M",
"Time to Publish": "2289.22 s",
"Source RPS in Kafka": "8737 records/s",
"GlassFlow RPS": "8556 records/s",
"Time to Process": "2337.666 s",
"Average Latency": "0.0001 s",
"Lag": "47.738 s"
}
}
Best Practices
-
Resource Management
- Monitor Docker container resource usage
- Ensure sufficient memory allocation
- Watch for container limits
-
Test Planning
- Start with conservative parameters
- Gradually increase load
- Monitor system resources
- Document all parameter changes
-
Data Verification
- Verify data consistency in ClickHouse
- Check for any data loss
- Validate processing order when required
Last updated on