System Components

GlassFlow consists of the following components running as Docker containers:

Docker Containers

1. NATS Server

• Docker Image: nats:alpine
• Purpose: Used as persistance KV storage and as the message broker for internal communication between components
• Features:
  • JetStream enabled for persistent messaging
  • High-performance message delivery
  • Automatic restart capabilities

2. Frontend UI

• Docker Image: glassflow/clickhouse-etl-fe:stable
• Purpose: Web-based user interface for pipeline management
• Features:
  • Intuitive pipeline configuration
  • Real-time monitoring
  • User-friendly interface for managing data operations

3. Backend Application

• Docker Image: glassflow/clickhouse-etl-be:stable
• Purpose: Core ETL engine that provides an API and orchestrates the pipeline
• Features:
  • Handles pipeline orchestration
  • Manages data processing
  • Runs ETL components: Deduplication, Join, ClickHouse sink
  • Maintains persistent logs

4. Nginx

• Docker Image: nginx:1.27-alpine
• Purpose: Serves as a reverse proxy
• Features:
  • Routes traffic to frontend and backend services
  • Serves static files
  • Handles load balancing

Volumes

The setup uses the following Docker volumes:

• logs: Stores application logs for persistence

Environment Variables

The backend service uses the following environment variables:

• GLASSFLOW_LOG_FILE_PATH: Path for log files (/tmp/logs/glassflow)
• GLASSFLOW_NATS_SERVER: NATS server address (nats:4222)