Kubernetes Components

GlassFlow runs in the glassflow namespace and deploys shared infrastructure alongside per-pipeline workloads.

Shared Components

These components are deployed once and shared across all pipelines.

Pod name	Kind	Purpose
`glassflow-api-*`	Deployment	REST API for pipeline CRUD, orchestration, and integration with the UI and Python SDK
`glassflow-ui-*`	Deployment	Web interface for pipeline configuration and real-time monitoring
`glassflow-controller-manager-*`	Deployment	Kubernetes operator that watches Pipeline CRDs and manages their lifecycle, scaling, and updates
`glassflow-postgresql-*`	StatefulSet	PostgreSQL database for storing pipeline configuration

Pod name	Kind	Purpose
`glassflow-nats-{0..N}`	StatefulSet	NATS JetStream cluster (3 or 5 nodes) providing persistent messaging, high availability, and automatic failover between pipeline stages
`glassflow-nats-box-*`	Deployment	Utility container with NATS CLI tools for debugging and cluster administration

Pod name	Kind	Purpose
`glassflow-otlp-receiver-*`	Deployment	Accepts OpenTelemetry logs, traces, and metrics over gRPC (4317) and HTTP (4318). Flattens OTLP protobuf to JSON and routes data to the correct pipeline via the `x-glassflow-pipeline-id` header. Deployed once and shared across all OTLP pipelines. Only deployed when at least one pipeline uses an OTLP source.

Pod name	Kind	Purpose
`glassflow-otel-collector-*`	Deployment	Collects metrics and logs from pipeline components; exposes Prometheus metrics on `/metrics` port `9090`
`glassflow-prometheus-nats-exporter-*`	Deployment	Exports NATS server and JetStream metrics for Prometheus

CRD name	Purpose
`pipelines.etl.glassflow.io`	Declarative Pipeline resource managed by the controller — defines sources, transforms, join, sink, and scaling for each pipeline

Each pipeline runs in its own namespace (pipeline-{pipeline-name}-{unique-id}) to isolate resources and security boundaries.

Workload	Kind	Name pattern	Purpose
Ingestor	StatefulSet	`ingestor-{replica-id}`	Consumes data from the configured source (Kafka or OTLP) and publishes to NATS JetStream. Scales horizontally.
Transform	StatefulSet	`dedup-{replica-id}`	Runs filter (expr), deduplication (BadgerDB), and stateless transformations (expr) in order. Each capability activates only when configured. Scales horizontally.
Sink	StatefulSet	`sink-{replica-id}`	Batches messages from NATS JetStream and writes to ClickHouse with retry logic and connection pooling. Scales horizontally.

For guidance on replica counts and resource values, see the Scaling Guide.

Area	Details
Namespace	All shared components run in the `glassflow` namespace with dedicated RBAC, network policies, and resource quotas
High availability	NATS cluster (3 or 5 nodes) provides fault tolerance, automatic failover, and data replication
Storage	Persistent volumes for NATS JetStream data and application logs; ConfigMaps and Secrets for configuration
Networking	Internal communication via Kubernetes DNS; NATS cluster via headless services; external access via LoadBalancer or Ingress; TLS encryption for NATS cluster traffic