Architecture

Discover GlassFlow architecture and key components.

Key components

The GlassFlow platform is composed of four main components:

GlassFlow architecture key components

• Developer Interface: Including Web Application, Command Line Interface (CLI), or Python SDK allows users to interact with GlassFlow’s core functionalities.

• GlassFlow API: Acts as the primary entry point for all requests from WebApp, CLI, or Python SDK, handling authentication, and authorization. Through the API, users can create, modify, and manage data pipelines, initiate data transformations, and monitor the status of their real-time data processes.

• Message Broker: Facilitates reliable event messages consuming and streaming, ensuring data flows smoothly between GlassFlow’s backend API service and serverless execution engine. GlassFlow uses NATS’s built-in distributed publish/subscribe messaging system called JetStream.

• Serverless Execution Engine: Enables quick execution of Python code in response to events from the message broker and easy deployment of custom functions to the cloud.

Data Flow

The following diagram illustrates and describes the overall data flow in GlassFlow, from pipeline creation to events processing.

Authentication

The authentication process in GlassFlow, powered by Auth0, provides a secure, efficient, and user-friendly way for users to access the platform through the Web Application or CLI.

GlassFlow authentication flow

Deployment and Operation

Users do not need to deploy their data pipelines manually. The whole process is managed by the GlassFlow platform using containerization (Docker), and orchestration tools (Kubernetes).

When you create a pipeline from a CLI or via the web app, your function code and configuration will be validated and dockerized, a new Docker image is generated, and this image is then submitted to the GlassFlow serverless engine, as part of the deployment process.

Infrastructure as code

When developing a Python-based stream processing application with GlassFlow, you create a new pipeline that defines the flow of data from a source to a destination. You specify the transformation functions that will process the data. The pipeline can be efficiently managed through CLI and visualized in the Web App.

Any modifications to the pipeline configuration in one environment can be seamlessly integrated into another. This capability supports thorough testing of changes in a separate environment before a production deployment.