Use cases

Streaming data pipelines

Data pipeline applications involve the movement and transformation of data from various sources to destinations like databases, data lakes, or analytics platforms. These pipelines are essential for data integration, ETL (extract, transform, load) processes, and ensuring data is available where and when needed. Many modern distributed applications require real-time data processing capabilities, which traditional batch processing pipelines struggle to provide.

GlassFlow operates in a continuous streaming mode instead of being periodically triggered. It can read records from sources that continuously produce data and move them with low latency to their destination.

How GlassFlow helps:

• GlassFlow efficiently transforms data streams with custom logic to fit the requirements of downstream applications or cloud storage solutions.

• Developers do not need to deal with infrastructure and data scientists should not need to know Kubernetes. GlassFlow provides a common platform suited to each role's specific needs. Data scientists can plan their data models and immediately run them on real-time data streams.

• With the CLI tool, GlassFlow integrates smoothly into existing CI/CD workflows, making it easier to deploy and update data pipelines alongside application code.

• GlassFlow streamlines the process of capturing, transforming, and loading event-based data into data warehouses. Data is loaded into the data warehouse continuously in the right format, at the right time.

Event-driven applications

Event-driven applications respond to actions triggered by users, systems, or sensors, requiring real-time processing to react promptly. They need to be designed to respond dynamically to events or messages rather than relying on traditional request-response interactions.

How GlassFlow helps:

• GlassFlow excels in the real-time ingestion and processing of events, ensuring that applications can immediately react to new information.

• GlassFlow's architecture is designed to manage out-of-order events and late-arriving data efficiently.

• GlassFlow provides seamless integration with a variety of data sources and sinks, such as NATs for message distribution and Change Data Capture (CDC) services like Debezium by directly connecting to your primary database databases like PostgreSQL and MySQL for storage.

• GlassFlow complements the microservice architecture, where applications are decomposed into smaller, independently scalable services that communicate through events. GlassFlow processes those events and maintains asynchronous communication for microservices.

Streaming data for ML applications

Machine Learning (ML) applications increasingly rely on streaming data for real-time analytics, predictions, and decision-making. Common examples of such workloads include machine learning for e-commerce websites, real-time bidding, and mobile gaming.

How GlassFlow helps:

• GlassFlow ensures that ML models have access to the most current data to make accurate predictions based on the latest information. ML models can be continuously trained and updated with new data so that models remain effective as data patterns change over time.

• GlassFlow's serverless architecture allows for the scalable processing of high-volume data streams by running parallel processes. This ensures that ML applications can handle large datasets without degradation in performance.

• GlassFlow enables ML engineers to effortlessly construct pipelines for processing real-time events, extracting vector embeddings via transformation functions, and continuously updating vector databases.

Other real-world scenarios