Changelog

GlassFlow now supports including any external Python libraries in your transformation function. This allows you to easily manage and integrate additional Python packages needed for your data transformations. You can easily add external libraries to your transformer functions by editing requirements.txt file in the WebApp.

Read more about Python dependencies for transformation.

Release notes: Version: 1.5.0

You can now configure a Webhook as a sink within your GlassFlow pipelines, allowing you to send transformed data directly to any custom endpoint that supports Webhooks.

Key Highlights:

• Real-Time Data Streaming: Instantly stream processed events to external services such as Slack, or your own custom applications.

• Custom Workflow Automation: This feature is perfect for integrating GlassFlow with automation tools like Zapier, providing endless possibilities for automating tasks and creating powerful workflows.

For more details on how to set up and use the Webhook Sink Connector, check out our updated documentation.

We’re continuously working on improving GlassFlow and adding more connectors to simplify your data integration and processing needs. Stay tuned for more exciting updates!

We added an easy-to-follow Getting Started Guide in Jupyter Notebook format.

• Includes step-by-step instructions on how to set up and configure a GlassFlow pipeline using the WebApp.

• Provides sample code for both data producers and consumers, allowing users to run all necessary commands in one place.

• Facilitates a hands-on learning experience, making it easier for new users to explore GlassFlow's capabilities directly in their local development environment.

Download and run the Jupyter Notebook guide.

These new integrations empower you to connect GlassFlow with a broader set of data sources, making it easier than ever to build powerful, real-time data pipelines that fit your specific needs. You can select a data source in GlassFlow WebApp as a part of the pipeline creation process. Amazon Simple Queue Service connector - GlassFlow now supports Amazon SQS as a data source. You can easily connect your pipelines to Amazon SQS queues, allowing seamless integration with your existing AWS infrastructure. Google Pub/Sub connector- We've added support for Google Pub/Sub as a data source in GlassFlow. You can now subscribe to Google Pub/Sub topics and ingest real-time events directly into your pipelines. This integration is perfect for building scalable, event-driven architectures in Google Cloud. Webhook connector - GlassFlow now allows you to set up custom integrations using Webhooks. This flexible feature enables you to receive events from any service that supports Webhooks, including automation platforms like Zapier. Explore how to connect now. Release notes: WebApp v1.3.3

• GlassFlow documentation and blog are hosted on new domains:

  • Newly designed blog: blog.glassflow.dev

  • Visit the documentation: docs.glassflow.dev

• Integration page with the type of connectors:

  • Updated the Integration page on documentation to include all possible types of connectors available in GlassFlow.

  • Added detailed descriptions and examples for each type of connector.

• New managed connectors:

  • Created new managed connectors for data sources: Webhook, Amazon SQS, and Google Pub/Sub.

  • These pages provide detailed instructions and examples for integrating with these data sources.

• Use Case updates:

  • Updated existing Use Cases to demonstrate the use of the GlassFlow WebApp for creating new pipelines, replacing the previous CLI instructions.

  • These updates provide a more user-friendly and visual guide for setting up pipelines.

  

When creating a new pipeline via the GlassFlow WebApp, you can write and edit your transformation function code in Python directly within the interface. This new editor simplifies the process of defining and customizing data transformations, allowing you to streamline your data pipeline setup without the need for external developer tools. There is also an option to choose a transformer template from the dropdown menu.

Try it out now: app.glassflow.dev Release notes: WebApp v1.3.3

We have streamlined the pipeline client usage via the Python SDK by removing the requirement for passing space_id, pipeline_id and pipeline_access_token. Now, you can send and receive events from the pipeline without specifying pipeline credentials explicitly in the code.

How it works

Set environment variables with your actual GlassFlow pipeline credentials such as PIPELINE_ID and PIPELINE_ACCESS_TOKEN:

export PIPELINE_ID=your_pipeline_id
export PIPELINE_ACCESS_TOKEN=your_access_token

GlassFlow locates environment variables automatically and passes them internally to the Python SDK client:

Example simplified SDK code:

import glassflow

client = glassflow.GlassFlowClient()
pipeline_client = client.pipeline_client()

data = {
    "id": "123",
    "source": "sdk testing"  
} # your JSON data to send to the pipeline 

response = pipeline_client.publish(data)

Old Way Still Supported

For those with existing setups, the old way of passing pipeline credentials is still supported. Try it out now: Published Data or Consume Data Release notes: Python SDK v1.0.5

We're excited to announce a powerful new feature in the GlassFlow WebApp: the Built-in Editor for transformation functions. Now, when creating a new pipeline, you can write and edit your transformation function code directly within the GlassFlow interface. Also, you can choose a sample transformer template from the menu. Try it out now: app.glassflow.dev For more details, visit the Quickstart using the Web App page.

You can now create your pipelines through a simplified process on GlassFlow WebApp. You are guided through each step of the pipeline creation process with the option to choose consumer, transformation function, and producer from the UI. Inline links to relevant documentation and learning materials help you understand each step.

Try it out now: app.glassflow.dev For more details, visit the Quickstart using the Web App page. Release notes: WebApp v1.3.1

With a new update on the GlassFlow WebApp, you can now copy your new/existing pipeline credentials (Pipeline ID, Space ID, and Access Token) with one action (Copy as Configuration File). So, you can easily paste them into a .env file instead of locating them from different places in the UI. Ready to discover? Choose the existing pipeline, click, and open the pipeline view. Click on the Copy Configuration File option. Release notes: WebApp v1.2.11