Publish data

This page explains how to publish data to GlassFlow pipelines.

Ways to publish data into GlassFlow

Publishing data in GlassFlow involves sending data to a pipeline for processing. You use built in integrations to publish data or write code in Python for a new custom integration.

Publishing data using integrations

Visit Integrations page for more information.

Publishing data using Python SDK

The Python SDK provides a programmatic way to interact with GlassFlow pipelines to produce or consume data continuously. Using the SDK you create a custom connector for any data source in Python.

Prerequisites

• You created a Pipeline.

• You have the pipeline credentials such as PIPELINE_ID and PIPELINE_ACCESS_TOKEN.

Install GlassFlow Python SDK

Install a GlassFlow SDK using the pipcommand in a terminal.

pip install glassflow

Set environment variables

Set environment variables with your actual GlassFlow pipeline credentials such as PIPELINE_ID and PIPELINE_ACCESS_TOKEN:

export PIPELINE_ID=your_pipeline_id
export PIPELINE_ACCESS_TOKEN=your_pipeline_access_token

Push Data to the pipeline

Create a new Python script file called producer.py and insert the code below. This Python script serves as a data producer and publishes the data to a GlassFlow pipeline.

import glassflow

# create a client for your pipeilne
pipeline_client = glassflow.GlassFlowClient().pipeline_client()
data = {
    "id": "123",
    "source": "sdk testing"  
} # your json data to send to the pipeline 

response = pipeline_client.publish(data)
if response.status_code == 200:
    print("Message sent to GlassFlow:", data)
else:
    print(f"Failed to send data to GlassFlow: {response.text}")

As you can see from the above code, sending data to the GlassFlow pipeline is straightforward:

1. Initializes a client for interacting with the GlassFlow pipeline. The SDK automatically reads the needed parameters (pipeline_id and pipeline_access_token) from the environment variables. Alternatively, you can also pass them as parameters when creating the client:

Without the pipeline credentials params:

pipeline_client = glassflow.GlassFlowClient().pipeline_client()

With the pipeline credentials params:

pipeline_client = glassflow.GlassFlowClient().pipeline_client(
    pipeline_id=pipeline_id, 
    pipeline_access_token=token
)

2. Publishes a new event into the pipeline. It takes one parameter: the JSON data to be published. Under the hood, it sends a request to the GlassFlow API endpoint with the provided data to publish the new event to the given pipeline.

response = glassflow_client.publish(data)

You receive a PublishEventResponse object in response. This response object contains:

• status_code: The status_code attribute holds the HTTP status code of the response.

Refer to Python SDK documentation for more details.

Run the script

Run the Python script producer.py

python producer.py

You will get output similar to the following:

Message sent to GlassFlow

Next

See how to consume the transformed data.