Consume data

This page explains how to consume data from GlassFlow pipelines.

Ways to consume data from GlassFlow

Consuming data is retrieving transformed data from a data pipeline in GlassFlow. You use built in integrations to consume data or write code in Python to implement a new custom integration.

Consuming data using integrations

Visit Integrations page for more information.

Consuming data via Python SDK

The Python SDK provides a programmatic way to interact with GlassFlow pipelines to produce or consume data continuously. Using the SDK you create a custom connector for any data sink in Python.

Prerequisites

• You created a Pipeline.

• You have the pipeline credentials such as PIPELINE_ID and PIPELINE_ACCESS_TOKEN.

Install GlassFlow Python SDK

Install a GlassFlow SDK using the pipcommand in a terminal.

pip install glassflow 

Set environment variables

Set environment variables with your actual GlassFlow pipeline credentials such as PIPELINE_ID and PIPELINE_ACCESS_TOKEN:

export PIPELINE_ID=your_pipeline_id
export PIPELINE_ACCESS_TOKEN=your_pipeline_access_token

Consume Transformed Data from the pipeline

Create a new Python script file called consumer.py and insert the code below.

import glassflow

pipeline_client = glassflow.GlassFlowClient().pipeline_client()

while True:
    try:
        # Consumes a new event from the pipeline
        response = pipeline_client.consume()

        if response.status_code == 200:
            data = response.json()
            print("Consumed Data:", data)
    except KeyboardInterrupt:
        print("exiting")
        break
    except Exception as e:
        print("Error:", e)
        break

This script continuously checks for newly transformed data from the pipeline and consumes it as needed. The main GlassFlow SDK usage revolves around creating a GlassFlow pipeline client instance to interact with the GlassFlow platform and consume data from the data pipeline.

1. Initializes a GlassFlow client to establish a connection with the GlassFlow platform for a specific pipeline. The SDK automatically reads the needed parameters (pipeline_id and pipeline_access_token) from the environment variables. Alternatively, you can also pass them as parameters when creating the client:

Without the pipeline credentials params:

pipeline_client = glassflow.GlassFlowClient().pipeline_client()

With the pipeline credentials params:

pipeline_client = glassflow.GlassFlowClient().pipeline_client(
    pipeline_id=pipeline_id, 
    pipeline_access_token=token
)

2. Consumes the transformed data from the pipeline. It returns a response object containing the consumed event data.

response = pipeline_client.consume()

You receive a ConsumeEventResponse object in response. This response object contains:

• status_code: The status_code attribute holds the HTTP status code of the response.

• json(): A helper function to get the transformed event as a JSON object.

Refer to Python SDK documentation for more details.

Run the script

python consumer.py

You will get output similar to the following:

Consumed Data: {...}

Next

See tutorials for complex scenarios.