Create a pipeline

This page explains how to create a new data pipeline in GlassFlow.

GlassFlow pipeline ingests real-time data from data sources, applies a custom transformation function, and sends it to data destinations continuously.

Creating a Pipeline with the WebApp

Use the GlassFlow WebApp to create and manage data pipelines in a low-code UI interface.

1. Log in to GlassFlow WebApp:

   • Navigate to the GlassFlow WebApp and log in to your account.

2. Create a New Space:

   • Go to the “Spaces” page and create a new space named MySpace to organize your pipelines.

3. Create a New Pipeline:

   • Within the new space, go to the “Pipelines” page and click “Create Pipeline.”

   • Provide a name for the pipeline, e.g., MyPipeline.

   • Choose the space you created (e.g., MySpace).

4. Configure a Data Source:

   • Choose the type of data source you wish to connect to (e.g., databases, message queues, Webhook).

   • Provide the necessary connection details and credentials.

     • If select the "SDK" option, this requires you to implement the logic for sending data to the GlassFlow pipeline GlassFlow SDK.

5. Add Transformation Stage (Optional):

   • To transform the incoming data, add a transformation function in GlassFlow.

   • Use the built-in visual editor or write custom transformation code to process the incoming data. For example:

   import json
   
   def handler(data, log):
       log.info("Echo:" + json.dumps(data))
   
       return data

6. Configure a Data Sink:

   • Select the destination for your processed data (e.g., databases, message queues, Webhook).

   • Provide necessary connection details and configurations.

   • If select the "SDK" option, this requires you to implement the logic for consuming data from the GlassFlow pipeline.

7. Confirm Pipeline Creation:

   • Review your pipeline configuration and click “Create Pipeline.”

   • Copy the new Pipeline ID and Access Token for accessing the pipeline from Python SDK or external services.

Creating a Pipeline with the CLI

Use the CLI tool to create and manage pipelines easily.

Prerequisites

• You have a GlassFlow account.

• You installed GlassFlow CLI and logged into your account via the CLI.

Create a New Space

Create a new space with <space_name> where one or more pipelines are contained.

glassflow space create <space_name>

After creating the space successfully, you will get a SpaceID in the terminal.

Create a New Pipeline

Create a new pipeline with <pipeline_name> in the existing space:

glassflow pipeline create <pipeline_name> --space-id=<space_id> --function=<location_of_transformation_function>

Parameters Explained

• <pipeline_name>: This is a name for your pipeline. Choose a name that reflects the purpose of the pipeline for easier management and reference.

• --space=<space_id>: Specify the workspace or "space" where your pipeline will be created. Spaces help organize and isolate your pipelines to manage projects and collaborate with team members.

• --function=<location_of_transformation_function>: Points to the location of the Python script containing your transformation logic. This script must include a handle a function that defines how incoming data is processed.

After successfully creating the pipeline, you will get a PipelineID and Access Token in the terminal.

See how to retrieve the pipeline credentials using GlassFlow CLI or WebApp.

Example

Suppose you want to create a pipeline named "weather-data-processing" in a space called "climate-analytics" and your transformation function is located at ./transformations/convert_temp.py.

The command would look like this:

glassflow pipeline create weather-data-processing --space-id={your_space_id}

Next

See how to publish data to the pipeline.