GlassFlow CLI

Learn how to use GlassFlow CLI.

The GlassFlow Command Line Interface (CLI) is designed to streamline the interaction with the GlassFlow platform, enabling developers to create an account and efficiently manage and operate their data pipelines directly from the command line.

Getting Started

Quickstart Create your first pipeline using CLI.

Resources

Release Notes

Discussions

Installation

Before you can start using the GlassFlow CLI, you need to install it on your system. The CLI is available for macOS, Linux, and Windows.

Install using Homebrew

Install the GlassFlow CLI using the Homebrew:

brew tap glassflow/tap
brew install glassflow

This installs the GlassFlow command globally so you can run glassflowcommands from any directory.

Install from the release package

Install on Linux

Step 1: For Linux-based systems, run the following command to install the GlassFlow CLI:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/glassflow/cli/master/install.sh)" 

To ensure that the GlassFlow CLI is accessible system-wide on a Linux system, move the GlassFlow executable to a directory that's included in your system's PATH. Run the following command: sudo mv ./glassflow /usr/local/bin/ This operation usessudo typically requires superuser privileges and you need to enter sudo passpord.

Step 2: Verify glassflow installation:

glassflow version

Install on Windows

Step 1: Install the GlassFlow CLI on Windows OS by downloading the latest/desired release version and compatible package from the GitHub release page. For example, glassflow-cli-1.0.10-windows-386.zip

Step 2: Create a folder somewhere in your system, for example: C:\GlassFlow\CLI\bin

Step 3: Unzip the downloaded file and move it into the folder you created.

Place GlassFlow CLI .exe file into a specific folder

Step 4: Add the folder path to the Windows Path Environment System Variables.

GlassFlow CLI set system environment variable on Windows

Step 5: Verify glassflow installation:

glassflow version

Update CLI version

Upgrading your GlassFlow CLI ensures you have the latest features, bug fixes, and performance improvements. Follow these guidelines to upgrade the GlassFlow CLI version.

For macOS (via Homebrew)

Upgrade the GlassFlow CLI package using Homebrew:

brew update && brew upgrade glassflow

Confirm the upgrade by checking the version:

glassflow version

For Linux

Execute the installation script again to fetch and install the latest version:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/glassflow/cli/master/install.sh)"

Confirm the upgrade by checking the version:

glassflow version

For Windows

Navigate to the GlassFlow CLI GitHub release page and download the latest version compatible with Windows.

Update the Executable

• If you previously created a folder for GlassFlow CLI (e.g., C:\GlassFlow\CLI\bin), navigate to this folder.

• Delete the old glassflow.exe file.

• Unzip the newly downloaded file and move the new glassflow.exe into the same folder.

Ensure the folder path where glassflow.exe resides is still in your Windows Path Environment System Variables. If you've not changed it since the initial installation, no action is needed here.

Confirm the upgrade by checking the version:

glassflow version

Sign Up

After installing the CLI, simply open your terminal and run the following command to create an account on GlassFlow:

glassflow signup

Upon executing this command, you'll be prompted to choose your preferred method of signup—either using your Google account or via GitHub authentication.

Login

Run the following command to log in:

glassflow login

Authenticate with the GlassFlow platform using your email or GitHub account credentials. This step is crucial for accessing and managing your pipelines securely.

Authenticate in GlassFlow CLI tool

Getting Help

Open your terminal and type glassflow --help to see a list of available commands and options. This command provides a quick reference to the capabilities of the CLI, including creating, removing, and managing your account, organization, space, and pipelines.

The general form of the CLI usage is:

glassflow command [subcommand] [options]

You can also see available subcommands for a given command by running glassflow command --help. For example:

glassflow pipeline --help

Usage: glassflow pipeline [OPTIONS] COMMAND [arg...]

Manage pipelines

Options:
  -o, --org         Organization ID

Commands:
  list              Get pipelines
  create            Create pipeline
  get               Get pipeline
  delete            Delete pipeline
  update-function   Update function
  logs              Get function logs
  push              Push event
  pull, consume     Pull event
  tokens            Get list of access tokens
  token-generate    Generate new access token
  token-rename      Rename access token
  token-revoke      Revoke access token

Run 'glassflow pipeline COMMAND --help' for more information on a command.

Key Commands

Organizations

Organizations are top-level objects. All other objects like Space and Pipeline always belong under an organization. When a new user gets created, an associated organization is created automatically. All actions that the user performs are then part of that organization.

Actions

Create Organization

Creates a new organization by its name:

glassflow organization create <your_organization_name>

Example:

glassflow organization create myorg

Users can create additional organizations. Currently, you are allowed to have up to 3 organizations.

List Organization

Shows all existing organizations:

glassflow organization list

Set Default Organization

Sets a default organization:

glassflow organization set-default

Space

Space allows you to organize and manage related pipelines and resources together. Space helps in structuring your pipelines and team collaborations more effectively. A user can have multiple spaces. All user permissions to pipelines are managed at the space level.

A default space main is created when the user first signs up and if no other existing spaces. This way users can directly start creating a pipeline after signing up.

Actions

Create Space

Initialize a new "space" within GlassFlow. The operation returns a unique ID for the space for access operations.

Example command to create a new space called myspace:

glassflow space create <your_space_name>

Example:

glassflow space create myspace

Spaces List

Shows all existing spaces:

glassflow space list

Delete Space

First, run the space list command to find a unique ID of the space you would like to delete:

glassflow space list

Removes a space by its ID:

glassflow space delete <space_id>

Pipeline

Pipeline facilitates a range of actions, from creating and deploying pipelines to monitoring their performance and managing their lifecycle. A pipeline always belongs to a space. You can create multiple pipelines within a single space.

If a user wants to invite someone to collaborate on a pipeline, they are added to the space to which the pipeline belongs and by default have access to all the pipelines within that space.

Actions

Create a Pipeline

Initializes a new data pipeline. This command allows you to define your pipeline's structure, space, and transformation function.

glassflow pipeline create <pipeline_name> --space-id=<space_id> --function=<location of transformation function>

Example: Command to create a new pipeline called mypipelineinside the space by its ID with the transform.py transformation function:

glassflow pipeline create mypipeline --space-id={space_id} --function=transform.py

Pipeline names can include uppercase, lowercase letters, numbers, and special characters like underscore "_" and dash "-". Read more about the naming guideline.

Creating a pipeline without a transformation function leads to a pipeline with a default echo function. You can update the function for the pipeline later on by using the Update Function command.

Pipelines List

Displays a list of all pipelines you have deployed in all spaces, providing a quick overview of their names and the spaces they belong to.

Example command to list all pipelines in all spaces:

glassflow pipeline list

Example command to list all pipelines in the given space by the space ID:

glassflow pipeline list --space-id={space_id}

Delete Pipeline

First, run the pipeline list command to find a unique ID of the pipeline you would like to delete:

glassflow pipeline list

Removes the pipeline by its ID:

glassflow pipeline delete {pipeline_id}

Update Function for Pipeline

Updates the transformation function with a new one by replacing the Python reference with another Python file:

glassflow pipeline update-function {pipeline_id} --function=transform.py

Pipeline Logs

Shows the logs of the pipeline by its ID:

glassflow pipeline logs {pipeline_id}

List Pipeline Token

Retrieves all access tokens of a given pipeline by its ID glassflow pipeline tokens {pipeline_id}

Pull Failed Events

GlassFlow backs up all untransformed records and you can consume them by running the following command:

glassflow pipeline consume-failed {pipeline_id}