Monitor pipeline

This page explains how to monitor pipelines in GlassFlow.

GlassFlow provides logging capabilities to monitor and debug your pipelines. To check these logs, you use the GlassFlow WebApp or GlassFlow CLI.

Using Logs in the Transformation Function

You can add logging to your transformation functions to capture important information about the processed events. To output runtime logs from your transformation function code, you use the built-in standard logging module passed as a log parameter to the function.

Here's a basic example of how to add logging in a transformation function:

import json

def handler(data, log):
    log.info("Event received: " + json.dumps(data))

    # Your transformation logic here
    transformed_data = perform_transformation(data)

    log.info("Transformed data: " + json.dumps(transformed_data))
    return transformed_data

def perform_transformation(data):
    # Example transformation logic
    data["processed"] = True
    return data

In this example, the log.info statements are used to log information about the received event and the transformed data. You can use different logging levels such as log.debug, log.warn, and log.error to capture various types of information.

Viewing logs with GlassFlow WebApp

Step 1: Navigate to Your Pipeline

1. Log in to the GlassFlow WebApp.

2. Navigate to the "Pipelines" section and select the pipeline you want to monitor.

Step 2: Access the Logs

1. Click on the "Logs" tab on the pipeline details page.

2. You will see a list of log entries generated by your transformation functions. Each log entry includes the timestamp, log level, and event payload.

Step 3: Filter and Download Logs

• Filter logs based on log levels.

GlassFlow pipeline see logs

• You can also adjust the time range to view logs from a specific period like the last hour, last 6 hours, etc.

• Also, there is an option to download logs.

Viewing logs with GlassFlow CLI

Prerequisite

Make sure you have the GlassFlow CLI installed. If not, you can install it by following the installation instructions.

Step 1: Log In to GlassFlow CLI

Log in to the GlassFlow CLI using your credentials:

glassflow login

Step 2: Access Pipeline Logs

Use the following command to access logs for a specific pipeline:

glassflow pipeline logs <your_pipeline_id>

Replace <your_pipeline_id> with the ID of the pipeline you want to monitor. This command will display the logs in the terminal, allowing you to filter and search through them as needed.

Step 3: Filter Logs by Level

You can filter logs by severity, start and end time options:

glassflow pipeline logs <your_pipeline_id> --severity=100

This command will display only the log entries with the level info. You can use other levels such as debug, warn, and error.