Real-time clickstream analytics

A practical example of creating a pipeline for analyzing clickstream data and continuously updating real-time dashboards.

Clickstream data contains the information gathered as a user navigates through a web application. Clickstream analytics involves tracking, analyzing, and reporting the web pages visited and user behavior on those pages. This data provides valuable insights into user behavior, such as how they discovered the product or service and their interactions on the website.

Clickstream dashboard with GlassFlow

In this tutorial, we will build a clickstream analytics dashboard using GlassFlow. We will use Google Analytics Data API in Python to collect clickstream data from a website and send them to a GlassFlow pipeline. Our transformation function will analyze the data to calculate additional metrics, and we will use Streamlit and Plotly to visualize the results.

Pipeline components

Clickstream analytics dashboard

Link to the GitHub project repository

Producer

There are two options for data producers:

• Use Python script fake_producer.py with the Faker library to generate mock clickstream data and push it to GlassFlow. You do not need to Set Up Google Analytics 4 API in this case.

• Use the Google Analytics 4 Data API integration example code in ga_producer.py Python script to push real-time report events to GlassFlow.

GlassFlow

GlassFlow is responsible for receiving real-time analytics data from the producer using Python SDK, applying the transformation function, and then making the transformed data available for consumption by the consumer.

Consumer

The dashboard component is built using Streamlit, a powerful tool for creating interactive web applications. This component visualizes the clickstream data by creating various charts and graphs in Plotly.

Setting Up the Pipeline with GlassFlow

You will use the GlassFlow WebApp to create a data processing pipeline.

Prerequisites

To start with this setup, you need a free GlassFlow account.

Sign up for a free account

Step 1. Log in to GlassFlow WebApp

Navigate to the GlassFlow WebApp and log in with your credentials.

Step 2. Create a New Pipeline

Click on "Create New Pipeline" and provide a name. You can name it "Clickstream Analytics".

Step 3. Configure a Data Source

Select "SDK" to configure the pipeline to use Python SDK for ingesting analytics event data from the API.

Step 4. Define the Transformer

To provide meaningful insights to the user based on the received dimensions and metrics from Google Analytics, we apply some computations in the transformation function. Copy and paste the following transformation function into the transformer's built-in editor.

https://github.com/glassflow/glassflow-examples/blob/main/use-cases/clickstream-analytics-dashboard/transform.py

import json


def handler(data, log):
    log.info("Event received: " + json.dumps(data))

    # Example dimensions received from Google Analytics
    country = data.get("country")
    city = data.get("city")
    device_category = data.get("deviceCategory")
    screen_name = data.get("unifiedScreenName")
    screen_page_views = data.get("screenPageViews", 0)
    event_count = int(data.get("eventCount", 0))
    screen_page_views = int(data.get("screenPageViews", 0))
    active_users = int(data.get("activeUsers", 0))

    # Compute engagement score
    engagement_score = (event_count + screen_page_views) / (
        active_users + 1
    )  # Adding 1 to avoid division by zero

    # Device usage insights
    device_usage = {"desktop": 0, "mobile": 0, "tablet": 0}
    device_usage[device_category] += 1

    # Content popularity
    content_popularity = {
        screen_name: {
            "event_count": event_count,
            "screen_page_views": screen_page_views,
        }
    }

    # Geographic distribution
    geographic_distribution = {country: {"city": city, "active_users": active_users}}

    # Add computed insights to data
    data.update(
        {
            "engagement_score": engagement_score,
            "device_usage": device_usage,
            "content_popularity": content_popularity,
            "geographic_distribution": geographic_distribution,
        }
    )

    log.info("Transformed data with insights: " + json.dumps(data))
    return data

Note that the handler function is mandatory to implement in your code. Without it, the transformation function will not be successful.

The sample transformation function enriches input event data with the following:

• Engagement Score: Calculates an engagement score based on event count, screen page views, and active users.

• Device Usage Insights: Analyzes the proportion of different device categories.

• Content Popularity: Tracks the popularity of different screens/pages.

• Geographic Distribution: Provides insights on user distribution based on geographic location.

Step 5. Configure a Data Sink

Select "SDK" to configure the pipeline to use Python SDK for sending data to the fleet management file. In a real-world project, you send data to dashboards and notification systems.

Step 6. Confirm the Pipeline

Confirm the pipeline settings in the final step and click "Create Pipeline".

Step 7. Copy the Pipeline Credentials

Once the pipeline is created, copy its credentials such as Pipeline ID and Access Token.

Setting Up Google Analytics 4 API

Prerequisites

You have a Google Analytics (GA) account if you use the GA as a data producer.

Google Analytics 4 (or GA4) has an API that provides access to page views, traffic sources, and other data points. With this API, you can build custom dashboards, automate reporting, and integrate with other applications. We focus only on accessing and exporting data to GlasFlow using Python. You can find more comprehensive information about how to set up the Google Cloud Project (GCP), enable the API, and configure authentication in the API quickstart, or follow this step-by-step guide.

1. Enable the Google Analytics Data API for a new project or select an existing project.

2. Go to https://console.cloud.google.com/apis/credentials. Click "Create credentials" and choose a "Service Account" option. Name the service user and click through the next steps.

3. Once more go to https://console.cloud.google.com/apis/credentials and click on your newly created user (under Service Accounts) Go to "Keys", click "Add key" -> "Create new key" -> "JSON". A JSON file will be saved to your computer.

4. Rename this JSON file to credentials.json and put it under use-cases/clickstream-analytics-dashboard. Then set the path to this file to the environment variable GOOGLE_APPLICATION_CREDENTIALS:

export GOOGLE_APPLICATION_CREDENTIALS=credentials.json

5. Add a service account to the Google Analytics property. Using a text editor or VS code, open the credentials.json file downloaded in the previous step and search for client_email field to obtain the service account email address that looks similar to:

[email protected]

Use this email address to add a user to the Google Analytics property you want to access via the Google Analytics Data API v1. For this tutorial, only Viewer permissions are needed.

6. Copy the Google Analytics property ID you are discovering and save it to variable value for GA_PROPERTY_ID in a .env file in the project directory.

Design Streamlit dashboard

The Streamlit dashboard code in consumer.py the script will visualize the output from the GlassFlow transformation, which includes additional insights such as engagement score, device usage, content popularity, and geographic distribution.

https://github.com/glassflow/glassflow-examples/blob/main/use-cases/clickstream-analytics-dashboard/consumer.py

import glassflow
import sys
from dotenv import load_dotenv
import streamlit as st
import plotly.express as px
import pandas as pd
import os
import time

load_dotenv()

pipeline_id = os.environ["PIPELINE_ID"]
token = os.environ["PIPELINE_ACCESS_TOKEN"]
client = glassflow.GlassFlowClient()
pipeline_client = client.pipeline_client(
    pipeline_id=pipeline_id, pipeline_access_token=token
)

st.title("Clickstream Analytics Dashboard")

placeholder = st.empty()


def update_dashboard(data):
    if data:
        df = pd.DataFrame(data)

        # User Engagement by Web Page and Location
        fig_engagement = px.bar(
            df,
            x="unifiedScreenName",
            y="engagement_score",
            color="city",
            title="User Engagement by Web Page and Location",
        )

        # Device Usage
        device_usage_df = (
            pd.json_normalize(df["device_usage"].tolist()).sum().reset_index()
        )
        device_usage_df.columns = ["Device", "Count"]
        fig_device = px.pie(
            device_usage_df,
            names="Device",
            values="Count",
            title="Device Usage Distribution",
        )

        # Content Popularity
        content_popularity_list = []
        for cp in df["content_popularity"]:
            for page, metrics in cp.items():
                content_popularity_list.append(
                    {
                        "page": page,
                        "event_count": metrics["event_count"],
                        "screen_page_views": metrics["screen_page_views"],
                    }
                )
        content_popularity_df = pd.DataFrame(content_popularity_list)
        fig_content = px.bar(
            content_popularity_df,
            x="page",
            y="event_count",
            title="Content Popularity (Event Count)",
        )

        # Geographic Distribution
        geographic_distribution_list = []
        for geo in df["geographic_distribution"]:
            for country, metrics in geo.items():
                geographic_distribution_list.append(
                    {
                        "country": country,
                        "city": metrics["city"],
                        "active_users": metrics["active_users"],
                    }
                )
        geo_dist_df = pd.DataFrame(geographic_distribution_list)
        fig_geo = px.bar(
            geo_dist_df,
            x="country",
            y="active_users",
            color="city",
            title="Geographic Distribution of Active Users",
        )

        # Display the figures in a 2x2 grid
        col1, col2 = st.columns(2)
        with col1:
            st.plotly_chart(fig_engagement)
            st.plotly_chart(fig_device)
        with col2:
            st.plotly_chart(fig_content)
            st.plotly_chart(fig_geo)


data = []

while True:
    with placeholder.container():
        try:
            # consume transformed data from the pipeline
            res = pipeline_client.consume()
            if res.status_code == 200:
                # get the transformed data as json
                event_data = res.body.event
                print("Data consumed successfully")
                print(event_data)

                data.append(event_data)
                print(data)

                # update the dashboard
                update_dashboard(data)
        except KeyboardInterrupt:
            print("exiting")
            sys.exit(0)
        time.sleep(2)

The dashboard is updated in real-time with data being continuously consumed from the GlassFlow pipeline.

Send and consume data from the pipeline

Prerequisites

To complete this part you'll need the following:

• Python is installed on your machine.

• Download and Install Pip to manage project packages.

Installation

1. Clone the glassflow-examples repository to your local machine:

   git clone https://github.com/glassflow/glassflow-examples.git

2. Navigate to the project directory:

   cd use-cases/clickstream-analytics-dashboard

3. Create a new virtual environment:

   python -m venv .venv && source .venv/bin/activate

4. Install the required dependencies:

   pip install -r requirements.txt

Create an environment configuration file

Add a .env file in the project directory with the following configuration variables and their values:

GA_PROPERTY_ID=your_ga_property_id # You do not need it if you generating mock events. 
PIPELINE_ID=your_pipeline_id
PIPELINE_ACCESS_TOKEN=your_pipeline_access_token

Replace your_pipeline_idand your_pipeline_access_token with appropriate values obtained from your GlassFlow account.

Run data producer

Run the ga_producer.py or fake_producer.py script to start publishing data:

python ga_producer.py

Run the dashboard

Use Streamlit command to run the dashboard:

streamlit run consumer.py

You see the output with several dashboards updating in real-time:

Sample GlassFlow docs analytics real-time dashboard

You learned how to integrate real-time analytics data from Google Analytics into GlassFlow for further processing and visualization. Analytics data can be also stored in a database like ClickHouse for future use.