Skip to content

Latest commit

 

History

History
101 lines (84 loc) · 3.88 KB

plan.md

File metadata and controls

101 lines (84 loc) · 3.88 KB
Raw

Certainly! I'll create a comprehensive document that summarizes all the relevant information and context for this project. Here's the document:

Segment Event Threshold Monitoring System

Project Overview

We're developing a serverless system on Cloudflare to act as a Segment destination, process incoming events, and monitor for user-defined thresholds. The system will allow users to set complex criteria for monitoring their customer data and receive notifications when these criteria are met.

Background

  • The project originated from a need to find an alternative to tools like Pocus for creating materialized daily views of top moving customers based on Segment events.
  • The system needs to handle composite metrics like number of users in a team, type of service, and monthly cost.
  • It should provide webhook notifications when certain thresholds are met.
  • The solution needs to be flexible and self-service, as existing vertical solutions were found insufficient.

Technical Stack

  • Cloudflare Workers for serverless computing
  • Cloudflare Workers Analytics Engine for data storage and querying
  • Cloudflare Workers KV for storing user-defined thresholds
  • Next.js for the frontend user interface

System Components

1. Segment Destination (Cloudflare Worker)

  • Receives events from Segment
  • Processes and stores events in Workers Analytics Engine
  • Creates composite indexes for efficient querying

2. Threshold Management (Cloudflare Worker)

  • Allows users to create, read, update, and delete threshold configurations
  • Stores configurations in Workers KV

3. Threshold Checking (Scheduled Cloudflare Worker)

  • Runs periodically to check if any thresholds have been crossed
  • Queries Workers Analytics Engine based on user-defined thresholds
  • Sends webhook notifications when thresholds are crossed

4. User Interface (Next.js)

  • Provides a dashboard for users to manage their threshold configurations
  • Allows viewing, creating, and editing of thresholds

Key Features

  1. Composite indexing of event data
  2. User-defined thresholds with complex criteria
  3. Flexible query language for defining thresholds
  4. Real-time or near-real-time event processing
  5. Scalable, serverless architecture

Implementation Details

Composite Index

Events are stored with a composite index combining multiple fields:

const compositeIndex = `${event.userId}:${event.teamSize}:${event.serviceType}:${event.monthlyCost}`;

Threshold Configuration

Thresholds are defined using a JSON structure:

{
  "conditions": [
    { "field": "teamSize", "operator": ">", "value": 10 },
    { "field": "monthlyCost", "operator": ">=", "value": 1000 }
  ],
  "aggregation": {
    "field": "value",
    "function": "sum",
    "timeWindow": "7d"
  },
  "triggerValue": 5000
}

UI Components

  1. ThresholdList: Displays all existing thresholds
  2. ThresholdForm: Allows creation and editing of thresholds

Next Steps

  1. Implement the Segment destination Worker
  2. Develop the threshold management and checking Workers
  3. Create API routes for the Next.js frontend
  4. Build out the user interface components
  5. Implement error handling and validation
  6. Add visualizations for threshold proximity
  7. Test the system with real Segment event data

This document provides a comprehensive overview of the project, its components, and the next steps for implementation. It should allow you to pick up where we left off and continue development of the Segment Event Threshold Monitoring System.

Dummy Data for Chart

Here is some dummy data that can be used for the chart:

{
  "data": [
    { "date": "2023-01-01", "value": 1500 },
    { "date": "2023-01-02", "value": 1800 },
    { "date": "2023-01-03", "value": 2100 },
    { "date": "2023-01-04", "value": 2400 },
    { "date": "2023-01-05", "value": 2700 },
    { "date": "2023-01-06", "value": 3000 },
    { "date": "2023-01-07", "value": 3300 }
  ]
}