Sprint - Dec 23 - Jan 3

[dmarticus]

Global Sprint Planning

3 things that might take us down

Urgent incident follow-ups

https://github.com/orgs/PostHog/projects/103/views/2

Team sprint planning

For your team sprint planning copy this template into a comment below for each team.

# Team ___

**Support hero:** ___

## Retro

<!-- Grab the high and low priority items from last time and add whether that item was completed or not -->

- 

## Hang over items from previous sprint

<!-- For each item, decide to re-prioritise (and add below) or deprioritise -->

- Item 1. prioritised/deprioritise

## OKR

1. OKR, status (red/yellow/green) and action points if yellow/red


### High priority

-

### Low priority / side quests

-

[jurajmajerik]

Team Experiments 🧪

Retro

• 🟢 Continue with onboarding tasks and get familiar with the codebase. @andehen

  • Read some papers on Bayesian statistics
  • Set up Growthbook locally to compare our experiment results with theirs
  • Django management script to generate Experiment test data
  • Several UX improvements for experiments
  • Clean ups of the dead code after HogQL migration

• 🟢 Test the Data Warehouse integration with 1-2 customers (Freepik, potentially ElevenLabs). @danielbachhuber

  • First experiment with a priority customer is live.

• 🟡 Finish support for multiple experiment metrics. @jurajmajerik

  • On track to release this before the end of the sprint.
  • Lots of UI complexity but looks promising.

• 🟡 Plug improved statistical calculation for Trends + add docs @danielbachhuber, @andehen, @jurajmajerik

  • Cleaned up/refactored our statistical methods and got final sign off from our contractor.
  • 2/3 methods are merged but some follow ups are still needed.

Sprint goals

Support hero: @jurajmajerik

All of us are only working for 2-3 days next sprint.

@danielbachhuber

• Continue working on the stats methods.

@jurajmajerik

• Get started on the reusable metrics UI, have a minimal version rolled out for our priority customer.

@andehen

• Finish hogql migration cleanup
• Set up some AA tests in prod
• Simulate several experiments to get more experience with our product

[daibhin]

Team Flags

Support

• 12-16 to 12-23 @dmarticus
• Let's talk about how to do support for the holidays

Retro

Things we got done

• testing/productionizing new /flags service @dmarticus
  • fixing P0 bugs along /flags hotpath
  • functional testing via django-compat integration tests
• Figure out how to cut off the flags bit of /decide (and, in future /flags) when folks go over – right now we don't stop serving flags when overage happens @dmarticus
  • going to do this in the new /flags endpoint

Things we didn't get done and why

• "real" testing via mirroring traffic to new service
  • still cleaning up results of local/integration tests, not ready for prod traffic just yet
• Continue to look at growth metrics and see what other properties we can add to better understand our growth @dmarticus
  • didn't prioritize
• Feature mgmt progress, support work took over majority of time

---

Sprint

High priority

• testing/productionizing new /flags service @dmarticus
  • integrate error spans with new error tracking service
  • implement holdout group targeting
  • add more metrics for production testing
  • "real" testing via mirroring traffic to new service
• Feature mgmt crud working end to end, get feedback @havenbarnes

Side Quests

• enforce singleton requirements for the python SDK @dmarticus
• Continue to make sentry more actionable and useful @havenbarnes
• Continue to look at growth metrics and see what other properties we can add to better understand our growth @dmarticus
• Fix team_id label on feature flags grafana dashboard @havenbarnes
• (Look into) adding /decide trx id to $feature_flag_called event @havenbarnes

Things to chew on

• Can we know when people are starting to make new features?

OKRs

1. 🟡 Ship new feature flags in production and accomplish the 10x scale + 2x speed boost performance targets (ideally decreases infra spend by N% as well, will try to figure out a more specific target going forward)
2. 🟢 Maintain current /decide service and make optimizations.

[benjackwhite]

Team Christmas Don't Panic (CDP)

• Support: @benjackwhite

Rough time off: ben 50%, Marius 100%, Marcus 50%, Olly mixed

Retro

NOTE: We swapped a few things around mid sprint to support some big customer imports

• 🟡 @oliverb123 - making strong progress on batch import work so that we can handle these kind of requests
• 🟢 @benjackwhite gradually roll out RemoteConfigs to everyone (providing the code for site-functions)
  • Only final bits needed just want to safely gradually roll out
• 🟡 @benjackwhite Setup internal event source destinations
• 🟢 @mariusandra finish Mappings work with focus on TikTok Ads
  • Mappings are out and working just blocked by RemoteConfig work
• 🟢 @MarconLP build out and test TikTok Ads / Snapchat client side integration
  • Still waiting on google ads to verify for standard access

OKRs

• 🟢 Widespread usage of the new CDP
• 🔴 Messaging V1
• 🟡 Ingestion is rock solid

High priority

• Keep things stable and running!
• Finish the internal event source destinations @benjackwhite
• Rollout RemoteConfig when it feels very safe (or just get more metrics to rollout post-break) @benjackwhite
• Tiktok / snapchat client side destinations - running on an actual website of some kind! @MarconLP (could chat with marketing team to see if there is a good thing we could test this with)
• Batch imports @oliverb123 - getting at least one of the target customers imported via this.
• Be ready to onboard 2 new people!

Stretch goal

• Server side mappings

[daibhin]

Team Error Tracking

@daibhin off (most days)
@neilkakkar 1-2 days

Retro

🟢 Launch closed beta @daibhin
30 teams sent at least 1 exception event in the last 24 hours
Updated docs with info on exception sampling & custom grouping
Outstanding: Billing repo still can't be upgraded so we're looking into it. Need to find some overlap with Zach

🟢 Cymbal improvements @oliverb123
Throughput not a problem since adding Kafka write batching. Also solves health check issues
Memory leak solved (cache invalidation is hard when you don't invalidate the cache)
Alerting & monitoring is still something we need to create a dashboard for

🟢 Alerting @daibhin
Decided on approach with Ben & splitting up the work. Should have issue action (created, reopened) alerts by EOW

🟢 Improved UI for issues page should be ready by EOW @daibhin

OKR

✅ Human readable stack traces for Javascript
✅ Ensure scalability of query time issue grouping
🟢 Alerting via CDP
🟡 Launch

High priority

• Merge Q1 goals PR @daibhin
• Keep the (Christmas) lights on and deal with any urgent beta issues
• Optimize loading of issues and events (e.g. materialize columns) @neilkakkar
• Threshold based alerting @neilkakkar

Low priority / Side quests

• Cleanup V2 table

[pauldambra]

Team (R)One (E)Door (P)Closes (L)and (A)nother (Y)Opens

xmas sprint

Support hero: @pauldambra
Manoel is off, Paul is off, Yiannis is flying solo/yolo

Retro

## what we shipped

• 🍾 big golden bottle of champagne in london to celebrate the final platform being released
  • we completely forgot to drink champagne 🫠 how?!
• size of mobile replay @marandaneto @ioannisj
  • did diff detection
  • ready to switch to webp in january
• ✅ manual API for starting and stopping all 4 platforms @marandaneto @ioannisj
  • pending but should be done (particularly since picking up some surveys work)
  • needs reset, resetSessionId, startRecording, stopRecording
  • needs full snapshot on resume in mobile
• 🔄 Capture network headers and body (assuming new instrumentation is in place by then) Network bodies for network logs posthog-ios#159 @ioannisj
• ✅ long sessions on mobile - Session Id didn't rotate after 23 hours posthog-ios#227. @ioannisj
  • should be in by Friday
• ✅ handover from @Phanatic

Replay

• replay 4000 UI
  • ✅ improve overview tab using sessions data
• 🔄 a brief diversion before querying network data - wrap websockets as well as fetch @pauldambra
  • wrapping websockets turns out to be a pain

Low priority / side quests

OKRs 🟡

1. OKR, status (red/yellow/green) and action points if yellow/red

🟢 📱Goal 1: People think of PostHog as a mobile solution
🟡 📹 Goal 2: Replay 4000
- this is now so excellent it's Replay 4000
- #26098
🟡 🎤 Goal3: Ship the things users keep asking for

• queryable network data
• start replay on trigger e.g. event

High priority

• brandy butter
• brandy cream

Mobile

• 🔄 Capture network headers and body (assuming new instrumentation is in place by then) Network bodies for network logs posthog-ios#159 @ioannisj

Low priority / side quests

• (can we|do we) measure watch time and buffer time? @pauldambra
• go through support and find people who wanted block list and event trigger @pauldambra
• 🔄 get people using iOS autocapture @ioannisj

[MichaelKutsch-ph]

Team (I)cicles (N)utmeg (F)ir (R)eindeer (A)angels

Christmas, Christmas, Christmas

Support: @bciaraldi @danielxnj @MichaelKutsch-ph checking some times a day to support
Off: basically everyone

(rest below is WIP)

Retro

## Retro

• 🟢 Frank is back! 🥳
• 🔴 install nOps agent to detect x-az traffic issues @danielxnj
  • not sure if we're following up here as we are likely not going with nOps
• 🟡 dive into details and decide about AWS Pricing: ongoing, crunching numbers, planning next steps
• 🔴 Lock in WS @timgl
  • follow up there and plan move to WS
• 🟡 Continue w/ internal cluster into the same state as our prod clusters (monitoring stack, SSO access etc.) - monitoring WIP 🟡 - prioritized making SIEM work again, SSO is in place 🟢
• 🟡 ArgoCD
  • cleaning up things
  • SSO yay 🥳
  • move more apps (e.g. workers)
  • this will take some time to move everything and be stable incl. alerting
  • we might get rid of auto-sync and build around it to make it our own and work for our WoW
• 🟢 exhaustion of IPs on prod
  • fixed issues with networking plugins
  • agreed to fix the networking long term (see mega issue)
  • next steps include adding more subnets

OKR

1. 🏃 Next level ArgoCD - Continuous Delivery
2. ☁️ 🕸️ Scalable networking
3. 💰 Mr. Bezos, bring down those costs!
4. 🎮 A stable pre-prod environment

High priority

• finish observability for internal-cluster @MichaelKutsch-ph
• iterate on cloud spend topic @MichaelKutsch-ph
• argoCD alerts @bciaraldi
• keeping the machines running during the holiday season
• we won't be doing big things as we're mostly off anyways and not breaking something because so many peeps are off as well

Low priority / side quests

[marandaneto]

Team (S)neaky (U)ndercover (R)eindeer (V)iciously (E)at (Y)our (S)tocking! 🎄🦌🎁 or just Team Farewell

@Phanatic offboarding December 23rd
@marandaneto PTO the whole sprint
@ioannisj a few days PTO
Lucas joining the team
Lucas PTO?

Retro

• ⌛ Fixing bugs reported by Annika and a few customers that we're getting more survey responses than allowed when using Adaptive Sampling.

• Issue filed here : Surveys: Adaptive survey collection limits aren't granular enough #27115

• ✅ Fixing bugs reported by customers of NPS recurring surveys who are confused about when the survey is shown or hidden.
  Customer toggled NPS recurring behavior on-off resulting in them seeing the survey multiple times to the same user.
• ✅ Handing off no-code experiments to the Experiments team.
• ✅ Handing off Surveys to @marandaneto and @ioannisj
• ✅ Customer support
  • Phani is OOO, Manoel and Dylan are on it
• 🚫 Q1 Planning
  • Early January

Sprint goals

• Customer support @dmarticus and @ioannisj
• Onboarding Lucas @dmarticus

[robbie-c]

Team (W)inter (E)njoyment (B)reak

Support hero: @rafaeelaudibert (1st week) & @robbie-c (2nd week)

@rafaeelaudibert: Working 26th & 27th Dec
@robbie-c: Working 2nd & 3rd Jan

Retro

Robbie
🟡 Finish cookieless prototype

Rafael
🟢 Conversion rate on all web analytics insights
🟢 Add more options for previous period comparison (use product analytics UI)
🟢 Comparison to the previous period on all queries
🟢 Get comfortable doing web analytics support
🟢 Some nextjs improvements

OKR

🟠 Continue to improve performance
🟢 Build a small amount of more advanced marketing features, to measure interest
🟡 Cookieless tracking prototype
🟠 Improve the NextJS experience
🔴 More page-level insights
🟢 Improve the documentation, and link to it more from the app

High priority

Rafael

• Christmas support tickets
• Core web vitals product research

Robbie

• Finish cookieless prototype
• New year support tickets

[fuziontech]

Team (C)hristmas (H)aus

OKR Q2 2024

Objective

Goals for Q4:

• Improve elasticity and flexibility of our data store by putting all our data in Iceberg
  • Work with Altinity effectively to ship read path for Iceberg on ClickHouse - Brett Hoerner
  • Setup infrastructure to ship all of our data to Iceberg on S3 - James Greenhill
  • Shipping Query logs to Iceberg - James Greenhill
• Continue improving CH operational expertise
  • Upgrade to a later version of Clickhouse - James Greenhill
  • Capacity planning - James Greenhill
  • Automation - Daniel Escribano
  • Put some of the basic mitigation operations in runbooks - Daniel Escribano
• Schema management
  • Tool for schema migration (coordinator schemas) - Daniel Escribano
  • Tool for long running mutations - Daniel Escribano
• Continued investment in performance
  • Tooling for other teams to understand which queries are slow and why - Ted Kaemming
  • Investigate variability of queries - Ted Kaemming
  • Per-team limits on queries/query complexity (needs product work) - Ted Kaemming

Board

https://github.com/orgs/PostHog/projects/85/views/2

Retro

🔥 Incidents 🔥

More than normal this cycle

• #inc-2024-12-11-ingestion-delays
• #inc-2024-12-11-clickhouse-node-down-us
• #inc-2024-12-10-bad-replay-migration
• #inc-2024-12-06-us-clickhouse-slowness

Last Sprint's Goals

Cost investigation

• Review increasing cost on all AWS @fuziontech 🟡
  • Cleared up a lot of EBS, but still have two clusters (will be one by Friday)
    Validate and move over to new cluster quickly
• Validate cluster data parity between old and new clusters @fuziontech 🔴
  • Clusters diverged, then an outage prevented us from reconciling. 🔥
• Fix deletes @fuziontech 🔴
• Benchmark to new 8 shard lower cost cluster w/ coordinator nodes @fuziontech 🟡
  • This one was actually done and early indications look good.
    Mat Col optimizations
• Investigate how to speed up index dropping for Mat Cols @tkaemming 🟢
  • Increased thread pool available to mutations 😎
• Yeet unused mat cols @tkaemming 🟢
  Below the fold
• Investigate what needs to be done for queue metering / QoS @fuziontech 🟢
  • https://posthog.slack.com/docs/TSS5W8YQZ/F0855S0TBPE
• Get query logs into iceberg. @fuziontech 🔴
• (low priority) Expose Iceberg table for testing @fuziontech 🔴
  • Current version this will be quite slow
  • New 24.8 version will be quite fast

This Sprint's Goals

Holiday plans

@fuziontech - Vancouver 24->26
@tkaemming - OOO 23 -> 27 travel 30
@Daesgar - OOO 25 & 1 & 6
@orian - OOO 23 -> 27

TL;DR

Not much - KTLO ideally

• Benchmark new cluster @fuziontech
• Shut down new cluster @fuziontech
• Get into a place where we can quickly restart new cluster @fuziontech
• chproxy into dev stack @orian
• Look into weekend event delete job @orian
• Look into iceberg for query log @orian
• Squash job @tkaemming

[EDsCODE]

Team Data Storehouse

OKR Q4 2024

Objective

Query 3000

• Key Results:
  • Autocomplete
  • Increase general BI experience/product BI meta#157
  • Declutter the data warehouse UI and make the features intuitive to find

Data Modeling MVP

• Key Results:
  • Infrastructure decided and implemented
  • Integrating external data with feature flags
  • External data everywhere in insights/persons/cohorts
  • Get billing team to use modeling in posthog for their invoices_with_annual table

Retro

Tom off for most of sprint

• Draft queries + reorg sidebar (Epic: Editor & Modeling 3000 #26193) @EDsCODE
• Materialization Tab (Epic: Editor & Modeling 3000 #26193) @EDsCODE
  • refactored most of new editor logic
• Permanent search bar
• Release data import to everyone @Gilbert09
  • Side quest to work through bigquery integration for customer. Import being testing tomorrow

Priority

Both off most of first week

• Draft queries + reorg sidebar (Epic: Editor & Modeling 3000 Epic: Editor & Modeling 3000 #26193) @EDsCODE
• Data import deployment and testing @Gilbert09

[aspicer]

Team Product Analytics

Q4 2024 objectives

Legacy 0x - Thomas owning

• All insights are query-based
• rm -rf posthog/queries/
• Insight query is stored in the URL for instant sharing of unsaved insights

⠀First-class environments - Michael owning

• Environments are rolled out to everyone

⠀Threshold-based alerts - Anirudh owning

• All users get to set up alerts on PostHog metrics
• Alerts are launched with Marketing, and positioned into paid plans
• We're using alerts ourselves for key trends and funnels

⠀Flexible funnels - Sandy owning

• UDF funnels rolled out to everyone, unlocking arbitrary complexity
• Allowing parts of funnels to be unordered. No more separation between regular, unordered, and strict funnels
• Optional funnel steps

⠀10x onboarding and ease of discovering features - Anirudh and Sandy owning

• We run an exercise in implementing PostHog in a hobby project. What stands out as painful or unclear?

⠀User delight 2x - support rotation owning

• Support is 1 hero + 1 sidekick
• Every week support folks ship those small requested features (see [list of requests so far](https://docs.google.com/spreadsheets/d/1-lXjfzo6313Ai2vMf6smmoA0zghitBQTRD4lPgZ-XqA/edit?gid=0#gid=0))
• Goal: 90% of tickets fulfill the SLA

⠀B2B analytics research - Anna owning

• Understand the market and user needs better in order to improve our current Group analytics product next quarter

🟢🟡🔴

Retro

Support hero: @aspicer @thmsobrmlr
SDK Support hero: @anirudhpillai @aspicer

• 🟢 Alerts [Anirudh]
  • Scaling with the rollout
    • Improving execution system for running alerts
    • Bugfixes and deploy monitoring
• 🟡 Carryover: Cached Dashboards, further rollout [Anirudh]
  • Thinking about API concurrency and scaling
• 🟢 Peter Onboarding [Sandy]
• 🟡 Carryover: Cohort migration production monitoring and bug fixing [Sandy]
• 🟢 Preparing for Planning [Anna]
  • Collect build better notes and user interviews for Q1 Planning
• 🔴 Migrate api endpoints to use hogql queries [Thomas]

Sprint Plan

Support hero: @anirudh @aspicer

• Monitor and fix issues with hogql cohort calculation [Sandy]
  • Stretch goal: roll it out as primary cohort calculation logic
• Dashboard creation / edit experience improvements [Peter]
• Rollover: Migrate api endpoints to use hogql queries [Thomas]

[zlwaterfield]

Team (G)rinch (R)aided (O)ur (W)hole (T)urkey (H)elplessly

Retro

Retro items

• @zlwaterfield
  • Support hero week 1
  • Misc fixes from RBAC middleware changes
  • RBAC UI behind a feature flag
  • Finish the 2FA improvements
  • Block usage reporting when on a trial (request from sales)
  • Finish billing postgres db improvements - pgbouncer and read replica -> infra is working on this
• @surbhi-posthog
  • Support her week 2
  • custom sql analysis on ICP data
  • load icp properties into Posthog?
  • Add events for Team's features in the backend
• @patricio-posthog
  • Keep monitoring the new forecasting, check that everything runs smoothly
  • Update /billing endpoint to include the new forecasting for customers
  • API query pricing
  • Start working on the new pricing for mobile RFC

Q4 Goals

✅=finished 🟡=in progress 🔴=won't finish ⚪=not started

1. 🟡 Get to a 4/10 in Enterprise platform support (Zach)
   • 🟡 Release Enterprise addon and iterate with sales to find a pricing model that makes sense
   • 🟡 Build RBAC
2. 🟡 Make revenue & forecasting more reliable (Patricio)
   • ✅ Rejig how we do sync_invoices so we don’t see big drops - respond to webhooks instead of large syncs
   • 🔴 Evaluate temporal vs celery for usage reports and billing tasks so we can adhere to rate limits better
   • 🔴 Side hustle, if time permits: Rejig how anonymous / identified events are billed, as the current implementation is confusing.
3. 🟡 Activation: reverse trials, analytics onboarding flow, understand CC activation (Raquel)
   • ✅ Dig into the data, watch replays to figure out why CC activation rate has been falling
   • 🔴 Add reverse trial to onboarding
   • ✅ Add reverse trial to billing page for teams

This sprint

Time off: lots of time off with xmas and such

• @zlwaterfield
  • RBAC focused - client side UI, documentation, edge cases and misc features, migration prep, getting ready to hand off to product teams, etc.
  • Customer.io emails in PostHog - RFC open
  • 2FA emails
• @surbhi-posthog
  • ICP data migration from salesforce to Posthog
  • New ICP Growth Dashboard. This is just the beginning, and I need to keep working on validation data quality and definitions
  • Helped with shipping 2fa PR
  • Product analytics hypothesis to help low ICP users. I have a PR out, and looking for feedback
  • Update docs about data migration learnings
  • Build final 2 events for team features.
• @patricio-posthog
  • PR for forecasting in the frontend is ready, once merged I'll monitor it https://github.com/PostHog/billing/pull/1046
  • Create new pricing on Stripe for Mobile Pricing
  • Update trust scores when customer updates their limits
  • Tool in Billing admin to assign credits to list of emails (ref: ElevenLabs credit program)