Skip to content

Behavioral Pipeline

How events travel from a tap in the app to the user model, and why analytics and serving are kept on separate tracks.

When a visitor acts, that signal needs to reach two very different consumers:

  • Analytics: dashboards, evaluation, research. Slow, batch, exploratory.
  • Serving: the live recommender. Fast, online, must react now.

Sending both through one funnel and letting each take what it needs keeps the app simple (it emits once) while letting the two consumers move at their own speed. The key principle: the analytics sink is never on the critical path of a recommendation. If the dashboard is slow, recommendations don't care.

This is the "path B (online)" model: behavior is captured once, fanned out, and the serving side keeps a warm, ready-to-read user model so a recommendation never waits on a rebuild.

flowchart TB
    app["In-memorial app"] -->|track| rs["RudderStack"]
    rs --> ph["PostHog<br/>(analytics, off hot path)"]
    rs -->|webhook| wh["/api/ingest"]
    wh --> norm["normalize_events()"]
    norm --> buf[("RedisEventBuffer<br/>sorted set, 30d window")]
    buf --> upd["UserModelUpdater.refresh"]
    upd --> ums[("RedisUserModelStore<br/>JSON, ~7d TTL")]
    classDef store fill:#EFEAE0,stroke:#A8895B,color:#423D34;
    class buf,ums store;
  • Transport: RudderStack fans events to PostHog (analytics) and the ai-engine webhook (serving). A Jitsu + ClickHouse stack ships in ai-engine-api/docker-compose.yml.
  • Normalize: normalize_events() maps raw payloads → InteractionEvent, drops unusable ones, sorts by time. normalize_content_id bridges content_1234 → 1234.
  • Buffer: RedisEventBuffer: one sorted set per user keyed by timestamp, auto-pruned to the window.
  • Refresh: UserModelUpdater.refresh rebuilds UserSignals from the buffer and saves to RedisUserModelStore (the rebuild-from-buffer pattern).

Full sequence + rationale: AI Engine → Online serving model.

Latency target

Requirement WP5-NFR-02 expects user-state estimation and availability within 1 second of an interaction, and WP5-NFR-01 expects suggestions within 1 second. The online, rebuild-from-buffer design keeps the read path fast enough to meet these. See Requirements.