Bandit / online learning¶

ranking/bandit.py is an optional linear contextual bandit (LinUCB-style) that learns the fusion weights the static ranking sets by hand. It is pure Python (no numpy: d is tiny, about 6, so a hand-rolled matrix inverse keeps it dependency-free) and slots in behind the same scorers: the context is their outputs, the reward is realized engagement.

context  x = [semantic, tag, recency, aversion, geo]   (per candidate)
reward   r = realized engagement strength of the view it produced (0 if shown, not opened)
model    E[r | x] = θ·x          serve score, + UCB bonus α·√(xᵀA⁻¹x) for exploration
update   A += x xᵀ ;  b += r x ;  θ = A⁻¹ b
prior    A₀ = ridge·I ,  b₀ = ridge·w_static   ⇒   θ₀ = w_static

The prior is the safety property¶

At θ₀ the bandit ranks identically to the static weighted fusion, because the static FusionWeights are the ridge prior (θ₀ = w_static, verified in tests/test_bandit.py). So ranking_mode="bandit" at the prior is byte-for-byte the static ranking, then learns away from it as data accrues. Enabling the bandit cannot regress day-one behaviour.

static vs bandit¶

`ranking_mode`	Behaviour
`static` (default)	weighted fusion; still logs feature vectors, so a bandit can be fit from traffic served before it is ever turned on
`bandit`	scores by `θ·x` (+ UCB bonus when `bandit_explore`); loads `BANDIT_STATE_PATH` at startup, or serves at the prior if absent

Because features are logged in both modes, you can collect data statically and switch on the learned policy only once it beats the prior (see evaluation).

How the loop closes¶

flowchart LR
    serve["/api/recommend"] -->|features x + request_id| imp[("served log + impression store")]
    serve -->|request_id| app["app"]
    app -->|CONTENT_VIEW_ENDED<br/>echoes request_id| ingest["/api/ingest"]
    ingest -->|reward r| join["join (request_id, content_id)"]
    imp --> join
    join --> theta["θ update"]

Serve. Every /api/recommend logs each item's feature vector x (to served/date=*/ and the impression store) and returns a request_id.
Reward. The app echoes that request_id (in properties.details.request_id) on the resulting CONTENT_VIEW_ENDED. Those events land in date=*/ with the reward signal (dwell, end_reason).
Train. Join served by events on (request_id, content_id), compute the reward per impression (engagement strength; shown-but-not-opened then 0), and fit θ.

Two ways to learn¶

Offline trainer (`bandit/train.py`)¶

Joins the two durable Parquet logs, forms (x, r) samples, and applies one LinUCB update per sample starting from the prior. It always starts fresh from the prior, so each run is a full fit over all accumulated data (not an incremental drift). Re-run it on a schedule.

# 1. collect traffic with EVENT_LOG_DIR set (static mode is fine)
EVENT_LOG_DIR=./data/eventlog  RECSYS_RANKING_MODE=static  uvicorn ai_engine.recsys.api:app

# 2. train
python bandit/train.py --log ./data/eventlog --out ./data/bandit_state.json
#    samples=... rewarded(+)=...
#    feature   prior_theta  trained_theta   <- see the weights move

# 3. serve the learned policy
RECSYS_RANKING_MODE=bandit  BANDIT_STATE_PATH=./data/bandit_state.json  uvicorn ai_engine.recsys.api:app

Online incremental updates¶

When RECSYS_BANDIT_ONLINE=true (with ranking_mode=bandit), the ingest endpoint updates θ live as reward events arrive, good for the low-data regime. On each CONTENT_VIEW_ENDED that echoes a request_id:

the served feature vector is looked up in the impression store (request_id then {content_id: features}),
θ is nudged by the realized engagement, and
the impression is consumed (dropped) so a redelivered reward (at-least-once webhook, retry) cannot double-count. The update is idempotent.

The new θ is persisted to BANDIT_STATE_PATH (atomic replace), so it survives a restart. The ingest response reports bandit_updates. A writer lock serialises the mutation.

Bootstrap from legacy data (`bandit/replay.py`)¶

θ is over abstract per-scorer features (tag-match vs semantic vs recency), not items, so weights transfer across studies even when the content differs, as long as the other study's items can be tagged and embedded. replay.py does a temporal replay (no leakage): for each view in a session, the feature vector is computed from the model built on earlier views only, and the reward is that view's engagement strength, exactly the serve-then-reward pairing the live trainer reconstructs.

python bandit/replay.py --content their_items.jsonl --sessions their_sessions.jsonl \
    --out ./data/bandit_state.json --weight 0.5     # down-weight foreign data

--weight < 1 down-weights foreign samples so live memorial data dominates later; --init continues from an existing state.

Held-out evaluation¶

bandit/eval.py answers "is this data actually good for the bandit?" It splits sessions into train / holdout, trains θ on train via temporal replay, then on the held-out sessions measures whether θ·x predicts engagement better than the static prior.

AUC: probability an engaged item scores above a non-engaged one (0.5 is chance).
corr(score, r): Pearson correlation of the score with realized reward.
Verdict: USE IT (trained beats prior, the data transfers), KEEP PRIOR (no transfer or training hurts on holdout), or INCONCLUSIVE (holdout has only one class).

Training health¶

LinearBandit.health() exposes how much data each weight has seen and how confident it is, so the Policy tab and evaluation can report a verdict rather than raw numbers:

Field	Meaning
`n_updates`	rewarded impressions folded in so far
`std[i]`	posterior std of θᵢ (`sqrt` of the `A⁻¹` diagonal); shrinks with data
`data[i]`	`A_ii − ridge`, the total `xᵢ²` mass observed; `0` means that feature never fired

/api/policy turns these into a per-weight confidence (0 at the prior, 1 confident), flags weights still at the prior (no_data), and emits a verdict: cold (< 20 updates), learning (mean confidence < 0.5), or converged.

Knobs¶

env / config	default	meaning
`RECSYS_RANKING_MODE`	`static`	`static` weighted fusion, or `bandit` learned θ
`BANDIT_STATE_PATH`	–	trained state JSON; absent then serve at the prior
`RECSYS_BANDIT_ONLINE` / `bandit_online`	`false`	update θ live as reward events arrive (vs offline batch)
`RECSYS_BANDIT_ALPHA` / `bandit_alpha`	`0.3`	UCB exploration strength (0 = greedy)
`RECSYS_BANDIT_RIDGE` / `bandit_ridge`	`1.0`	prior strength (how tightly θ₀ holds the weights)
`bandit_explore`	`true`	add the UCB bonus when serving

Scope / next¶

Global policy (one θ shared across users). A per-segment θ (e.g. by persona bucket) is the next step: more tailored weighting, sparser data per arm, the bridge to the explainable clusters.
Reward is a proxy: the offline trainer reuses engagement strength with a nominal reading-time estimate (the logs lack word_count). Relative reward ordering drives learning; swap in the exact strength once word_count is logged at serve.

Full auto-generated reference

Code reference -> Recsys package.