Evaluation¶
How the engine will be assessed against its goals for engagement and learning. Performance and engagement measured quantitatively; user experience and satisfaction qualitatively.
Building the engine is not enough; it has to demonstrably help visitors learn and engage, and do so respectfully. The evaluation plan groups checks into three modules, each noting whether it is run internally (by the team) or externally (with users), and each with example metrics so assessment is concrete rather than vague.
User modelling and adaptation (internal) assesses how well the system builds profiles and adapts over time.
| Aspect | Metric |
|---|---|
| User persona alignment | correlation between survey/persona data and generated recommendations (qualitative) |
| Adaptivity over time | improvement in relevant content after repeated interactions; simulations of extreme interest, frustration, disinterest (quantitative) |
| Cold-start handling | success rate for new or anonymous profiles, measured as sustained engagement in the chosen topic (quantitative) |
User experience (external) measures engagement, learning, and perceived personalisation, coordinated with WP7.
| Aspect | Metric |
|---|---|
| Knowledge gain | pre- and post-surveys and user interviews |
| Engagement and immersion | average session duration and navigation depth from implicit indicators |
| Personalisation experience | survey results, target mean at least 4 on a 5-point Likert scale, plus interviews |
Ethical (internal) evaluates sensitivity and balance, per the MEMORISE ethical guidelines.
| Aspect | Metric |
|---|---|
| Diversity and sensitivity | distributional analysis of recommended content across themes, communities, and narratives, attending to similarity and dissimilarity so the nuance of HNP is communicated |
These modules will be put into practice and reported in the next deliverable, D5.4 (performance evaluation of the Individual Experience framework on final applications).
Bandit training health¶
The learned ranking policy reports whether it has seen enough data to be trusted, so the
Policy tab and operators can read a verdict rather than raw weights. From
LinearBandit.health() and surfaced by GET /api/policy:
| Diagnostic | Meaning |
|---|---|
n_updates |
rewarded impressions folded into θ so far |
per-weight confidence |
derived from the posterior std (sqrt of the A⁻¹ diagonal); 0 at the prior, approaching 1 as a weight gains data |
no_data |
weights still at the prior (their feature never fired) |
verdict |
cold (< 20 updates), learning (mean confidence < 0.5), or converged |
This makes "the bandit is still cold, keep the prior" an observable state rather than a guess. See Bandit / online learning.
Held-out legacy evaluation¶
Before enabling the bandit, bandit/eval.py tests whether legacy interaction data actually
transfers. It splits sessions into train / holdout, trains θ on train via temporal replay,
then on the held-out sessions checks whether θ·x predicts engagement better than the
static prior.
| Metric | Reads |
|---|---|
| AUC | probability an engaged item scores above a non-engaged one (0.5 is chance) |
| corr(score, r) | Pearson correlation of the score with realized reward |
| verdict | USE IT (trained beats prior), KEEP PRIOR (no transfer / training hurts), INCONCLUSIVE (holdout one-class) |
So the decision to switch on the learned policy is evidence-backed, not assumed.
End-to-end evaluation plan¶
A hands-on checklist validates the whole system with a colleague, mostly from the
Inspector with a few curl calls (budget about 45 minutes). Each point is
Action then Expect then Result (pass / fail / note), grouped into seven sections:
| Section | Covers |
|---|---|
| A. Serving & ranking | cold-start never-empty, breakdown bars, the distractor at rank 3/4, exclude-seen, compact mode, location filter, geo |
| B. User model | ingest a deep read, model built, history, negative signal, recs react, survey persona |
| C. Explainability & cohort | Falk persona + rationale, grounded prose, Personas reference, clusters, cohort content |
| D. Learning | static vs bandit mode, prior == static, the request_id online update, idempotency, training health |
| E. Durability & ops | served log, Parquet event log on disk, metrics, survives restart, live settings |
| F. Security & guards | ingest guard, PII guard on user-model / stats endpoints, serving stays open |
| G. Robustness | unknown user, empty filter, malformed event, geo without coords, exhausted location re-show |
Blockers (must pass): cold-start (A.1), the distractor (A.3), ingest building the model (B.1,
B.2), and the online bandit update via request_id (D.3). The full checklist with exact
actions and expected diagnostics lives in the repo (docs/e2e-eval.md).