Research

Frozen cohort evaluation

Before trusting archetype rankings in production, we ran a fixed synthetic cohort study: same users, same sessions, same decisions — only the ranker and weekly snapshots changed.

Jul 21 – Aug 8, 2026 Status: closed eval seed 42 20% session holdout

Weeks tracked

Closed early; metrics stable

16.5%

Final hit@1

1413

Test sessions

Ranker version

Listwise softmax

Three headline metrics

session_hit_at_1 — ranker's top pick equals softmax-sampled choice
by_mode — stratified hit@1 across viewing contexts
by_dominant_archetype — stratified by user's strongest archetype weight

Compared against an affinity baseline that sorts by raw archetype–movie score.

Weekly snapshot trend

Week	Date	Hit@1
1	2026-07-21	16.6%
2	2026-08-03	16.5%
3	2026-08-08	16.5%

Flat curve → ranker converged; no benefit from extended study window.

By viewing mode (final)

impress

22.2%

kids win

19.9%

compromise

18.6%

obligation

17.1%

solo

12.4%

By dominant archetype (final)

Lover

18.9%

King

16.9%

Magician

15.4%

Warrior

14.4%

Methodology notes

Why synthetic?

Real users don't arrive with King/Warrior/Magician/Lover weights. Synthetic cohorts let us test the evaluation harness and ranker architecture before collecting labeled field data.

Why freeze?

cohort_frozen: true stops Stations 2–4 from regenerating users/sessions. Weekly snapshots measure model changes, not data drift.

What we learned

Hit@1 ≈ 16.5% beats random (10%) but tracks the affinity baseline closely. Mode and archetype stratification reveals where signal is strongest (impress, Lover).

Published artifacts

Dataset	Contents
`users.parquet`	500 archetype profiles
`sessions.parquet`	6,796 viewing contexts
`decisions.parquet`	91,793 surfaced + counterfactual rows
`v1.parquet`	Phase 2 export with optional LLM rationales
`cohort_study_snapshots.jsonl`	4 weekly metric records

Kaggle → theodorerubin/movietwirl · HuggingFace → datamatters24/movietwirl · Source → GitHub