A/B Testing Recommendation Systems

**Engagement metrics:**

• CTR (Click-Through Rate) — Are users clicking?
• Watch time / dwell time — Are they engaging?
• Session length — Are they staying?

**Business metrics:**

• Conversion rate — Are they buying/subscribing?
• Revenue per user — What's the dollar impact?
• Basket size — Are they buying more items?

**Long-term metrics:**

• Retention / return visits — Are they coming back?
• Lifetime value — What's the long-term impact?

**Health metrics (guardrails):**

• Catalog coverage — Are we showing diverse items?
• Novelty — Are we surfacing non-obvious items?
• User satisfaction surveys

**Best practices:**

• Run experiments long enough (2-4 weeks minimum)
• Use holdout groups to measure long-term effects
• Correct for position bias in analysis
• Monitor for data leakage between groups

**How it works:**

• Mix results from two rankers in a single list
• Track which ranker's items users prefer
• Detects differences 10-100x faster than A/B

**Why it's faster:**

• Every user provides signal for both models
• Directly compares relevance, not aggregate behavior
• Eliminates variance from user population differences

**Limitations:**

• Only measures ranking quality, not diversity or other factors
• Can't measure revenue or conversion directly
• More complex to implement

**Offline metrics:**

• Precision@k — Are top k items relevant?
• Recall@k — Are relevant items in top k?
• NDCG — Are relevant items ranked correctly?
• Hit rate — Did we retrieve any relevant item?

**Building evaluation sets:**

• Hold out recent data for testing
• Create human-labeled relevance judgments
• Include challenging cases (new items, edge cases)