Compare recommenders in an experiment
Recommenders can use different strategies — for example, one objective-based and one rule-based — so the one that looks better in theory may not win in practice. An experiment settles that by running both live and measuring the result, instead of guessing.
Set up the comparison
- From Personalization, create an experiment and add one variation per recommender you want to compare.
- Point each variation at the point where the recommender is used (for example, a page or email placement), swapping only the recommender between variations.
- Split traffic evenly across the variations and pick a single success metric (for example, click-through or conversion).
- Start the experiment and let it run until it reaches statistical significance.
- Open the experiment’s results, confirm the winner, then promote it and archive the experiment.
Recommender vs. general experience experiments
Comparing recommenders uses the same experiment mechanics as comparing any other personalized experience, but what you swap between variations differs.
| Setup aspect | Recommender comparison | General experience comparison |
|---|---|---|
| What varies between variations | The recommender (e.g. objective-based vs. rule-based, or two objectives) | The experience itself (e.g. content, layout, or campaign copy) |
| Where it runs | Any point where a recommender is attached | Any point serving a personalized experience |
| Typical success metric | Click-through or conversion on the recommended items | Engagement or conversion on the experience |
| Change discipline | Change only the recommender between variations | Change only the element under test between variations |
Discussion
Comments are not configured for local development. They appear once the site owner connects GitHub Discussions (Giscus).