The grader knew which website version was new
A company runs an A/B test of a website redesign. Because the winning metric is subjective "page quality", an analyst hand-scores a sample of sessions from each version. The analyst, who championed the redesign, can see which version each session came from.
What bias does this introduce, and how should the scoring be done instead?
Your answer
This one is open-ended. Work it through, then check your reasoning against the full solution.