Recommendations systems are powerful revenue engines: the better the system, the longer the users stay on the platform or the more the clients spend on the website. But systems are numerous and some perform better than others. In particular, some systems are automated and based on other users’ behaviours whereas others are more manual and based on experts. Which is best? For which objective? In which context? What are the strategic implications?
The research: The crowd against the few: Measuring the impact of expert recommendations. By Herm-Stapelberg and Rothlauf.
In this work, the authors tackle a stimulating question comparing the impacts of various recommendations systems on users behaviours of an online video-on-demand service. More than 6,000 users of this service accepted to be part of the experiment. They were randomly assigned either to the treatment group (receiving recommendations from experts) and the control group (receiving system-generated recommendations).
The main results
- Experts can significantly increase platform usage, are able to keep users longer on the website, and make users come back more frequently. Using experts leads to an increase in the number of watched clips (+9.8%), the use of recommendations (+17.1%), and whether people still used the platform after 10 weeks (+15% retain rate). Users visited the platform more often (+9%) and had more frequent visits where they watched more than one clip (+15.7%).
- Surprisingly, even though the treatment group significantly used the platform more often expert-based suggestions were used less than expected. This is in line with the finding that the probability of clicking on another recommendation decreases when the first clicked recommendation was generated by experts. Nevertheless, users who did not receive expert suggestions at all, have an even lower probability of clicking on another recommendation.
Even if expert-based recommendations are used less than expected, they have a large and significant impact on user behaviour. Expert recommendations are more diverse and better cover the taste of users.
What does it mean?
- Recommendation systems embed implicit (and often not debated) hypotheses about behaviours. For content-based systems (which use previously consumed items by the user), it is that people tend to favor what they already experienced themselves. For collaborative systems (based on other users consumptions), it is that people tend to mimic what their peers are consuming. For expert-based systems, it is that users have a wide range of preferences (diversity) and specific taste.
- There is no silver bullet. Some recommendations systems perform better than others in some industries, in a given context, for specific types of users or for particular objectives.
Useful questions for strategy-making
- Recommendation systems embed hypotheses about behaviours => How far have you analysed and understood your customers’ behaviors? Are they more sensitive to their peers or to experts?
- Recommendation systems can be a source for differentiation => How well do you know your competitors’ recommendation systems? What is your positioning (copying with a better performance or changing the logic of recommendation)?
- Recommendation systems are powerful revenue engines and we know more and more about their types and logic => Which systems are you using? Do you assess their performance against other systems (e.g.: user behaviour vs general trends)?
This article was first published on Prof. Benyayer’s blog.
This post gives the views of its author, not the position of ESCP Business School.