Future Perfect Prediction?
Wednesday, 10 July 2019By Now&ZYen
Future Perfect Prediction? Fun with Recruitment Games
Prediction Markets
The future perfect tense in English isn’t straightforward – “will you have predicted that?”. RBS Insurance has used (present perfect tense) prediction games about the future at recruiting events for new staff, e.g. young actuary conferences. The games ask participants to estimate likely insurance claims for a few classes of business in return for potential prizes and, of course, discussing a possible career with RBS Insurance.
James Surowiecki, in his fascinating book, The Wisdom of Crowds, explores where large numbers of people can be very good at arriving at correct answers to complex problems. He looks at situations ranging from guessing the weight of cattle at agricultural fairs to complex algorithms such as finding submarines. He concludes that to use crowds to predict well four conditions must typically be met:
- “diversity of opinion – each person should have some private information, even if it’s just an eccentric interpretation of the known facts;
- independence – people’s opinions are not determined by the opinions of people around them [a subtle point];
- decentralisation – people are able to specialise and draw on local knowledge;
- and aggregation – some mechanism exists for turning private judgements into a collective decision.”
Prediction markets continue to get trendier. The University of Iowa’s Electronic Market for elections dates back to 1988. Z/Yen created ExtZy, allowing people to trade web entities such as country news hits, back in 1999. The Hollywood Stock Exchange has captured press attention by letting people trade virtual stars and predict movie success, more recently for real money. RBS’s games are interesting, and provide a fun field test of prediction markets.
RBS Insurance Games
RBS Insurance uses four simple games – space car insurance, pet robot insurance, space travel insurance and dwelling pod insurance. Obviously these are really four contemporary insurance lines – cars, pets, travel and housing – dressed up for fun. RBS Insurance puts up a large poster with five variables for the insurance and fifteen examples of the average claim histories. RBS Insurance then gives participants the five variables for two new examples and asks them guess the claim averages. A sample poster is Figure 1 – Space Car Insurance – where the variables are operating height, environmental impact, maximum passenger capacity, maximum hover speed and operator licence type. The games have been tried out five times (pet insurance twice) with 210 players giving two guesses each (a few players have played multiple games). RBS Insurance recorded the guesses and we thought it would be fun to analyse the results a bit.
Figure 1 – Space Car Insurance
In order to analyse the results we needed some comparators. Two comparators came to mind. First, the very fancy, a support vector machine combined with a neural network ought to provide a reasonable automated guess. The tool we used was PropheZy, Z/Yen’s proprietary implementation of a Support Vector/Neural Net, though there are free shareware tools around that can provide a Support Vector/Neural Net service. Support Vector/Neural Net was unlikely to do well on such a sparse data set, but should form a minimal comparator. Second, we needed a ‘naïve’ estimator, or benchmark. Something that could be performed by a simpleton. We chose Simple Averaging of the results.
Let’s look at one game in detail, home insurance. This table show the answers to playing the game on home insurance:
Here the Players haven’t done badly. But how well did Support Vector/Neural Network do?
The players seem to have done much better than sophisticated statistics. But were the players better than a very simple averaging guess?
In the home insurance game, Unknown 1 goes to Simple Averaging and Unknown 2 goes to the Players. Over the ten guesses we have, the Players did better than Simple Averaging, seven wins for the Players, three for Simple Averaging and nil for Support Vector/Neural Network. Further in the three cases where the Players did poorly, they were only narrowly beaten in two cases and by 10% in the third, while where Simple Averaging lost it could be significantly off, e.g. by nearly 55% in one case.
However, when money is brought into the equation, then simple averaging actually works better, just. The average Simple Averaging difference across the games came to -£74.96 while for the Players -£75.70. However, on total absolute difference the Players came to £506.90 as opposed to £578.86 for Simple Averaging. Hmmm.
Conclusion
This small example shows that in one case the crowds do a good job, slightly better than a simple averaging and quite a bit better than some advanced statistical tools, though to be fair the advanced tools are meant for much larger datasets. What we’d like to learn about in future is how the good players that RBS Insurance recruits do in real life. Or perhaps even to hear about RBS Insurance’s plans to use the guess-work of crowds in the cloud to replace their own actuarial teams. But perhaps that’s too futuristic.
Thanks
With thanks to Jez Horne and Richard Leeds of Z/Yen for providing the numbers and the support vector/neural network testing; to the team at RBS Insurance, especially Claire Baker and Robert Warner.
Further Reading
- SUROWIECKI, James, The Wisdom of Crowds: Why the Many Are Smarter Than the Few, Little, Brown, 2004.
http://www.forecastingprinciples.com/marketsforforecasting/index.php/research