IDLab Workshop

Timofey Shvaya presented the study "Crowd Size and Prediction Precision: an Analysis of Metaculus Data". Co-authors of this research: Petr Parshakov, Iuliia Naidenova, Dennis Coates

This paper investigates the optimal crowd size for making accurate predictions across various fields, leveraging data from the online forecasting platform Metaculus. Crowdsourcing, the practice of soliciting contributions from a broad audience via the Internet, taps into the "wisdom of the crowd" to make more accurate predictions than any single individual could. Despite its recognized potential, debates persist regarding the ideal characteristics and size of the crowd for maximal predictive accuracy. Previous studies have presented mixed findings, with some advocating for larger groups and others suggesting small, knowledgeable teams may yield better results. Utilizing a dataset of 1,687 observations from Metaculus, this study analyzes prediction accuracy, measured by the Brier score, across different crowd sizes and topics, including politics, technology, and science. The analysis reveals a U-shaped relationship between the number of forecasts and prediction accuracy, suggesting an optimal forecast number of approximately 1200. The findings indicate that while larger crowds can improve predictions, there is a threshold beyond which additional forecasts do not enhance and may even diminish accuracy. This paper contributes to the understanding of crowd-based forecasting's dynamics, providing insights into how to effectively harness collective intelligence for predictive purposes.