3.4 The Delphi method
The Delphi method was invented by Olaf Helmer and Norman Dalkey of the Rand Corporation in the 1950s for the purpose of addressing a specific military problem. The method relies on the key assumption that forecasts from a group are generally more accurate than those from individuals. The aim of the Delphi method is to construct consensus forecasts from a group of experts in a structured iterative manner. A facilitator is appointed in order to implement and manage the process. The Delphi method generally involves the following stages:
- A panel of experts is assembled.
- Forecasting tasks/challenges are set and distributed to the experts.
- Experts return initial forecasts and justifications. These are compiled and summarised in order to provide feedback.
- Feedback is provided to the experts who now review their forecasts in light of the feedback. This step may be iterated until a satisfactory level of consensus is reached.
- Final forecasts are constructed by aggregating the experts’ forecasts.
Each stage of the Delphi method comes with its own challenges. In what follows we provide some suggestions and discussion about each one of these.1
Experts and anonymity
The first challenge of the facilitator is to identify a group of experts that can contribute to the forecasting task. The usual suggestion is somewhere between 5 and 20 experts with diverse expertise. Experts submit forecasts and also provide detailed qualitative justifications for these.
A key feature of the Delphi method is that the participating experts remain anonymous at all times. This means that the experts cannot be influenced by political and social pressures in their forecasts. Furthermore, all experts are given an equal say and all are made accountable for their forecasts. This avoids the situation where a group meeting is held and some members do not contribute, while others dominate. It also prevents members exerting influence based on seniority or personality. There are suggestions that even something as simple as seating arrangements in a group setting can influence the group dynamics. Furthermore, there is ample evidence that a group meeting setting promotes enthusiasm and influences individual judgement leading to optimism and overconfidence.2
A byproduct of anonymity is that the experts do not need to meet as a group in a physical location. An important advantage of this is that it increases the likelihood of gathering experts with diverse skills and expertise from varying locations. Furthermore, it makes the process cost-effective by eliminating the expense and inconvenience of travel, and it makes it flexible as the experts only have to meet a common deadline for submitting forecasts rather than having to set a common meeting time.
Setting the forecasting task in a Delphi
In a Delphi setting, it may be useful to conduct a preliminary round of information gathering from the experts before setting the forecasting tasks. Alternatively, as experts submit their initial forecasts and justifications, valuable information not shared between all experts can be identified by the facilitator when compiling the feedback.
Feedback to the experts should include summary statistics of forecasts and outlines of qualitative justifications. Numerical data summaries and graphical representations can be used to summarise experts’ forecasts. As the feedback is controlled by the facilitator, there may be scope to direct attention and information from the experts to areas where it is most required. For example the facilitator may direct the experts’ attention to responses that fall outside the interquartile range, and the qualitative justification for such forecasts.
The process of the experts submitting forecasts, receiving feedback, and re-viewing their forecasts in light of the feedback, is repeated until a satisfactory level of consensus between the experts is reached. Satisfactory consensus does not mean complete convergence in the forecast value; it means that the variability of the responses has decreased to a satisfactory level. Usually two or three rounds are sufficient. Experts are more likely to drop out as the number of iterations increases, so too many rounds should be avoided.
The final forecasts are usually constructed by giving equal weight to all experts’ forecasts. The facilitator should keep in mind the possibility of extreme values which can distort the final forecast.
Limitations and variations
Applying the Delphi method can be time consuming. In a group meeting, final forecasts can possibly be reached in hours or even minutes — something almost impossible to do in a Delphi setting. If it is taking a long time to reach a consensus in a Delphi setting, the panel may lose interest and cohesiveness.
In a group setting, personal interactions can lead to quicker and better clarifications of qualitative justifications. A variation to the Delphi method often applied is the "estimate-talk-estimate method", where experts can interact between iterations. The forecast submissions can still remain anonymous. A disadvantage of this variation is the possibility of the loudest person exerting undue influence.
The role of the facilitator is of utmost importance. The facilitator is largely responsible for the design and administration of the Delphi process. The facilitator is also responsible for accommodating feedback to the experts and generating the final forecasts. In this role the facilitator needs to be experienced enough to recognise areas which may need more attention than others, and to direct the attention of the experts to these. Also, as there is no face-to-face interaction between the experts, the facilitator is responsible for disseminating important information. The efficiency and effectiveness of the facilitator can dramatically improve the probability of a successful Delphi method in a judgmental forecasting setting.
For further reading, refer to
G. Rowe (2007). A guide to Delphi. Foresight: The International Journal of Applied Forecasting 8, 11–16.
G. Rowe and G. Wright (1999). The Delphi technique as a forecasting tool: issues and analysis. International Journal of Forecasting 15, 353–375. ↩
R. Buehler, D. Messervey and D. Griffin (May 2005). Collaborative planning and prediction: Does group discussion affect optimistic biases in time estimation? Organizational Behavior and Human Decision Processes 97(1), 47–63. ↩