When is expert elicitation most valuable in decision-making?
In evidence-based practice, the hierarchy of evidence classifies sources of information according to their ability to minimise bias and produce reliable, generalisable conclusions. Expert elicitation, which involves gathering judgements or insights from subject matter experts, is typically positioned lower in this hierarchy due to its reliance on subjective input rather than empirical data. However, it becomes an essential tool in situations where high-quality data such as that from systematic literature reviews (SLRs) or randomised controlled trials (RCTs) is unavailable, or impractical to obtain.
Such uncertainty often arises in therapy areas such as rare diseases, or for advanced therapy medicinal products (ATMPs) or emerging technologies: these are cases where limited patient populations, ethical constraints, or the novelty of interventions restrict the availability of robust evidence. In these contexts, expert elicitation can provide a structured and transparent means of informing reimbursement and policy decisions, enabling stakeholders to navigate uncertainty while maintaining a rigorous approach to evidence generation.
Which expert elicitation methodology should you use?
The National Institute of Care and Excellence (NICE) provides clear guidance on the use of expert elicitation and expert opinion in health technology assessments (HTAs), emphasising a strong preference for structured methodologies to minimise bias and quantify uncertainty. Structured approaches such as modified Delphi panels and the Sheffield Elicitation Framework (SHELF) are increasingly favoured for their ability to elicit well-informed judgements in a transparent and reproducible manner. These methods complement traditional evidence sources and are expected to follow established protocols, such as those outlined by the Medical Research Council (MRC), which typically involve training sessions for the consulted experts to mitigate common cognitive biases.
Structured expert elicitation (SEE) has also been explicitly recognised in HTA guidance by other agencies such as Zorginstituut Nederland (ZIN) and the Canada Drug Agency (CDA), reflecting a growing international consensus regarding the value of SEE. This alignment underscores the critical role of structured elicitation in supporting robust, transparent, and credible HTA submissions, particularly in contexts where empirical data is limited or uncertain.
When the objective is to obtain strategic insights, gauge price perceptions, or collect large volumes of relatively straightforward quantitative data, less structured approaches – such as interviews, surveys, and advisory boards – may be more suitable. These methods enable broader stakeholder engagement and can enrich or supplement structured techniques by capturing a diverse range of perspectives. The comparative strengths and limitations of these methods are outlined in the table on the next page.
Becoming familiar with a diverse set of expert elicitation techniques, and picking the correct one for each specific context, is therefore essential, not only for generating robust evidence but also for supporting comprehensive and informed decision-making within HTA processes.
A comparative overview of expert elicitation approaches for use in HTA
How do I best approach expert elicitation?
As outlined throughout, conducting expert elicitation effectively and robustly is essential, particularly in the absence of empirical data, to ensure a process that yields reliable outcomes likely to be accepted as valid by HTA agencies. Collaborating with a well-aligned, experienced organisation can be the key to success.
Partnering with an organisation like ourselves can help you significantly streamline your expert elicitation process. Our proven experience in facilitating expert elicitation enables us to deliver high-quality, reliable evidence, empowering you to make well-informed decisions with confidence. If you are interested in what we can offer you, please get in touch to learn more!
Abbreviations: ATMP, advanced therapy medicinal products; CEM, cost-effectiveness model; CDA, Canada Drug Agency; HCP, healthcare professional; HTA, health technology assessment; IDEA, investigate, discuss, estimate, aggregate; MRC, Medical Research Council; NICE, National Institute for Health and Care Excellence; RCT, randomised controlled trial; SEE, structured expert elicitation; SHELF, Sheffield Elicitation Framework; SLR, systematic literature review; ZIN, Zorginstituut Nederland.
