IDEAL Policy Workshop Report

IDEAL Virtual Policy Workshop: Monday 19th October 2020

This workshop was held to explore the potential usefulness of IDEAL to UK government organisations which deal with the scientific evaluation of safety and effectiveness of medical treatment. The meeting was supported by funding from UKRI, Research England and the University of Oxford’s QR fund.

Representatives attended from the four government organisations with a major role in evaluation in this area (MHRA, NICE, NHS England Clinical Commissioning and NIHR) and also from the medical device industry. 

Five working groups were formed, based on these major stakeholders.  Groups were facilitated by members of the IDEAL Collaboration and met online several times before the plenary conference.  The four organisation-based groups were each asked to produce a consensus report on the feasibility and desirability of integrating IDEAL into their organisations’ infrastructure for evaluation or (in the case of NIHR) creation of clinical evidence.  The industry group were asked to critique the work of the four organisations in their current state, and consider whether their fitness for purpose could be enhanced by integration of IDEAL.

Proposals from the IDEAL Collaboration suggested several ways of integrating IDEAL into infrastructure, with the potential for benefit roughly correlating with the cost of implementing them in terms of finance, human resources and organisational change required.  These were:

  • Level 1: Adopting the language of IDEAL to identify and describe innovations, evaluations and likely evidence gaps
  • Level 2: Recommending IDEAL as a useful framework for designing research (providing tools)
  • Level 3: Requiring evidence which addresses key IDEAL Stage questions
  • Level 4: Linking staged approval to staged evidence production, using IDEAL to guide process

The industry group indicated that using IDEAL as a common language shared by NICE, MHRA and Commissioners to describe treatments and their stage of evolution, and the study types used to evaluate them would be welcome  They recognised that IDEAL could help identify research needs at different stages in the device life cycle.  It would help industry if all three government bodies whose approval they would need approved and accepted IDEAL format studies, so that evidence could be “produced once and used often”.  They emphasised also that evidence requirements need to be tailored to the risk profile of the device.

The four working groups based on organisations all saw the arguments for integrating IDEAL into their systems, and agreed there was a potential benefit to varying degrees.  The MHRA were constrained by their ongoing development of an entirely new UK-specific approval system for devices, which prevented them from commenting in a range of areas, but engaged positively in discussion about the potential benefits of more standardised set of guidelines for studies to support approval applications.  The NHS Commissioning group was very positive about the potential for IDEAL to be integrated into their work and suggested further contacts to explore this.  NICE were interested in the possibility of using IDEAL as a common language, but were more cautious about other measures, arguing that their current systems appeared to be functioning well and had received widespread approval.  The NIHR group considered that IDEAL had great potential and set out a series of practical steps which could help ensure it is used more in research.  These included encouraging funding and scientific boards within the NIHR programmes to use IDEAL language when discussing studies of complex interventions and promoting the use of IDEAL pilot studies leading on to RCTs.

The meeting closed with an agreement that IDEAL would follow up with each stakeholder group to explore whether there were specific opportunities for change which they wished to discuss further.

Read the full report here:  IDEAL Policy Workshop Report Oct 2020 FInal



Redesigning the IDEAL website

In this post, we invite you to participate in helping with the IDEAL website redesign.  You can get started right away if you want by completing this one-question survey about your specialty or discipline.  (If you’ve already completed the pop-up one, you don’t need to do it again.)

It’s now over 10 years since the IDEAL Collaboration was launched.  The time has come to bring our website up to date.  This blog is about how we are planning on doing this, and how you can help.

A lot of water has flowed under the bridge since 2010, but one thing has not changed.  New health care interventions must be underpinned by reliable evidence of effectiveness and safety. As we anticipate a future of further depleted resources and hugely increased pressure on health spending, so new interventions will have to prove their worth.  We urgently need better study designs that encourage innovation whilst reliably demonstrating effectiveness.

IDEAL is now internationally recognised as an integrated evaluation pathway for evaluation of complex interventions.  Starting with surgery, it has now been used in invasive non-surgical procedures, physiotherapy, radiotherapy, quality improvement studies and the evaluation of therapeutic devices.

The IDEAL guidelines for reporting were published in 2020, and appear on the EQUATOR guidelines pages.  Endorsements have come from major journals (Lancet, BMJ, Annals of Surgery) and professional bodies (Royal College of Surgeons).

There are IDEAL centres in the Netherlands and China as well as the Oxford centre.  Plans for a North American IDEAL network are progressing quickly.  IDEAL is being used in Health Technology Assessment and commissioning/purchasing decisions in Scotland, the Netherlands and Canada.  The head of the Chinese IDEAL Centre has been appointed as the scientific advisor for the Chinese technology evaluation centre in Hainan province.

The number of papers using or citing IDEAL continues to grow, and the NIHR in the UK has funded a number of IDEAL format studies, including an ongoing study of image guided brain cancer surgery.

Our recent Policy Forum was attended by high level representatives of NIHR, NICE, the MHRA (the UK’s regulator for medicine and medical devices) and the Commissioners at NHS England.  We are now in discussions about using IDEAL more systematically in the Commissioning process.

We have developed a range of training materials and a faculty of presenters, and have recently started to offer an Advisory service for those wishing to develop their research or evaluation using IDEAL principles.

The Collaboration continues to move forward intellectually too.  Hani Marcus from London has led the development of detailed guidance on IDEAL Stage 0 – the pre-clinical evaluation of devices.  Arsenio Paez from New York has led a group to discuss how to decide whether a randomised trial is necessary for new devices.  Both papers will be published shortly.  Meanwhile IDEAL has begin a major project on developing guidelines for the evaluation of surgical robots.

Meet our team and our vision for the website redesign

Peter McCulloch

Here’s what we want to achieve with the updated website:

  • Improved adherence to accessibility standards
  • A modern, engaging visual identity
  • Easier access to IDEAL resources
  • Better support for educational events
          • Better support for innovators and researchers planning evaluations of surgical procedures.


Allison Hirst

Because surgery is procedure-based, it presents challenges to gathering, reporting, and applying sound evidence that are not seen in medicine. Therefore our broad aim to increase awareness and utilisation of the IDEAL framework in surgical research.

A user-centred approach

Arsenio Paez

Evidence from human-computer interaction research consistently shows that involving users early on in any design process increases the chances of success.

This ought to be a truism but isn’t.  Sometimes, we assume we know what users want without asking them.  Often, we lack a clear understanding of what exactly users want to achieve in interacting with a resource.  On the other hand, user research can be a never-ending piece of string.


Mudathir Ibrahim


So the key challenge is:

How do we get best value from user involvement using the resources we have available?



Baptiste Vasey

Here’s how we intend to approach it:

  1. A series of simple surveys on the IDEAL website, to understand who uses the site and what they are looking for.  Take the first one now!
  2. Recruiting an informal, representative group of website users who can provide insight on the user experience
  3. Conducting cycles of design, development, and evaluation  with this group as we progress through the project.


IDEAL is a website that has something for a varied audience.  In particular, we need to involve surgeons, researchers, editors, and commissioners.

If you would like to help, we’ll be happy to hear from you.  Tweet us at @IDEALcollab, or email Allison Hirst at, tell us a little about yourself and we’ll take it from there.


IDEAL Reporting Guidelines published in Annals of Surgery

We are delighted to announce the completion and publication of the IDEAL Reporting Guidelines which we hope will improve the quality of surgical research by increasing awareness of the key methodological issues throughout the innovation life-cycle. We hope they will serve to assist researchers/authors in their study planning, conduct and reporting of IDEAL Stage studies. We encourage Journals to alert authors to these guidelines in their Instructions to Authors.

We would like to thank the IDEAL Reporting Guidelines Working Group for their hard work in the consensus guideline development process:

Jane Blazeby, Maroeska Rovers, Christopher Pennell, Joel Horovitz, Angelos Kolias, Janet Martin, Tom Lewis, Riaz Agha, Josh Feinberg,

We would also like to thank the participants who participated in 2 Rounds of the Delphi Survey:

Stage Checklists available here:

Word and PDF versions for each

Stage 1 Word  IDEAL Reporting Guidelines Stage 1 checklist

Stage 1 PDF  IDEAL Reporting Guidelines Stage 1 checklist

Stage 2a Word  IDEAL Reporting Guidelines Stage 2a checklist

Stage 2a PDF  IDEAL Reporting Guidelines Stage 2a checklist

Stage 2b Word  IDEAL Reporting Guidelines Stage 2b checklist

Stage 2b PDF  IDEAL Reporting Guidelines Stage 2b checklist

For Stage 3 use: SPIRIT for RCT protocols

and CONSORT for RCT full study reports

Stage 4 Word    IDEAL Reporting Guidelines Stage 4 checklist

Stage 4 PDF  IDEAL Reporting Guidelines Stage 4 checklist

Statement on Independent Medicines and Medical Devices Safety Review

The Independent Medicines and Medical Devices Safety Review, chaired by Baroness Julia Cumberlege, has today published its report. Titled “First Do No Harm”, it comes after a two-year review of harrowing patient testimony and a large volume of other evidence concerning three medical interventions: Primodos, sodium valproate and pelvic mesh.

“First Do No Harm” sets out nine major recommendations to bring much-needed help and support to those who have suffered as a result of these interventions, and to reduce the risk of avoidable harm from medicines and medical devices in the future.

The Review’s major recommendations include:

  • That the Government immediately issues a fulsome apology on behalf of the healthcare system to the families affected by Primodos, sodium valproate and pelvic mesh.
  • That a Patient Safety Commissioner is appointed. This person would be the patients’ port of call, listener and advocate, who holds the system to account, monitors trends, and demands action.
  • Separate schemes should be set up for Hormone Pregnancy Tests, valproate and pelvic mesh to meet the cost of providing additional care and support to those who have experienced avoidable harm and are eligible to claim.
  • A Redress Agency for those harmed by medicines and medical devices in future should be established.
  • The establishment of two types of specialist centres, located regionally – for mesh, and separately for those affected by medications taken during pregnancy.
  • The regulator of medicines and medical devices, the MHRA, needs to put patients at the heart of its activity, and to overhaul adverse event reporting and medical device regulation.
  • That a central database should be created by collecting key details including the patient, the implanted device, and the surgeon.
  • That the register of the General Medical Council (GMC) should be expanded to include a list of financial and non-pecuniary interests for all doctors, as well as doctors’ clinical interests and specialisms.
  • Finally, that the Government immediately sets up a task force to implement the Review’s recommendations.

The IDEAL Collaboration has released the following statement on the occasion of the publication of the Cumberlege report:
“The report of the Independent Medicines and Medical Devices Safety Review (the Cumberlege Review) documents the serious suffering caused by three treatments intended to help patients but which ended up injuring many. The lessons emphasised by the report include the establishment of better ways of allowing the patient’s voice to be heard, improved informed consent procedures and the implementation of remedies to help those who have suffered harm. As with all enquiries into failure, the most important of its several objectives is to ensure that reform occurs which will prevent the story from being repeated. In the case of vaginal mesh, the focus of attention must be on improving the systems for licensing, regulating and monitoring medical devices. Without better methods for ensuring that the safety and effectiveness of medical devices is evaluated thoroughly, further scandals of this type are sadly inevitable.

The IDEAL Collaboration has developed recommendations for scientific study of devices which provide an integrated evaluation pathway throughout their life cycles. This begins with maximum transparency about the device, its use and the outcomes from the very first case. We advocate a Total Product Life Cycle (TPLC) approach to evaluation and licensing, in which permission to market a device is widened in stages as more clinical evidence is provided. At each stage in the life cycle, sponsors should be required to conduct studies following a standardised template designed to provide the highest quality and most important evidence relevant to that stage. This is much more likely to identify safety issues early than current arrangements. The Cumberlege report references some of these important ideas, and makes recommendations for evidence development at both early stages (Para 5.52) and during longer term follow up (Para 1.48) aligned with IDEAL recommendations, which we very much welcome.

The report correctly points out the current opportunity for reform of device regulation in the UK in the wake of Brexit. The notified body system used in the EU drastically limits the power of national regulators to specify evaluation methods and to see the results of tests done for regulatory purposes. A new framework in which sponsors of new devices will be required by the MHRA to provide data from prospective ethically approved studies using standardised formats in return for progressive degrees of market access (and to contribute to the costs of maintaining a long term device registry) would make repetition of the mesh story much less likely. IDEAL provides an explicit framework for achieving this”

Prof Peter McCulloch, Chair, IDEAL Collaboration on behalf of the IDEAL Council

FREE virtual IDEAL meeting: How should we evaluate innovations in a crisis? Friday 3rd July 2020

We had a very interesting meeting discussing how we should evaluate surgical and device innovations in emergency scenarios on 3rd July.

The programme is below and can be downloaded here: Final Programme_3rdJuly IDEAL Virtual meeting

Presentations now available here:

Rovers for IDEAL meeting 3 July

PeterMcCulloch IDEAL Amsterdam

Campbell – ideal-innovations-jul2020-FINAL (1)

Hutchinson – ethics of innovation in a crisis_IDEAL presentation_final

Derek Stewart video –


How should we evaluate innovations in a crisis?

Free online IDEAL event – Friday 3 July 2020 (13.00 -15.30 UK time)


13:00 – 13:05  Short word of welcome by Maroeska Rovers, Professor of Evidenced-Based Surgery at Radboud University Medical Centre, Nijmegen, Netherlands

13:05 – 13:17  Recap on IDEAL and why the framework needs to be adjusted to deal with emergency scenarios – Peter McCulloch, Professor of Surgical Science and Practice, Nuffield Dept of Surgical Sciences, University of Oxford, UK

13:17 – 13:29  Innovations in disaster relief scenarios – Mamoon Rashid, Professor of Plastic Surgery, STM University & Head, Dept. of Plastic Surgery, Shifa International Hospital. Islamabad, Pakistan

13:29 – 13:37  Questions/discussion

13:37 – 13:49  How to assess innovations in a crisis – Marion Campbell, Vice Principal (Research) & Prof of Health Services Research, Health Services Research Unit, University of Aberdeen, UK

13:49 – 13:57  Questions/discussion

13:57 – 14:09  Ethical considerations  – Dr Katrina Hutchison, Research Fellow (MQRF) Department of Philosophy, Macquarie University, NSW, Australia

14:09 – 14:17  Questions/discussion

14:17 – 14:29  Public & Patient Involvement in emergency research  – Derek Stewart, PPI Lead with the MRC NIHR Trials Methodology Research Partnership

14:29 – 14:40  General discussion about all aspects

14:40 – 14:45   Grab a coffee or tea yourself 

14:45 – 15:30  Session 1: Tea corner meet people from the IDEAL collaboration to discuss more about our work

14:45 – 15:30  Session 2: Sandpit session where researchers can pitch their idea and get advice/feedback from others in that session

Best wishes

IDEAL Organising Committee: Maroeska Rovers, Peter McCulloch, Allison Hirst, Crispin Boyd


New service: advice on IDEAL stage study design

Advisory Service for designing IDEAL Stage studies

We are pleased to announce that we are piloting a service for researchers and innovators seeking guidance on planning a study and writing a protocol for their evaluation of a surgical technique or therapeutic device appropriate to any stage of the IDEAL Framework.

The service will be available free of charge to clinicians and scientists doing this work as part of a grant-funded research project, or as an unfunded service improvement. Contact

A separate Consultancy arrangement is available for commercial companies, through our associated non-profit company, IDEAL Innovation CIC.  To access this, please send initial enquiries to

If you are unsure which service is appropriate for you, please enquire to the e-mail address below in the first instance.

What do you need to do?

We ask you to contact us in the first instance to check your work is suitable – email

We will then ask you to identify the IDEAL stage and key purpose/question of your study using our key IDEAL papers and Flowchart to do so;

Once we have confirmed the relevant stage with you we will allocate you an expert IDEAL Advisor to help you develop your protocol using stage-specific guidance which we will provide. Our advisors for the service include; Peter McCulloch, Art Sedrakyan, Arsenio Paez, Joel Horovitz, Angelos Kolias, Maroeska Rovers, Jane Blazeby, Fei Shan, Andrew Cook, Hani Marcus, Natalie Blencowe , Nicole Bilbro, Joshua Feinberg and Christopher Pennell.

The purpose of the service is for you to produce a clear written protocol for your study which you might consider publishing with your advisor as co-author.



Evaluation of the service  

We aim to evaluate how useful our tools and service are to researchers in order to a) improve our service and b) learn from real cases studies where the IDEAL Framework/Recommendations can be amended/updated. As soon as possible after completion of your protocol we will ask you to complete a short online survey asking for your feedback on the experience.


Thank you.

We look forward to hearing from you.

Peter McCulloch (Chair) and Allison Hirst (Project Manager) on behalf of the IDEAL Team

Postponement of IDEAL 2020 Conference in Amsterdam due to COVID-19

Due to the COVID-19 pandemic we have taken the difficult decision to postpone our international IDEAL 2020 Conference scheduled to take place on 11th-12th June 2020 in Amsterdam. We now anticipate holding the event in April 2021 in Amsterdam. Please bear with us as the Organising Committee works through the next steps and confirms the new date.

We will be sending further communications to individuals with accepted abstracts as soon as possible and more information will be posted on our websites about the future event as it becomes available: and

Thank you all for your understanding in these uncertain times, and we wish you all the best.

We will endeavour to host a successful and enjoyable IDEAL 2021 Conference in Amsterdam in the spring.

Maroeska Rovers and Peter McCulloch, On behalf of the Organising Committee

How should innovation be evaluated in global surgery? Contribute your thoughts. Survey is now open.

Invitation on behalf of the Global Health Research Group in Surgical Technologies

Are you a surgeon or surgical researcher with an interest in global surgery, surgical innovation or technology? Or perhaps you have or want experience in evaluating surgical technology and/or innovations in global surgery?

Can you spare just 20 minutes to complete our survey? This study aims to develop a framework for the evaluation of innovation in global surgery. The survey explores barriers and facilitators to evaluating innovation and technology in global surgery. In particular, how should we evaluate the adoption of an established technology in a novel context or setting?

If so please click the link here

We are a group of researchers from the University of Leeds called the Global Health Research Group in Surgical Technologies ( The primary aim of the group is to develop and evaluate novel technologies to improve surgical training and care in LMICs.

All the questions are in English. You do not have to complete the survey and your participation is entirely voluntary. You will not be asked for your name or email address as part of the survey itself, however once you complete the survey you will have the opportunity to express your interest to be involved in a semi-structured interview about your experiences with a member of the research team. This may be conducted online.

More information can be found on the opening page of the survey in the participant information page before you begin the survey. To view this, just click the survey link above.

If you have any questions about this study please email Dr William Bolton at the University of Leeds, 

This study has received ethical approval from the University of Leeds School of Medicine Research Ethics Committee MREC 18-102.

Thank you for your time,

Dr William Bolton on behalf of the NIHR Global Health Research Group in Surgical Technologies

NIHR Clinical Research Fellow
Global Surgical Technologies
Co-Founder MedTechFoundation


IDEAL Conference 2020 – Amsterdam 11-12th June

IDEAL2020 in Amsterdam, the Netherlands: Innovation and Evaluation in Surgery

Save the date: 11th-12th June 2020

IDEAL2020 will bring together clinicians and researchers from all over the world to discuss the latest scientific methodological developments on investigating new surgical interventions and medical devices in a variety of clinical areas.

The programme will include state-of-the-art plenary lectures, workshops, sandpit session and selected presentations from submitted abstracts. There will be ample opportunity for interactive discussions and networking.

The deadline for abstract submission is 1st February 2020, and the early bird registration deadline is 22nd March 2020. Please visit this link to submit your abstract and/or to register for the conference:

The congress venue is located at one of the famous Amsterdam canals, and can be reached very easy by tram from the central station. It is just a short walk from Amsterdam’s leading cultural centre.

For more information please keep an eye on our twitter accounts: