Is "IDEAL" Enough? Proposal for a New Framework (METEOR)

Dirk Stengel, M.D., PhD, MSc
Kathleen Füssler, MSc
Centre for Clinical Research,
Department of Trauma and Orthopaedic Surgery
Unfallkrankenhaus Berlin, Germany

Mohit Bhandari, M.D., MSc, FRCSC
Department of Surgery and Department of Clinical Epidemiology & Biostatistics, McMaster University
Hamilton, ON

Many surgical research projects commence with ideas, technical modifications and individual observations rather than specific hypotheses. The methodological difficulties of evaluating surgical complexity by established clinical research designs have been realized for years, but remain a matter of controversy and debate. In a series of three papers, the Balliol Group highlighted current shortcomings and summarized the archetype of surgical clinical research in the IDEAL statement (Idea, Development, Exploration, Assessment, Long-term Surveillance)1-3. While IDEAL gives a detailed overview of the past and reflects the present, it neither offers forward-looking solutions nor addresses the specific issues of orthopaedic surgery.


The latter is intrinsically tied to the development of new equipment and implants, creating complex interactions between demographic and biological variables, preferences, manual skills of the surgeon and hardware properties, all of which may contribute to the patient's clinical progress. Biological principles (e.g., open reduction and internal fixation, intramedullary nailing, locked plating), together with the expertise of the surgeon and the process quality of the institution, have a much higher impact on outcomes than individual implants. Because of diminishing marginal values, novel products with similar biomechanical features are unlikely to alter patient outcomes. Ceiling effects prevent the demonstration of marked improvements in endpoints like function, health-related quality of life and radiological surrogates (e.g., fracture healing, restoration of the mechanical limb axis) with a new as compared to an established implant. Strangely, approval regulations like the FDA's 510(k) process or the European CE certificate still accept equivalence to available devices as entry criterion to the market, whereas manufacturers later solicit the same products as superior to their competitors.

Recent changes in device approval regulations demand proof of effectiveness of the experimental intervention. According to the first philosophical bridge principle coined by Hans Albert ("Ought implies Can")4, the marginal expected gains with a new intramedullary nail compared to an established one represent a discussion stopper. Whereas an uncontrolled flood of implants and the approval of useless devices must be averted by regulatory and legal measures, there remains a delicate balance between avoiding extra costs to health care budgets and abolishing any commercial or scientific incentives for developers. This is very similar to the type I (alpha) error: accepting the 5% threshold exposes one in 20 patients to the risk of receiving an ineffective treatment, but lowering alpha to 0% means that no patient will have the chance to benefit from innovative therapies in future.

It is noteworthy that neither regulatory bodies nor manufacturers currently have convincing strategies to define the additional value and utility of new devices and implants compared to virtually similar products already available on the market.

Understandably, consumers, health care professionals, payers and producers are irritated and concerned about current device approval regulations, monitoring of safety issues, methods to demonstrate clinical benefits and so on.

Regardless of local or global financial crises, innovation in health care is socially and politically demanded. Also, manufacturers are under pressure to release new health care products, whether drugs or devices, on a regular basis to stay market competitive. Apparently, Health Technology Assessment (HTA) must not only consider the outcomes (including safety) observed with a new treatment, but the underlying theoretical construct and development efforts as well. The way to the product may be as valuable as the product itself, and should be accredited.

In addition to IDEAL, there is probably a need for a new approach to assess the utility and value of orthopaedic devices and related surgical procedures to distinguish between innovations that are likely to change the practice of care, interesting novelties that satisfy surgeons' needs, and technical modifications that may overcome minor weaknesses in established technology and techniques.

In a multi-institutional, international project (METEOR, Matrix to Evaluate Technology in Orthopaedics) led by the Unfallkrankenhaus Berlin, Germany, and the McMaster University, Hamilton, Canada, we set out to generate a tool that may: 1) allow the classification of the innovative potential of a new orthopaedic concept and/or implant, 2) ease communication between various stakeholders and, 3) add to established Levels of Evidence (LoE) and Grades of Recommendation (GoR) in HTA.

These methods should evolve from within the clinical community of orthopaedic and trauma surgeons to ensure acceptance. They also must be developed with professional methodological support to avoid scientific separatism.

The first step of the project was a thorough literature search to gather various definitions about utility and value in different disciplines, settings, populations and countries.

A key finding was the following simple but ubiquitous equation of value:


where V = value of a product or process, %I = relative importance of the new product or process (compared to an established one) and %C = its relative costs.

Another result was that potential innovations may simply be classified as: 1) radical (new biological principle), 2) substantial (improvement of established concepts), and 3) incremental (technical modification)5.

Table 1 summarizes the dimensions, categories and classifications of the preliminary METEOR matrix.


Need for a change in established principles

Grade of Innovation












Material / Surface


Implantation technique






Categorical / Likert-scale / continuous

Categorical / Likert-scale / continuous

Categorical / Likert-scale / continuous

Binary / categorical

Table 1.        Preliminary METEOR matrix to assess the value and utility of presumed innovations in orthopaedics.


  1. Barkun J.S., Aronson J.K., Feldman L.S. et al. Evaluation and stages of surgical innovations. Lancet 2009;374(9695):1089-1096.
  2. Ergina P.L., Cook J.A., Blazeby J.M. et al. Challenges in evaluating surgical innovation. Lancet 2009;374(9695):1097-1104.
  3. McCulloch P., Altman D.G., Campbell W.B. et al. No surgical innovation without evaluation: the IDEAL recommendations. Lancet 2009;374(9695):1105-1112.
  4. Albert H. Traktat über kritische Vernunft.  1991. Stuttgart, UTB.
  5. Coccia M. Classifications of Innovations. Survey and Future Directions. Ceris-Cnr Workpackage 2006.
Key Summary Points
  1. Innovation in surgery is a process that has recently been analyzed and formally described by the Balliol Group in the IDEAL statement (Innovation, Development Exploration, Assesment and Long-term Monitoring).
  2. The IDEAL statement may have limited application in bringing implants or devices to market.
  3. We propose METEOR (matrix to evaluate technology in orthopaedics) as an adjunct to IDEAL to assist orthopaedic surgeons, patients, producers and payers in appraising the need for a change in established principles, grade of innovation, value and safety of novel products and procedures.






Submit Community Content

If you have orthopedic information that you would like to share with the Orthogate Community, please register/login and submit your news, event, job, article, case or workshop from the Submit Content menu under the My Account area. Learn more!