Research Article - (2017) Volume 3, Issue 2
Rationale, aims and objectives: Comparative Effectiveness Research (CER) seeks to identify what health care interventions work best for improving health at both the individual and population level. The objective of this study was to determine whether published comparative effectiveness research studies adhere to accepted methodological principles.
Methods: Structured literature search of CER articles published in high-impact general medicine journals between 2009 and 2015, and assessment of their adherence to five methodological principles.
Results: 93 articles were retrieved from the search and 40 articles were finally selected. All of the studies included active comparators, 35% of the studies did not evaluate safety, 97% did not evaluate costs, 95% of the studies did not included patient perspectives, and 60% did not use any procedure to determine the heterogeneity of the response.
Conclusion: The sample of CER papers examined did not meet the recommended requisites for this type of studies. Our findings suggest that the majority of CER studies may not be useful to guide physicians, purchasers, and policy makers to make informed decisions that improve health care at both the individual and population levels.
Keywords: Health care; CER studies; Clinical setting; Heterogeneity; Medical interventions; Placebo; Safety analysis; Quality journals; Health outcomes
Comparative Effectiveness Research (CER) studies compare the relative effectiveness of different medical interventions in the diagnosis or treatment of a specific disease. In recent years, CER has garnered the advocacy and support of the US Institute of Medicine (IOM), numerous government healthcare agencies, and international organizations [1-3]. According to the formal definition by the IOM, CER is “the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor or improve the delivery of care”. CER has the potential of providing information “to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels” [4]. The two main characteristic aspects of CER studies are the comparison of active treatments (active comparator studies) rather than the comparison of an active treatment with a placebo or the absence of a comparator, and the evaluation of the effects of interventions in a real-world setting. CER studies should contrast alternatives that are already used in the clinical setting but for which effectiveness data are not available or are insufficient to demonstrate superiority over other treatments [5-7]. The comparison interventions in CER may include medications, procedures, medical and assistive devices and technologies, behavioural change strategies, and delivery systems.
Apart from these two characteristics, the definition of CER suggests that the study of effectiveness cannot be the only objective of the programmes. Aspects such as safety and cost are relevant elements as well [7]. Additionally, CER studies should not only assess the best treatment option for the average patient but also analyse the best option for each individual patient [8]. Since the available evidence may be incomplete or does not apply to certain patient populations, such as children or the elderly, organisations such as the Patient-Centred Outcomes Research Institute (PCORI) promotes CER studies to generate new evidence in different health care interventions and across a broad range of medical conditions and patient populations [9,10].
Despite the extensive amount of CER literature, little is known about the characteristics of published CER studies and how they fit into the CER concept. Therefore, the goal of this work was to characterize relevant CER studies to analyse if they met comparative effectiveness major objectives.
The MEDLINE and EMBASE databases were searched using the search terms “Comparative AND effectiveness (Title) AND ("2009/05/18" (PDat): "2015/05/18" (PDat))”. All original articles with a title containing the words "comparative" and "effectiveness" were selected. The search was limited to articles published between 2009 and 2015 in general-medicine journals with the highest 2013 impact factor (NEJM, IF: 54.42; Lancet, IF: 39.20; JAMA-J Am Med Assoc, IF: 30.38; Brit Med J, IF: 16.37; Ann Intern Med, IF: 16.10; PLOS Med, IF: 14.00; Arch Intern Med, IF: 13.24; BMC Med, IF: 7.27; Cochrane DB Syst Rev, IF: 5.93). After the search and the elimination of duplicates, 93 articles were obtained. These articles were manually reviewed to eliminate Opinions and Editorials. Finally, 40 articles met the requirements. These articles were double-checked by two independent investigators to extract the relevant data included in this analysis. Disagreement was resolved by consensus.
The following variables were analysed in each of the selected articles: use and type of treatment (active or placebo), safety as primary or secondary objective, cost assessment, inclusion of Patient Reported Outcomes (PROs), analysis of the heterogeneity in the response, type of intervention evaluated, study design, therapeutic area, and funding source. The variables were analysed using descriptive statistics.
Study characteristics of the sample of 40 CER publications included in this review are presented in Table 1. Of these, 25 were comparative studies of medicines and 15 of other types of medical intervention. Independently of the intervention evaluated, and in relation to the most important features of CER studies, we found that active comparators were used in all of the studies, and 20% of them also used a placebo. 62.5% described comparisons between medications, while 37.5% studied other types of medical interventions (e.g., behavioural, surgical devices, diagnostic strategies).
| Medications (N=25) | Other interventions (N=15) | Total (N=40) | |
|---|---|---|---|
| Reference number | [29-53] | [54-68] | - |
| Active vs. inactive comparators | |||
| Active | 25 | 15 | 40 (100%) |
| Placebo | 8 | 0 | 8 (20%) |
| Type of comparator | |||
| Different drugs | 24 | 2 | 26 (65%) |
| Non-pharmacological interventions | 3 | 10 | 13 (33%) |
| Different strategies | 1 | 4 | 5 (13%) |
| Safety (adverse effects) | |||
| Yes (primary or secondary) | 17 | 9 | 26 (65%) |
| No | 8 | 6 | 14 (35%) |
| Cost analysis | |||
| Yes | 1 | 0 | 1 (3%) |
| No | 24 | 15 | 39 (97%) |
| Inclusion of PROs | |||
| Yes | 2 | 0 | 2 (5%) |
| No | 23 | 15 | 38 (95%) |
| Assess the heterogeneity in the response | |||
| Yes | 10 | 6 | 16 (40%) |
| Subgroup analysis | 9 | 5 | 14 |
| Propensity score | 4 | 4 | 8 |
| Risk adjustment | 0 | 0 | 0 |
| No | 15 | 9 | 24 (60%) |
| Study design | |||
| Randomised clinical trials (RCT) | 2 | 4 | 6 (15%) |
| Observational (prospective or retrospective) | 4 | 5 | 9 (23%) |
| Meta-analysis/Systematic review | 17 | 5 | 22 (55%) |
| Model | 2 | 1 | 3 (8%) |
| Therapeutic area | |||
| Pain | 1 | 0 | 1 |
| Cardiovascular | 7 | 6 | 13 |
| Infectious | 4 | 0 | 4 |
| Arthritis | 1 | 0 | 1 |
| Diabetes | 4 | 3 | 7 |
| Glaucoma | 1 | 0 | 1 |
| Osteoporosis | 1 | 0 | 1 |
| Chronic obstructive pulmonary disease | 1 | 0 | 1 |
| Central nervous system | 2 | 1 | 3 |
| Oncology | 2 | 3 | 5 |
| Others | 1 | 2 | 3 |
| Funding | |||
| Public | 18 | 12 | 30 (75%) |
| Private | 7 | 3 | 10 (25%) |
Table 1: Characteristics of the CER studies analysed.
The safety of the interventions was evaluated in 65% of the studies as primary or secondary objective. Cost analyses were performed in 3% of the papers. Respect to the "patient centricity” of the studies, PROs were included in 5% of the publications and 40% of them used some procedure to determine the heterogeneity of the response.
The most common CERs were meta-analyses or systematic reviews (55%), followed by observational studies (23%) and Randomized Clinical Trials (RCTs) (15%). With regard to the source of funding, 75% of the publications received public support. In our sample, the most represented medical field is the cardiovascular area, followed by diabetes.
The results of this study show that the CER articles analysed did not meet the criteria commonly accepted for CER studies. Although all papers included active-drug comparators, 20% also included comparisons with placebo. As previously stated, comparison groups in CER studies should reflect clinical choices in real world practice. The comparison of interventions with a clinically meaningful alternative is usually a better choice from a methodological perspective than the comparison with an untreated group. The option of “no treatment” may not meet usual standards of care where multiple therapeutic options are available and should be limited only to certain clinical situations. Our finding about the use of placebo in CER studies can be partially explained by the methodology used in many of the studies analysed, which were meta-analyses and systematic reviews that included RTCs with different treatment comparison arms including placebo.
Safety analysis was included in 65% of the publications. Since the ultimate goal of CER is to provide decision makers with accurate and scientifically-rigorous information for comparing alternative clinical options, CER studies should systematically include not just the effectiveness but also an evaluation of safety. We observe an imbalance in the measuring of the treatment efficacy over treatment risk and safety. Although the methodology to assess the safety of medical interventions is less standardized than the assessment of their efficacy, we believe than CER studies should focus on the assessment of the risk-benefit of the compared interventions [11].
Regarding cost comparisons, an economic analysis of the alternatives was performed in only 3% of the studies. Although CER should allow making clinical decisions in an environment of limited resources, this is a controversial point with which not all stakeholders agree [12]. According to the detractors, if economic considerations were included, there would be many interventions with small but positive health benefits that would not be compensated by the additional cost and that, therefore, would not be recommended. However, as some medical organizations have recently highlighted, performing Cost-Effectiveness Analysis (CEA) is important to compare and contextualize the price of any intervention [12-14].
With regard to patients’ perspectives, 5% of the analysed studies collected PROs. A fully informative CER should incorporate patients’ perspectives standardised in the form of PROs [15]. Patients’ perspectives can generate valuable information for clinical health professionals in their decision-making processes [16-18]. The importance of incorporating PROs has been emphasised by regulatory (i.e., the U.S. Food and Drug Administration) and National Health Agencies (i.e., the U.S. National Institutes of Health) [19] as well as by organisations such as PCORI, with the objective of developing patientcentred medicine [3]. However, our analysis highlights the fact that few studies take patient perspectives into consideration.
Regarding the assessment of the heterogeneity of the response in patient subgroups, the goal of CER is to identify interventions that work best in specific subtypes of patients. The majority of the studies included in this study (60%) did not use any procedures directed at identifying uniqueness in the response of patient subgroups. In order to increase the generalizability of the results, CER studies usually use wide selection criteria. However, it is increasingly evident that although most therapeutic decisions rely on information obtained based on the average patient, the one-size-fits-all approach is not the ideal to optimise healthcare as it does not cover the singularities of the individual patient [20,21]. CER programmes should include patientoriented research methods to assess the responses in individuals and subgroups with the objective of "particularising" the results [8,22-24]. Therefore, current research falls short of the ambitious goal of improving “health care at both the individual and population levels”, as stated in the CER definition.
Although pragmatic RCTs could be considered as an adequate design for the evaluation of CER, other designs such as observational studies or mixed treatment comparisons (MTCs) [25], systematic reviews, meta-analyses, or decision analysis models [26] are also recommended. In our review, most of the CER studies were systematic reviews or meta-analyses (55%), followed by observational studies (23%) and clinical trials (15%) (Of which half were pragmatic RCTs). The low number of pragmatic RCTs agrees with the results of other reviews [27,28]. The implementation of CER based on pragmatic RCTs and observational studies should be encouraged.
This study has a number of limitations. The most important one is the potential information bias caused by the inclusion of only highimpact- factor general medicine journals. Our search selected only those articles that included the words "comparative" and "effectiveness" in the title. However, the biases introduced by the selection of journals and articles support our hypothesis because by including CER studies of high quality, one would expect that they would better comply with the theoretical requirements of these studies. It is foreseeable that the identified problems will be even greater with less stringent criteria, including lower quality journals and articles that do not include the CER concept in their titles.
Our study shows that most of the CERs analysed did not adhere to the five basic requirements that these studies should fulfil. Among other issues, they often included comparisons with placebo, did not evaluate the safety or the cost of the interventions, did not usually include PROs, and did not take into account heterogeneity in the response. In our opinion, it is necessary to encourage true CER studies primarily based on pragmatic RCTs or observational studies carried out under conditions of actual clinical practise. Only in this way it will be possible to achieve higher quality healthcare systems capable of improving the health outcomes of individual patients.
