Pragmatic Free Trial Meta
Pragmatic Free Trail Meta is an open data platform that allows research into pragmatic trials. It shares clean trial data and ratings using PRECIS-2 allowing for multiple and diverse meta-epidemiological studies that compare treatment effects estimates across trials with different levels of pragmatism and other design features.
Background
Pragmatic trials are becoming more widely recognized as providing real-world evidence for clinical decision making. The term "pragmatic", however, is used inconsistently and its definition and assessment require clarification. The purpose of pragmatic trials is to guide clinical practices and policy choices, rather than confirm a physiological hypothesis or clinical hypothesis. A pragmatic trial should try to be as close as possible to real-world clinical practices that include recruiting participants, setting up, implementation and delivery of interventions, determining and analysis results, as well as primary analyses. This is a key distinction from explanatory trials (as described by Schwartz and Lellouch1), which are designed to provide more thorough proof of a hypothesis.
Truly pragmatic trials should not blind participants or the clinicians. This could lead to an overestimation of the effects of treatment. Pragmatic trials should also seek to recruit patients from a wide range of health care settings, to ensure that their findings can be applied to the real world.
Finally, pragmatic trials should focus on outcomes that are vital to patients, like quality of life or functional recovery. This is particularly important when trials involve the use of invasive procedures or could have harmful adverse effects. The CRASH trial29 compared a two-page report with an electronic monitoring system for patients in hospitals with chronic cardiac failure. The catheter trial28 however was based on symptomatic catheter-related urinary tract infections as its primary outcome.
In addition to these features, pragmatic trials should minimize the trial's procedures and requirements for data collection to reduce costs. In the end, pragmatic trials should aim to make their findings as relevant to actual clinical practice as is possible. This can be achieved by ensuring their primary analysis is based on the intention to treat method (as described in CONSORT extensions).
Many RCTs that do not meet the criteria for pragmatism but have features that are contrary to pragmatism have been published in journals of varying types and incorrectly labeled pragmatic. This can lead to misleading claims of pragmatism, and the use of the term should be standardized. The creation of a PRECIS-2 tool that can provide a standardized objective evaluation of the pragmatic characteristics is a first step.
Methods
In a pragmatic research study the aim is to inform policy or clinical decisions by showing how an intervention could be integrated into routine care in real-world contexts. This is distinct from explanation trials that test hypotheses regarding the cause-effect connection in idealized situations. In this way, pragmatic trials may have less internal validity than studies that explain and are more susceptible to biases in their design analysis, conduct, and design. Despite these limitations, pragmatic trials can provide valuable information to decisions in the context of healthcare.
The PRECIS-2 tool measures the level of pragmatism that is present in an RCT by scoring it across 9 domains, ranging from 1 (very explanatory) to 5 (very pragmatic). In this study, the recruitment, organisation, flexibility: delivery and follow-up domains received high scores, but the primary outcome and the procedure for missing data were not at the practical limit. This suggests that it is possible to design a trial that has high-quality pragmatic features, without compromising the quality of its outcomes.
It is, however, difficult to assess how practical a particular trial is, since the pragmatism score is not a binary attribute; some aspects of a study can be more pragmatic than others. Additionally, logistical or protocol changes during the trial may alter its score in pragmatism. Additionally 36% of the 89 pragmatic trials discovered by Koppenaal and co. were placebo-controlled or conducted before licensing, and the majority were single-center. Therefore, they aren't as common and can only be described as pragmatic if their sponsors are tolerant of the lack of blinding in such trials.
Additionally, a typical feature of pragmatic trials is that researchers attempt to make their findings more meaningful by analysing subgroups of the trial sample. This can result in imbalanced analyses and lower statistical power. This increases the chance of omitting or ignoring differences in the primary outcomes. This was the case in the meta-analysis of pragmatic trials as secondary outcomes were not adjusted for differences in covariates at baseline.
Furthermore, pragmatic trials can also present challenges in the gathering and interpretation of safety data. This is due to the fact that adverse events are usually self-reported, and are prone to errors, delays or coding differences. It is essential to improve the quality and accuracy of the outcomes in these trials.
Results
Although the definition of pragmatism does not require that all clinical trials are 100% pragmatic there are benefits of including pragmatic elements in trials. These include:
Increasing sensitivity to real-world issues, reducing cost and size of the study, and enabling the trial results to be more quickly translated into actual clinical practice (by including patients from routine care). However, pragmatic trials have disadvantages. For example, the right type of heterogeneity can help a trial to generalise its results to different patients and settings; however the wrong type of heterogeneity could reduce assay sensitivity, and thus decrease the ability of a study to detect even minor effects of treatment.
A number of studies have attempted to categorize pragmatic trials with a variety of definitions and scoring systems. Schwartz and Lellouch1 developed a framework to discern between explanation-based studies that prove the physiological hypothesis or clinical hypothesis and pragmatic studies that guide the selection of appropriate therapies in the real-world clinical practice. The framework was comprised of nine domains, each scored on a scale of 1 to 5, with 1 indicating more lucid and 5 suggesting more pragmatic. The domains included recruitment, setting, intervention delivery, flexible adherence, follow-up and primary analysis.
The initial PRECIS tool3 included similar domains and a scale of 1 to 5. Koppenaal et al10 developed an adaptation of the assessment, known as the Pragmascope that was simpler to use for systematic reviews. They found that pragmatic reviews scored higher in most domains, but scored lower in the primary analysis domain.
This difference in primary analysis domains could be explained by the way most pragmatic trials analyse data. Certain explanatory trials however don't. The overall score for pragmatic systematic reviews was lower when the domains of management, flexible delivery and following-up were combined.
It is important to remember that a study that is pragmatic does not mean that a trial is of poor quality. In fact, there is a growing number of clinical trials that employ the term 'pragmatic' either in their title or abstract (as defined by MEDLINE but which is not precise nor sensitive). The use of these terms in abstracts and titles could indicate a greater understanding of the importance of pragmatism but it isn't clear if this is reflected in the contents of the articles.
Conclusions
In recent times, pragmatic trials are increasing in popularity in research because the value of real world evidence is increasingly recognized. They are randomized trials that evaluate real-world treatment options with experimental treatments in development. They involve patient populations that are more similar to those who receive treatment in regular care. This method can help overcome the limitations of observational studies, such as the biases that arise from relying on volunteers, and the limited availability and coding variability in national registries.
Other benefits of pragmatic trials include the ability to use existing data sources, as well as a higher chance of detecting meaningful changes than traditional trials. However, they may still have limitations that undermine their credibility and generalizability. Participation rates in some trials could be lower than anticipated because of the healthy-volunteering effect, financial incentives, or competition from other research studies. The need to recruit individuals in a timely manner also limits the sample size and the impact of many practical trials. Some pragmatic trials also lack controls to ensure that the observed differences aren't due to biases in the trial.
The authors of the Pragmatic Free Trial Meta identified RCTs published from 2022 to 2022 that self-described themselves as pragmatic. They assessed pragmatism by using the PRECIS-2 tool, which consists of the eligibility criteria for domains, recruitment, flexibility in intervention adherence, and follow-up. They found that 14 trials scored highly pragmatic or pragmatic (i.e. scoring 5 or higher) in at least one of these domains.
Read the Full Guide with a high pragmatism score tend to have more expansive eligibility criteria than traditional RCTs which have very specific criteria that are not likely to be used in the clinical environment, and they comprise patients from a wide range of hospitals. The authors argue that these traits can make the pragmatic trials more relevant and applicable to everyday practice, but they do not guarantee that a pragmatic trial is free of bias. The pragmatism characteristic is not a fixed characteristic; a pragmatic test that does not have all the characteristics of an explanatory study can still produce valid and useful outcomes.