The journey to reliable evidence: evaluating reliability of observational research findings via assessment of repeatability, reproducibility, replicability, generalizability, robustness, and calibration.
- Analytics use case(s): Estimation
- Study type: Methods Research, Clinical Application
- Tags: GLP1, DPP4, diabetes, CLRD, Athma, COPD
- Study lead: Mitch Conover
- Study lead forums tag: [conovermitch]
- Study start date: August 17, 2021
- Study end date: -
- Protocol: -
- Publications: -
- Results explorer: -
A high-quality observational comparative cohort study conducted using a nation-wide sample of U.S. administrative claims was recently published demonstrating a protective effect of Glucagon-like peptide 1 receptor agonists (GLP-1RA) medications on the risk of chronic lower respiratory disease (CLRD) exacerbations. As a methodologic experiment intended to address what has been described as a “replication/reproducibility” crisis in health research, we propose to independently reproduce the original study according to its description in the recent publication and supplemental materials, using the same database as was used in the original study. In addition to reproducing the study, we plan to evaluate the robustness of the study findings by conducting sensitivity analyses, assessing: 1) changes to definitions of exposure and outcome phenotypes, 2) use of alternate study designs, including the self-controlled cohort and self-controlled case series designs, 3) calibration of effect estimates using empirical null distributions, and 3) observable study diagnostics that inform the validity of a given analysis. Finally, we plan to explore the generalizability of the findings by executing the analysis on several study databases that vary with respect to the populations they include (e.g. U.S. and non-U.S.) and their mechanisms of data capture (e.g. administrative claims, electronic health records data).