Publications
Emulation of Randomized Clinical Trials With Nonrandomized Database Analyses: Results of 32 Clinical Trials.
Shirley V Wang, Sebastian Schneeweiss ; RCT-DUPLICATE Initiative
JAMA. 2023 Apr 25. doi: 10.1001/jama.2023.4221
Importance: Nonrandomized studies using insurance claims databases can be analyzed to produce real-world evidence on the effectiveness of medical products. Given the lack of baseline randomization and measurement issues, concerns exist about whether such studies produce unbiased treatment effect estimates.
Objective: To emulate the design of 30 completed and 2 ongoing randomized clinical trials (RCTs) of medications with database studies using observational analogues of the RCT design parameters (population, intervention, comparator, outcome, time [PICOT]) and to quantify agreement in RCT-database study pairs.
Design, setting, and participants: New-user cohort studies with propensity score matching using 3 US claims databases (Optum Clinformatics, MarketScan, and Medicare). Inclusion-exclusion criteria for each database study were prespecified to emulate the corresponding RCT. RCTs were explicitly selected based on feasibility, including power, key confounders, and end points more likely to be emulated with real-world data. All 32 protocols were registered on ClinicalTrials.gov before conducting analyses. Emulations were conducted from 2017 through 2022.
Exposures: Therapies for multiple clinical conditions were included.
Main outcomes and measures: Database study emulations focused on the primary outcome of the corresponding RCT. Findings of database studies were compared with RCTs using predefined metrics, including Pearson correlation coefficients and binary metrics based on statistical significance agreement, estimate agreement, and standardized difference.
Results: In these highly selected RCTs, the overall observed agreement between the RCT and the database emulation results was a Pearson correlation of 0.82 (95% CI, 0.64-0.91), with 75% meeting statistical significance, 66% estimate agreement, and 75% standardized difference agreement. In a post hoc analysis limited to 16 RCTs with closer emulation of trial design and measurements, concordance was higher (Pearson r, 0.93; 95% CI, 0.79-0.97; 94% meeting statistical significance, 88% estimate agreement, 88% standardized difference agreement). Weaker concordance occurred among 16 RCTs for which close emulation of certain design elements that define the research question (PICOT) with data from insurance claims was not possible (Pearson r, 0.53; 95% CI, 0.00-0.83; 56% meeting statistical significance, 50% estimate agreement, 69% standardized difference agreement).
Conclusions and relevance: Real-world evidence studies can reach similar conclusions as RCTs when design and measurements can be closely emulated, but this may be difficult to achieve. Concordance in results varied depending on the agreement metric. Emulation differences, chance, and residual confounding can contribute to divergence in results and are difficult to disentangle.
Shirley V Wang, Sebastian Schneeweiss ; RCT-DUPLICATE Initiative
JAMA. 2023 Apr 25. doi: 10.1001/jama.2023.4221
Importance: Nonrandomized studies using insurance claims databases can be analyzed to produce real-world evidence on the effectiveness of medical products. Given the lack of baseline randomization and measurement issues, concerns exist about whether such studies produce unbiased treatment effect estimates.
Objective: To emulate the design of 30 completed and 2 ongoing randomized clinical trials (RCTs) of medications with database studies using observational analogues of the RCT design parameters (population, intervention, comparator, outcome, time [PICOT]) and to quantify agreement in RCT-database study pairs.
Design, setting, and participants: New-user cohort studies with propensity score matching using 3 US claims databases (Optum Clinformatics, MarketScan, and Medicare). Inclusion-exclusion criteria for each database study were prespecified to emulate the corresponding RCT. RCTs were explicitly selected based on feasibility, including power, key confounders, and end points more likely to be emulated with real-world data. All 32 protocols were registered on ClinicalTrials.gov before conducting analyses. Emulations were conducted from 2017 through 2022.
Exposures: Therapies for multiple clinical conditions were included.
Main outcomes and measures: Database study emulations focused on the primary outcome of the corresponding RCT. Findings of database studies were compared with RCTs using predefined metrics, including Pearson correlation coefficients and binary metrics based on statistical significance agreement, estimate agreement, and standardized difference.
Results: In these highly selected RCTs, the overall observed agreement between the RCT and the database emulation results was a Pearson correlation of 0.82 (95% CI, 0.64-0.91), with 75% meeting statistical significance, 66% estimate agreement, and 75% standardized difference agreement. In a post hoc analysis limited to 16 RCTs with closer emulation of trial design and measurements, concordance was higher (Pearson r, 0.93; 95% CI, 0.79-0.97; 94% meeting statistical significance, 88% estimate agreement, 88% standardized difference agreement). Weaker concordance occurred among 16 RCTs for which close emulation of certain design elements that define the research question (PICOT) with data from insurance claims was not possible (Pearson r, 0.53; 95% CI, 0.00-0.83; 56% meeting statistical significance, 50% estimate agreement, 69% standardized difference agreement).
Conclusions and relevance: Real-world evidence studies can reach similar conclusions as RCTs when design and measurements can be closely emulated, but this may be difficult to achieve. Concordance in results varied depending on the agreement metric. Emulation differences, chance, and residual confounding can contribute to divergence in results and are difficult to disentangle.
Emulating Randomized Clinical Trials with Nonrandomized Real-World Evidence Studies: First Results from the RCT DUPLICATE Initiative.
Jessica M. Franklin, Elisabetta Patorno, Rishi J. Desai, Robert J. Glynn, David Martin, Kenneth Quinto, Ajinkya Pawar, Lily G. Bessette, Hemin Lee, Elizabeth M. Garry, Nileesa Gautam, and Sebastian Schneeweiss
Circulation. 2020 Dec 17. doi: 10.1161/CIRCULATIONAHA.120.051718
Background: Regulators are evaluating the use of non-interventional real-world evidence (RWE) studies to assess the effectiveness of medical products. The RCT-DUPLICATE initiative uses a structured process to design RWE studies emulating randomized controlled trials (RCTs) and compare results. Here, we report findings of the first 10 trial emulations, evaluating cardiovascular outcomes of antidiabetic or antiplatelet medications.
Methods: We selected 3 active-controlled and 7 placebo-controlled RCTs for replication. Using patient-level claims data from US commercial and Medicare payers, we implemented inclusion/exclusion criteria, selected primary endpoints, and comparator populations to emulate those of each corresponding RCT. Within the trial-mimicking populations, we conducted propensity score matching to control for >120 pre-exposure confounders. All study parameters were prospectively defined and protocols registered before hazard ratios (HRs) and 95% confidence intervals (CIs) were computed. Success criteria for the primary analysis were pre-specified for each replication.
Results: Despite attempts to emulate RCT design as closely as possible, differences between the RCT and corresponding RWE study populations remained. The regulatory conclusions were equivalent in 6 of 10. The RWE emulations achieved a HR estimate that was within the 95% CI from the corresponding RCT in 8 of 10 studies. In 9 of 10, either the regulatory or estimate agreement success criteria were fulfilled. The largest differences in effect estimates were found for RCTs where second-generation sulfonylureas were used as a proxy for placebo regarding cardiovascular effects. Nine of 10 replications had a standardized difference between effect estimates of <2, which suggests differences within expected random variation.
Conclusions: Agreement between RCT and RWE findings varies depending on which agreement metric is used. Interim findings indicate that selection of active comparator therapies with similar indications and use patterns enhances the validity of RWE. Even in the context of active comparators, concordance between RCT and RWE findings is not guaranteed, partially because trials are not emulated exactly. More trial emulations are needed to understand how often and in what contexts RWE findings match RCTs.
Emulation Differences vs. Biases When Calibrating Real‐World Evidence Findings Against Randomized Controlled Trials.
Jessica M. Franklin, Robert J. Glynn, Samy Suissa, Sebastian Schneeweiss
Clinical Pharmacol Ther. 2020 Feb 12. doi: 10.1002/cpt.1793
Actionable real-world evidence (RWE) requires accurate estimation of causal treatment effects. Calibration of RWE against randomized controlled trials (RCTs) is sometimes done to demonstrate that RWE can support the same causal conclusion as RCTs. Disagreements can occur when studies in each pair asked different questions in different populations or due to the presence of residual bias. Distinguishing among reasons for differences will impact the level of confidence in RWE.
Jessica M. Franklin, Robert J. Glynn, Samy Suissa, Sebastian Schneeweiss
Clinical Pharmacol Ther. 2020 Feb 12. doi: 10.1002/cpt.1793
Actionable real-world evidence (RWE) requires accurate estimation of causal treatment effects. Calibration of RWE against randomized controlled trials (RCTs) is sometimes done to demonstrate that RWE can support the same causal conclusion as RCTs. Disagreements can occur when studies in each pair asked different questions in different populations or due to the presence of residual bias. Distinguishing among reasons for differences will impact the level of confidence in RWE.
Nonrandomized Real-World Evidence to Support Regulatory Decision Making: Process for a Randomized Trial Replication Project.
Jessica M. Franklin, Ajinkya Pawar, David Martin, Robert J. Glynn, Mark Levenson, Robert Temple and Sebastian Schneeweiss
Clinical Pharmacol Ther. 2019 Sept 21. doi: 10.1002/cpt.1351.
Recent legislation mandates that the US Food and Drug Administration issue guidance regarding when real‐world evidence (RWE) could be used to support regulatory decision making. Although RWE could come from randomized or nonrandomized designs, there are significant concerns about the validity of RWE assessing medication effectiveness based on nonrandomized designs. We propose an initiative using healthcare claims data to assess the ability of nonrandomized RWE to provide results that are comparable with those from randomized controlled trials (RCTs). We selected 40 RCTs, and we estimate that approximately 30 attempted replications will be completed after feasibility analyses. We designed an implementation process to ensure that each attempted replication is consistent, transparent, and reproducible. This initiative is the first to systematically evaluate the ability of nonrandomized RWE to replicate multiple RCTs using a structured process. Results from this study should provide insight on the strengths and limitations of using nonrandomized RWE from claims for regulatory decision making.
Jessica M. Franklin, Ajinkya Pawar, David Martin, Robert J. Glynn, Mark Levenson, Robert Temple and Sebastian Schneeweiss
Clinical Pharmacol Ther. 2019 Sept 21. doi: 10.1002/cpt.1351.
Recent legislation mandates that the US Food and Drug Administration issue guidance regarding when real‐world evidence (RWE) could be used to support regulatory decision making. Although RWE could come from randomized or nonrandomized designs, there are significant concerns about the validity of RWE assessing medication effectiveness based on nonrandomized designs. We propose an initiative using healthcare claims data to assess the ability of nonrandomized RWE to provide results that are comparable with those from randomized controlled trials (RCTs). We selected 40 RCTs, and we estimate that approximately 30 attempted replications will be completed after feasibility analyses. We designed an implementation process to ensure that each attempted replication is consistent, transparent, and reproducible. This initiative is the first to systematically evaluate the ability of nonrandomized RWE to replicate multiple RCTs using a structured process. Results from this study should provide insight on the strengths and limitations of using nonrandomized RWE from claims for regulatory decision making.
Evaluating the Use of Nonrandomized Real-World Data Analyses for Regulatory Decision Making.
Jessica M. Franklin, Robert J Glynn, David Martin, Sebastian Schneeweiss
Clin Pharmacol Ther. 2019 Jan 13. doi: 10.1002/cpt.1351.
The analysis of longitudinal healthcare data outside of highly controlled parallel-group randomized trials, termed real-world evidence (RWE), has received increasing attention in the medical literature. In this paper, we discuss the potential role of RWE in drug regulation with a focus on the analysis of healthcare databases. We present several cases in which RWE is already used and cases in which RWE could potentially support regulatory decision making. We summarize key issues that investigators and regulators should consider when designing or evaluating such studies, and we propose a structured process for implementing analyses that facilitates regulatory review. We evaluate the empirical evidence base supporting the validity, transparency, and reproducibility of RWE from analysis of healthcare databases and discuss the work that still needs to be done to ensure that such analyses can provide decision-ready evidence on the effectiveness and safety of treatments.
Jessica M. Franklin, Robert J Glynn, David Martin, Sebastian Schneeweiss
Clin Pharmacol Ther. 2019 Jan 13. doi: 10.1002/cpt.1351.
The analysis of longitudinal healthcare data outside of highly controlled parallel-group randomized trials, termed real-world evidence (RWE), has received increasing attention in the medical literature. In this paper, we discuss the potential role of RWE in drug regulation with a focus on the analysis of healthcare databases. We present several cases in which RWE is already used and cases in which RWE could potentially support regulatory decision making. We summarize key issues that investigators and regulators should consider when designing or evaluating such studies, and we propose a structured process for implementing analyses that facilitates regulatory review. We evaluate the empirical evidence base supporting the validity, transparency, and reproducibility of RWE from analysis of healthcare databases and discuss the work that still needs to be done to ensure that such analyses can provide decision-ready evidence on the effectiveness and safety of treatments.
Use of Health Care Databases to Support Supplemental Indications of Approved Medications
Michael Fralick, MD; Aaron S. Kesselheim, MD, JD, MPH; Jerry Avorn, MD; Sebastian Schneeweiss, MD, ScD JAMA Intern Med; 2018 Jan 1; 178(1):55-63
Manufacturers of US Food and Drug Administration–approved prescription drugs often apply for additional indications based on randomized clinical trials. Real-world database analyses on a medication’s use and outcomes in routine settings of care might help to inform decision making regarding such supplemental indications.
Michael Fralick, MD; Aaron S. Kesselheim, MD, JD, MPH; Jerry Avorn, MD; Sebastian Schneeweiss, MD, ScD JAMA Intern Med; 2018 Jan 1; 178(1):55-63
Manufacturers of US Food and Drug Administration–approved prescription drugs often apply for additional indications based on randomized clinical trials. Real-world database analyses on a medication’s use and outcomes in routine settings of care might help to inform decision making regarding such supplemental indications.
Synergies From Integrating Randomized Control Trials and Real-World Data Analyses
Mehdi Najafzadeh, Joshua J. Gagne and Sebastian Schneeweiss Clin Pharmacol Ther 2017 Dec;102(6):914-916
Analyses using administrative claims databases or national registries provide estimates of benefits and harms of medications in real-world settings for large and diverse patient populations. Whereas claims-based nonrandomized studies and randomized-controlled trials (RCTs) have distinct limitations, their strengths are complementary. Integrating RCT and claims data offers substantial synergies. We propose obtaining routinely collected longitudinal claims data from RCT participants and discuss the added value of the novel evidence that can be derived from this "information overlap."
Mehdi Najafzadeh, Joshua J. Gagne and Sebastian Schneeweiss Clin Pharmacol Ther 2017 Dec;102(6):914-916
Analyses using administrative claims databases or national registries provide estimates of benefits and harms of medications in real-world settings for large and diverse patient populations. Whereas claims-based nonrandomized studies and randomized-controlled trials (RCTs) have distinct limitations, their strengths are complementary. Integrating RCT and claims data offers substantial synergies. We propose obtaining routinely collected longitudinal claims data from RCT participants and discuss the added value of the novel evidence that can be derived from this "information overlap."
When and How Can Real World Data Analyses Substitute for Randomized Controlled Trials?
Franklin JM and Schneeweiss S. Clin Pharm & Ther 2017 Dec; 102(6): 924-933
Regulators consider randomized controlled trials (RCTs) as the gold standard for evaluating the safety and effectiveness of medications, but their costs, duration, and limited generalizability have caused some to look for alternatives. Real world evidence based on data collected outside of RCTs, such as registries and longitudinal healthcare databases, can sometimes substitute for RCTs, but concerns about validity have limited their impact. Greater reliance on such real world data (RWD) in regulatory decision making requires understanding why some studies fail while others succeed in producing results similar to RCTs. Key questions when considering whether RWD analyses can substitute for RCTs for regulatory decision making are WHEN one can study drug effects without randomization and HOW to implement a valid RWD analysis if one has decided to pursue that option. The WHEN is primarily driven by externalities not controlled by investigators, whereas the HOW is focused on avoiding known mistakes in RWD analyses.
Franklin JM and Schneeweiss S. Clin Pharm & Ther 2017 Dec; 102(6): 924-933
Regulators consider randomized controlled trials (RCTs) as the gold standard for evaluating the safety and effectiveness of medications, but their costs, duration, and limited generalizability have caused some to look for alternatives. Real world evidence based on data collected outside of RCTs, such as registries and longitudinal healthcare databases, can sometimes substitute for RCTs, but concerns about validity have limited their impact. Greater reliance on such real world data (RWD) in regulatory decision making requires understanding why some studies fail while others succeed in producing results similar to RCTs. Key questions when considering whether RWD analyses can substitute for RCTs for regulatory decision making are WHEN one can study drug effects without randomization and HOW to implement a valid RWD analysis if one has decided to pursue that option. The WHEN is primarily driven by externalities not controlled by investigators, whereas the HOW is focused on avoiding known mistakes in RWD analyses.
Comparing the performance of Propensity Score Methods in Healthcare Database Studies with Rare Outcomes
Jessica M. Franklin, Wesley Eddings, Peter C. Austin, Elizabeth A. Stuart, and Sebastian Schneeweiss Stat Med 2017 May 30; 36 (12); 1946-1963
Nonrandomized studies of treatments from electronic healthcare databases are critical for producing the evidence necessary to making informed treatment decisions, but often rely on comparing rates of events observed in a small number of patients. In addition, studies constructed from electronic healthcare databases, for example, administrative claims data, often adjust for many, possibly hundreds, of potential confounders. Despite the importance of maximizing effciency when there are many confounders and few observed outcome events, there has been relatively little research on the relative performance of different propensity score methods in this context. In this paper, we compare a wide variety of propensity-based estimators of the marginal relative risk. In contrast to prior research that has focused on specifc statistical methods in isolation of other analytic choices, we instead consider a method to be defined by the complete multistep process from propensity score modeling to fnal treatment effect estimation. Propensity score model estimation methods considered include ordinary logistic regression, Bayesian logistic regression, lasso, and boosted regression trees. Methods for utilizing the propensity score include pair matching, full matching, decile strata, fne strata, regression adjustment using one or two nonlinear splines, inverse propensity weighting, and matching weights. We evaluate methods via a ‘plasmode’ simulation study, which creates simulated datasets on the basis of a real cohort study of two treatments constructed from administrative claims data. Our results suggest that regression adjustment and matching weights, regardless of the propensity score model estimation method, provide lower bias and mean squared error in the context of rare binary outcomes
Jessica M. Franklin, Wesley Eddings, Peter C. Austin, Elizabeth A. Stuart, and Sebastian Schneeweiss Stat Med 2017 May 30; 36 (12); 1946-1963
Nonrandomized studies of treatments from electronic healthcare databases are critical for producing the evidence necessary to making informed treatment decisions, but often rely on comparing rates of events observed in a small number of patients. In addition, studies constructed from electronic healthcare databases, for example, administrative claims data, often adjust for many, possibly hundreds, of potential confounders. Despite the importance of maximizing effciency when there are many confounders and few observed outcome events, there has been relatively little research on the relative performance of different propensity score methods in this context. In this paper, we compare a wide variety of propensity-based estimators of the marginal relative risk. In contrast to prior research that has focused on specifc statistical methods in isolation of other analytic choices, we instead consider a method to be defined by the complete multistep process from propensity score modeling to fnal treatment effect estimation. Propensity score model estimation methods considered include ordinary logistic regression, Bayesian logistic regression, lasso, and boosted regression trees. Methods for utilizing the propensity score include pair matching, full matching, decile strata, fne strata, regression adjustment using one or two nonlinear splines, inverse propensity weighting, and matching weights. We evaluate methods via a ‘plasmode’ simulation study, which creates simulated datasets on the basis of a real cohort study of two treatments constructed from administrative claims data. Our results suggest that regression adjustment and matching weights, regardless of the propensity score model estimation method, provide lower bias and mean squared error in the context of rare binary outcomes
A Bias in the Evaluation of Bias Comparing Randomized Trials with Nonexpiramental Studies
Jessica M. Franklin, Sara Dejene, Krista F Huybrechts, Shirley V Wang, Martin Kulldorff and Kenneth J Rothman Epidemiol Methods 2017 Apr 22;6(1)
In a recent BMJ article, the authors conducted a meta-analysis to compare estimated treatment effects from randomized trials with those derived from observational studies based on routinely collected data (RCD). They calculated a pooled relative odds ratio (ROR) of 1.31 (95 % confidence interval [CI]: 1.03– 1.65) and concluded that RCD studies systematically over-estimated protective effects. However, their metaanalysis inverted results for some clinical questions to force all estimates from RCD to be below 1. We evaluated the statistical properties of this pooled ROR, and found that the selective inversion rule employed in the original meta-analysis can positively bias the estimate of the ROR. We then repeated the random effects meta-analysis using a different inversion rule and found an estimated ROR of 0.98 (0.78–1.23), indicating the ROR is highly dependent on the direction of comparisons. As an alternative to the ROR, we calculated the observed proportion of clinical questions where the RCD and trial CIs overlap, as well as the expected proportion assuming no systematic difference between the studies. Out of 16 clinical questions, 50 % CIs overlapped for 8 (50 %; 25 to 75 %) compared with an expected overlap of 60 % assuming no systematic difference between RCD studies and trials. Thus, there was little evidence of a systematic difference in effect estimates between RCD and RCTs. Estimates of pooled RORs across distinct clinical questions are generally not interpretable and may be misleading.
Jessica M. Franklin, Sara Dejene, Krista F Huybrechts, Shirley V Wang, Martin Kulldorff and Kenneth J Rothman Epidemiol Methods 2017 Apr 22;6(1)
In a recent BMJ article, the authors conducted a meta-analysis to compare estimated treatment effects from randomized trials with those derived from observational studies based on routinely collected data (RCD). They calculated a pooled relative odds ratio (ROR) of 1.31 (95 % confidence interval [CI]: 1.03– 1.65) and concluded that RCD studies systematically over-estimated protective effects. However, their metaanalysis inverted results for some clinical questions to force all estimates from RCD to be below 1. We evaluated the statistical properties of this pooled ROR, and found that the selective inversion rule employed in the original meta-analysis can positively bias the estimate of the ROR. We then repeated the random effects meta-analysis using a different inversion rule and found an estimated ROR of 0.98 (0.78–1.23), indicating the ROR is highly dependent on the direction of comparisons. As an alternative to the ROR, we calculated the observed proportion of clinical questions where the RCD and trial CIs overlap, as well as the expected proportion assuming no systematic difference between the studies. Out of 16 clinical questions, 50 % CIs overlapped for 8 (50 %; 25 to 75 %) compared with an expected overlap of 60 % assuming no systematic difference between RCD studies and trials. Thus, there was little evidence of a systematic difference in effect estimates between RCD and RCTs. Estimates of pooled RORs across distinct clinical questions are generally not interpretable and may be misleading.
From Trial to Target Populations — Calibrating Real-World Data
Mehdi Najafzadeh, Sebastian Schneeweiss N Engl J Med 376.13 (2017): 1203-5
Mehdi Najafzadeh, Sebastian Schneeweiss N Engl J Med 376.13 (2017): 1203-5
Improving Therapeutic Effectiveness and Safety Through Big Healthcare Data
S Schneeweiss Clin Pharmacol Ther 2016 Feb 17; 99(3);262-5
Big healthcare data—electronically recorded longitudinal data generated during the provision and administration of healthcare for millions of patients—have become essential for understanding the effectiveness and safety of therapeutics. They are most effectively used in concert with experimental and laboratory research throughout the life cycle of a drug. Applications range from providing phenotype and health outcomes information in genomewide association studies to postmarketing studies that assure prescribers of the safety of approved drugs
S Schneeweiss Clin Pharmacol Ther 2016 Feb 17; 99(3);262-5
Big healthcare data—electronically recorded longitudinal data generated during the provision and administration of healthcare for millions of patients—have become essential for understanding the effectiveness and safety of therapeutics. They are most effectively used in concert with experimental and laboratory research throughout the life cycle of a drug. Applications range from providing phenotype and health outcomes information in genomewide association studies to postmarketing studies that assure prescribers of the safety of approved drugs
Real World Data in Adaptive Biomedical Innovation: A Framework for Generating Evidence Fit for Decision-Making
S Schneeweiss, H-G Eichler, A Garcia-Altes, C Chinn, A-V Eggimann, S Garner, W Goettsch, R Lim ,W Lobker,D Martin ,T Muller, BJ Park, R Platt, S Priddy, M Ruhl, A Spooner, B Vannieuwenhuyse and RJ Willke Clin Pharm and Ther 2016 Dec; 100(6):633-64
Analyses of healthcare databases (claims, electronic health records [EHRs]) are useful supplements to clinical trials for generating evidence on the effectiveness, harm, use, and value of medical products in routine care. A constant stream of data from the routine operation of modern healthcare systems, which can be analyzed in rapid cycles, enables incremental evidence development to support accelerated and appropriate access to innovative medicines. Evidentiary needs by regulators, Health Technology Assessment, payers, clinicians, and patients after marketing authorization comprise (1) monitoring of medication performance in routine care, including the materialized effectiveness, harm, and value; (2) identifying new patient strata with added value or unacceptable harms; and (3) monitoring targeted utilization. Adaptive biomedical innovation (ABI) with rapid cycle database analytics is successfully enabled if evidence is meaningful, valid, expedited, and transparent. These principles will bring rigor and credibility to current efforts to increase research efficiency while upholding evidentiary standards required for effective decision-making in healthcare.
S Schneeweiss, H-G Eichler, A Garcia-Altes, C Chinn, A-V Eggimann, S Garner, W Goettsch, R Lim ,W Lobker,D Martin ,T Muller, BJ Park, R Platt, S Priddy, M Ruhl, A Spooner, B Vannieuwenhuyse and RJ Willke Clin Pharm and Ther 2016 Dec; 100(6):633-64
Analyses of healthcare databases (claims, electronic health records [EHRs]) are useful supplements to clinical trials for generating evidence on the effectiveness, harm, use, and value of medical products in routine care. A constant stream of data from the routine operation of modern healthcare systems, which can be analyzed in rapid cycles, enables incremental evidence development to support accelerated and appropriate access to innovative medicines. Evidentiary needs by regulators, Health Technology Assessment, payers, clinicians, and patients after marketing authorization comprise (1) monitoring of medication performance in routine care, including the materialized effectiveness, harm, and value; (2) identifying new patient strata with added value or unacceptable harms; and (3) monitoring targeted utilization. Adaptive biomedical innovation (ABI) with rapid cycle database analytics is successfully enabled if evidence is meaningful, valid, expedited, and transparent. These principles will bring rigor and credibility to current efforts to increase research efficiency while upholding evidentiary standards required for effective decision-making in healthcare.
Regularized Regression Versus the High-Dimensional Propensity Score for Confounding Adjustment in Secondary Database Analyses
Jessica M. Franklin, Wesley Eddings, Robert J. Glynn, and Sebastian Schneeweiss Am J Epidemol 2015 Oct;182(7):651-9
Selection and measurement of confounders is critical for successful adjustment in nonrandomized studies. Although the principles behind confounder selection are now well established, variable selection for confounder adjustment remains a difficult problem in practice, particularly in secondary analyses of databases. We present a simulation study that compares the high-dimensional propensity score algorithm for variable selection with approaches that utilize direct adjustment for all potential confounders via regularized regression, including ridge regression and lasso regression. Simulations were based on 2 previously published pharmacoepidemiologic cohorts and used the plasmode simulation framework to create realistic simulated data sets with thousands of potential confounders. Performance of methods was evaluated with respect to bias and mean squared error of the estimated effects of a binary treatment. Simulation scenarios varied the true underlying outcome model, treatment effect, prevalence of exposure and outcome, and presence of unmeasured confounding. Across scenarios, high-dimensional propensity score approaches generally performed better than regularized regression approaches. However, including the variables selected by lasso regression in a regular propensity score model also performed well and may provide a promising alternative variable selection method.
Jessica M. Franklin, Wesley Eddings, Robert J. Glynn, and Sebastian Schneeweiss Am J Epidemol 2015 Oct;182(7):651-9
Selection and measurement of confounders is critical for successful adjustment in nonrandomized studies. Although the principles behind confounder selection are now well established, variable selection for confounder adjustment remains a difficult problem in practice, particularly in secondary analyses of databases. We present a simulation study that compares the high-dimensional propensity score algorithm for variable selection with approaches that utilize direct adjustment for all potential confounders via regularized regression, including ridge regression and lasso regression. Simulations were based on 2 previously published pharmacoepidemiologic cohorts and used the plasmode simulation framework to create realistic simulated data sets with thousands of potential confounders. Performance of methods was evaluated with respect to bias and mean squared error of the estimated effects of a binary treatment. Simulation scenarios varied the true underlying outcome model, treatment effect, prevalence of exposure and outcome, and presence of unmeasured confounding. Across scenarios, high-dimensional propensity score approaches generally performed better than regularized regression approaches. However, including the variables selected by lasso regression in a regular propensity score model also performed well and may provide a promising alternative variable selection method.
A review of uses of health care utilization databases for epidemiologic research on therapeutics
Sebastian Schneeweiss, Jerry Avorn J Clin Epidemiol 2005 Apr; 58(4);323-37
Large health care utilization databases are frequently used in variety of settings to study the use and outcomes of therapeutics. Their size allows the study of infrequent events, their representativeness of routine clinical care makes it possible to study real-world effectiveness and utilization patterns, and their availability at relatively low cost without long delays makes them accessible to many researchers. However, concerns about database studies include data validity, lack of detailed clinical information, and a limited ability to control confounding
Sebastian Schneeweiss, Jerry Avorn J Clin Epidemiol 2005 Apr; 58(4);323-37
Large health care utilization databases are frequently used in variety of settings to study the use and outcomes of therapeutics. Their size allows the study of infrequent events, their representativeness of routine clinical care makes it possible to study real-world effectiveness and utilization patterns, and their availability at relatively low cost without long delays makes them accessible to many researchers. However, concerns about database studies include data validity, lack of detailed clinical information, and a limited ability to control confounding
