AHRQ/PCORI EXPLORE K12 initiative · Learning Health Systems
Making Formulary Decisions Learnable
Project overview
Health systems make consequential decisions about which medications to add, restrict, or remove from their formularies and treatment pathways. Yet the evidence and assumptions behind those decisions are not always explicit, and their effects are rarely evaluated systematically after implementation.
During my K12-supported work, I partnered with pharmacy, clinical, and quality-improvement teams to develop a practical Learning Health Systems (LHS) framework for the health system Pharmacy & Therapeutics (P&T) Council. The framework helps decision-makers define the question, weigh comparative effectiveness, safety, value, feasibility, and uncertainty, and identify what local evidence is needed when uncertainty remains.
The central idea is simple: a formulary decision should be the beginning of an evidence cycle, not the end of an evidence review. The process connects decision aids and value assessment with routinely collected EHR data to generate real-world evidence (RWE), alongside causal inference, target-trial emulation, implementation science, and Plan-Do-Study-Act methods. This gives the system a pathway to make a decision, monitor how it performs in practice, and revisit it as new evidence emerges.
From framework to practice
Evaluate effectiveness in the local clinical context
I used difference-in-differences and target-trial-emulation approaches to assess the real-world impact of medication policies and treatment guidelines, including the implementation of tocilizumab for critically ill patients with COVID-19. The work combined local EHR evidence with trial benchmarking to help decision-makers interpret rapidly changing evidence.
Translate evidence into safer, more cost-effective care
In a large, multi-campus quality-improvement initiative, I combined EHR measurement, clinician feedback, implementation strategies, audit and feedback, and iterative PDSA cycles to improve neuromuscular-block monitoring and reversal practices. Across 35,198 surgical cases, quantitative monitoring increased from 42% to 83%, while postoperative respiratory complications decreased by 41% over the course of the initiative.
Use local data to answer questions before or alongside traditional research
An observational evaluation of patiromer use for acute hyperkalemia in the emergency department used a matched retrospective cohort design to assess whether adding patiromer to insulin improved sustained potassium reduction and reduced additional treatment. The study produced locally relevant evidence and identified questions for further investigation; its findings were consistent with a clinical trial published later.
Beyond medications: evaluating pharmacist-led programs
The same principles apply to clinical programs and services, not only to drugs. I helped develop and test rigorous approaches for evaluating pharmacist-led transitions-of-care programs, with particular attention to the biases that can make these interventions appear more effective than they are.
Evaluating only patients who attended a post-discharge clinic can introduce selection and immortal-time bias: patients must survive and remain engaged long enough to receive the intervention. In a heart-failure transitions-of-care study, readmissions were lower among clinic attendees, but not among all eligible referrals in an intention-to-treat analysis. This distinction changes the practical interpretation of the program’s effect and offers a general lesson for evaluating real-world care-delivery interventions.
I applied the same careful approach to a pharmacist-supported meds-to-beds program after percutaneous coronary intervention. The evaluation found no adjusted reduction in 30-day all-cause readmissions; an apparent reduction in cardiac-related readmissions was attenuated after adjustment.
Why this work matters
The work was embedded in a pharmacy-driven learning health system research and quality-improvement center, which I co-founded in 2018. The center brings together pharmacists, physicians, nurses, health economists, administrators, informaticists, and quality-improvement specialists.
The center has supported more than 20 research and quality-improvement initiatives, including de-implementing low-value practices, improving medication use and outcomes, expanding research capacity, and supporting health system P&T decision-making. Its stakeholder-engaged structure and hybrid use of institutional resources and external funding were designed to make rigorous, actionable work sustainable.
The products and lessons from this project are collected here as a resource for others working to build learning health systems, strengthen medication-use decisions, or make better use of routinely collected healthcare data.
Selected publications and analyses
Using local EHR data, this medication-use evaluation examined both the population impact of adding tocilizumab to institutional guidelines and the effect of receiving treatment. The analysis used difference-in-differences and matched target-trial-emulation approaches and benchmarked local estimates against clinical trials and published evidence. Tocilizumab was administered to 69% of eligible patients; the estimates supported a beneficial effect while illustrating the uncertainty that can accompany rapid, real-world policy evaluation.
This matched retrospective cohort study compared patiromer plus insulin with insulin alone among 266 adults treated in the emergency department. Overall, adjunctive patiromer did not produce a greater sustained potassium reduction in the 4–12-hour window. A subgroup with eGFR greater than 30 mL/min received numerically fewer potassium-lowering interventions, a finding that warrants further study.
This study showed how the choice between an as-treated and an intention-to-treat analysis can change the apparent effectiveness of a clinical program. Readmissions were lower among clinic attendees, but not among all patients referred to the program, highlighting selection and immortal-time bias in evaluations that focus only on participants.
This observational evaluation examined bedside delivery of a 30-day supply of dual antiplatelet therapy paired with pharmacist-directed education. The program was not associated with a reduction in adjusted 30-day all-cause readmissions; an unadjusted difference in cardiac-related readmissions was attenuated after adjustment.
Across 35,198 surgical cases at a diverse, multi-campus medical center, a multidisciplinary intervention increased quantitative Train-of-Four monitoring from 42% to 83% and was associated with a 41% reduction in postoperative respiratory complications. The initiative combined practice mapping, clinician interviews, implementation strategies, audit and feedback, and PDSA cycles to translate evidence into routine care.
Methods and training
This work brought together epidemiology, causal inference, real-world evidence generation, implementation science, health economic modeling, and performance improvement. Relevant training included an implementation science certificate, a performance improvement fellowship, Harvard Chan target-trial-emulation training, and ISPOR health economic modeling training.
The quality-improvement work was recognized as a performance improvement project of the year.
Current initiatives
- Does inpatient SGLT2 availability lead to better transitions of care and increase outpatient use in eligible patients?
- Do combination triple-therapy COPD inhalers (LAMA/LABA/ICS) on the inpatient formulary improve transitions of care and increase use among eligible patients despite their higher cost?
Funding and disclaimer
This work was supported through the AHRQ/PCORI EXPLORE K12 initiative. The findings and conclusions presented here are those of the authors and do not necessarily represent the official views or positions of the Agency for Healthcare Research and Quality (AHRQ), the Patient-Centered Outcomes Research Institute (PCORI), or their affiliates.