What the Model Card Generator actually produces

# Model Card: 30-Day Hospital Readmission Risk Predictor ## Model Details - **Model Name:** Meridian Readmission Risk Classifier v1.0 - **Model Type:** Gradient-boosted classifier (XGBoost v1.7) - **Developer:** Meridian Health Analytics - **Date:** [REQUIRES DATA TEAM INPUT — model training completion date] - **Version:** 1.0 - **Input Features:** Diagnosis codes (ICD-10), vital signs, length of stay, prior 12-month utilization (ED visits, inpatient admissions), age, sex, insurance type, hospital identifier - **Output:** Probability of 30-day unplanned readmission (continuous score 0–1; operational threshold 0.35 for high-risk flag) ## Intended Use - **Primary Users:** Care coordinators at Meridian partner hospitals - **Primary Use Case:** Prioritize discharge follow-up outreach to high-risk patients within 48 hours of discharge - **Out-of-Scope Uses:** Autonomous clinical decision-making; determining insurance coverage or reimbursement; direct patient communication without clinician review; use outside 4-state Midwest region; pediatric or psychiatric patients ## Factors - **Protected Attributes:** Race/ethnicity, insurance type (included in training; fairness analysis performed) - **Geographic Scope:** 4 Midwestern hospitals; regional variation in discharge practices may affect generalization - **Patient Populations Excluded:** Pediatric (<18), psychiatric unit admissions, obstetric admissions - **Time Period:** Model trained on 2018–2022 data; operational performance monitored on 2023 admissions ## Metrics - **AUC-ROC:** 0.81 (held-out 2023 test set, n=28,000) - **Precision (at 0.35 threshold):** 0.77 | Recall: 0.72 | F1: 0.74 - **Calibration (Brier score):** 0.14 | Platt scaling applied to training set - **Baseline:** [REQUIRES DATA TEAM INPUT — readmission rate in test population] ## Evaluation Data - **Source:** De-identified EHR records from 4 Meridian partner hospitals - **Time Period:** 2023 (held-out test set; temporally distinct from training) - **Size:** n=28,000 admissions - **Exclusions:** Same as training data (pediatric, psychiatric, obstetric) - **Data Characteristics:** [REQUIRES DATA TEAM INPUT — test set demographic breakdown by race/ethnicity, insurance type, age distribution, readmission rate] ## Training Data - **Source:** De-identified EHR records (2018–2022) - **Hospitals:** 4 Midwestern health systems - **Volume:** ~210,000 admissions - **Features:** Diagnosis codes, vitals at discharge, length of stay, prior ED/inpatient utilization (12-month lookback), demographics - **Exclusions:** Pediatric units, psychiatric admissions, obstetric units - **Data Quality:** [REQUIRES DATA TEAM INPUT — missing value rates by feature; any imputation strategy] - **Demographic Breakdown:** [REQUIRES DATA TEAM INPUT — % by race/ethnicity, insurance type, age bands in training cohort] - **Consent & Licensing:** [REQUIRES DATA TEAM INPUT — IRB exemption/approval status; data-use agreement references] ## Quantitative Analyses - **Subgroup Performance (Race/Ethnicity):** [REQUIRES DATA TEAM INPUT — AUC-ROC, Precision, Recall, False Negative Rate by racial/ethnic group in test set] - **Subgroup Performance (Insurance Type):** [REQUIRES DATA TEAM INPUT — AUC-ROC, Precision, Recall by insurance category (commercial, Medicare, Medicaid, uninsured) in test set] - **Known Disparity:** False Negative Rate gap for Black patients flagged at +6% (Black FNR 0.28 vs. non-Black FNR 0.22). **Impact:** Approximately 6% higher likelihood of missing high-risk Black patients at current threshold. - **Threshold Sensitivity:** [REQUIRES DATA TEAM INPUT — ROC curve analysis showing FNR by threshold for affected subgroups] ## Ethical Considerations - **Potential Harms:** (1) Disparate prioritization of follow-up resources by race/ethnicity if +6% FNR gap in Black patients not mitigated; (2) Over-reliance on algorithmic score without clinician judgment; (3) Feedback loop if care coordination quality varies by predicted risk tier. - **Mitigation (In Progress):** Threshold adjustment strategy under review to equalize False Negative Rate across racial groups; care coordinator training on model limitations; mandatory clinician review of top 10% risk patients. - **Transparency:** Model scores provided to care coordinators with accompanying uncertainty and subgroup performance documentation. ## Caveats and Recommendations - **Generalization Limits:** Model trained on 4 Midwestern hospitals; performance on independent regional health systems unknown. Recommend validation before expansion. - **Data Drift:** Readmission risk drivers may shift post-pandemic; monitoring plan required (see Recommendations). - **Threshold Calibration:** Current 0.35 threshold chosen for [REQUIRES DATA TEAM INPUT — rationale/business constraint]. Threshold adjustment to address FNR disparity will change Precision/Recall trade-offs; operational impact assessment needed. - **Feature Limitations:** Model does not capture social determinants (housing, food insecurity, transportation), substance use severity, or outpatient follow-up appointment scheduling—all known readmission drivers. - **Protected Attribute Use:** Race/ethnicity included in training features. While fairness analysis performed, use of protected attributes in operational scoring warrants legal and ethics review. ## Recommendations 1. **Immediate (Pre-Deployment):** Complete subgroup performance validation in independent test cohort; finalize threshold adjustment strategy for FNR disparity; document care coordinator training curriculum on model limitations and bias mitigation. 2. **Post-Deployment Monitoring:** (a) Monthly calibration checks; (b) quarterly subgroup performance audits by race/ethnicity and insurance type; (c) escalate if any subgroup FNR drifts >5% from baseline; (d) track care coordination contact rate and readmission outcome by predicted risk tier to detect feedback loops. 3. **Future Work:** Retrain annually on most recent 2 years of data; evaluate addition of social determinant variables (via validated linkage); conduct external validation at ≥2 non-Meridian health systems before regional deployment.

## Fairness Audit Plan ### Subgroup Analyses **Protected Attributes (Required Ongoing):** - Race/Ethnicity: White, Black, Hispanic, Asian, Other/Unknown. Compute separately for each; flag if any subgroup has >5% disparity on any metric vs. majority group. - Insurance Type: Commercial, Medicare, Medicaid, Uninsured. Compute separately; flag if any type has >5% disparity. - Age Bands: <40, 40–65, >65 (clinical relevance). Flag if any band shows AUC-ROC <0.75 or FNR >0.30. **Intersectional Analysis (Quarterly):** - Cross race/ethnicity × insurance type (e.g., Black + Medicaid, White + Commercial). Minimum cell size n=200 per intersectional segment; flag if any intersection has <100 admissions (insufficient reliability). - High-volume hospital site interaction: compute metrics separately by each hospital; flag if any site shows >10% subgroup disparity not explained by patient mix. ### Fairness Metrics **Primary Metrics (Monthly Reporting):** 1. **Demographic Parity:** Proportion of high-risk flags (score ≥0.35) by subgroup. Target: ≤5% absolute difference from overall high-risk rate. 2. **Equal Opportunity (False Negative Rate):** FNR by subgroup (already flagged +6% for Black patients). Target: ≤3% absolute difference; threshold adjustment triggered if any subgroup FNR >0.30. 3. **Equalized Odds:** Simultaneous equality of FPR and FNR across subgroups. Report FPR and FNR separately; flag if FPR disparity >5% or FNR disparity >5%. 4. **Calibration:** Brier score and calibration slope by subgroup. Target: Brier score ≤0.16 within any subgroup; recalibrate if slope <0.85 or >1.15. **Secondary Metrics (Quarterly):** 5. **Positive Predictive Value (PPV) by subgroup:** If any subgroup PPV <0.70, document clinical impact (e.g., false alarm rate). 6. **Treatment Equality:** Actual readmission rate by predicted risk tier and subgroup. If predicted high-risk group has <70% readmission rate within any subgroup, model may be miscalibrated for that population. ### Escalation Thresholds - **Automatic Escalation to Compliance & Chief Medical Officer:** Any subgroup with (a) AUC-ROC <0.75, (b) FNR >0.30, or (c) demographic parity disparity >10%. - **Model Retraining Trigger:** If any protected attribute shows >5% disparity in any fairness metric sustained over 2 consecutive monthly audits. - **Threshold Adjustment Trigger:** If current +6% FNR gap for Black patients persists after retraining; propose threshold lowering for this subgroup or stratified thresholds (separate cutoffs by race) pending legal review. - **Deployment Pause:** If any intersectional segment (e.g., Black + Medicaid, n≥200) shows AUC-ROC <0.70 or FNR >0.35. ### Audit Governance - **Responsible Party:** Meridian Data Science team, with quarterly review by compliance and clinical leadership. - **Data Sources:** De-identified readmission outcomes (30-day unplanned), predicted scores from operational system, demographics from EHR. - **Audit Frequency:** Monthly automated metric computation; quarterly manual review; annual comprehensive fairness report. - **Documentation:** Store audit results in version-controlled repository; maintain change log for all threshold adjustments or retrainings.

## Reviewer Questions for Non-Data Scientists **Regulatory & Compliance:** 1. Has an IRB or compliance officer reviewed the use of protected attributes (race/ethnicity) in the training features and operational scoring? What is the legal basis for their inclusion? 2. Does the organization have a data-use agreement with each of the 4 hospitals covering model development and deployment? What are the contractual limits on geographic or population expansion? 3. Is there a documented policy for threshold adjustments in response to fairness audit findings (e.g., if the +6% False Negative Rate gap widens)? Who approves threshold changes? **Performance & Risk:** 4. What is the baseline readmission rate in the test population? (A 0.81 AUC-ROC is only meaningful relative to baseline.) Is the model better than simple rule-based prioritization (e.g., age >75 or length of stay >7 days)? 5. The model has a +6% False Negative Rate disparity for Black patients. In plain language: of 100 Black patients who will be readmitted, how many will the model miss (not flag as high-risk) compared to non-Black patients? What is the care coordination plan for missed high-risk patients? 6. Precision is 0.77; this means 23% of flagged patients are not readmitted. What is the operational impact of calling patients who don't need intervention? Will this overwhelm care coordinator capacity or create alert fatigue? **Operationalization & Monitoring:** 7. Who will monitor model performance post-deployment, and with what frequency? What triggers a model retrain or immediate review? 8. The model was trained on 2018–2022 data. How will you detect if readmission drivers have shifted (e.g., due to new discharge protocols, telehealth uptake, or long-COVID)? What is the decision rule for retraining? 9. Are care coordinators informed that this model does not capture social determinants (housing, food insecurity, transportation) that strongly predict readmission? What is the process for escalating patients with known social barriers? **Fairness & Equity:** 10. The fairness audit plan flags automatic escalation if any subgroup has a False Negative Rate >0.30. If this occurs for a particular insurance type or hospital, what is the remediation timeline? Who decides whether to pause deployment in that region? 11. Intersectional analysis requires a minimum of 200 patients per subgroup combination (e.g., Black + Medicaid). If a hospital admits <200 such patients per quarter, how will you ensure adequate monitoring for that intersection? 12. If threshold adjustment (e.g., lowering the cutoff for Black patients to equalize False Negative Rates) is implemented, will care coordinators and clinicians be informed that different risk thresholds apply to different racial groups? How will this be communicated without causing bias or mistrust? **External Validation & Scope:** 13. Before expanding to hospitals outside the 4-hospital development cohort, will the model be independently validated on a separate health system? What performance thresholds must be met to proceed? 14. The model excludes pediatric and psychiatric patients. Are there plans to develop separate models for these populations, or will this tool never be used for them? How is this restriction enforced in the operational system? **Human-in-the-Loop:** 15. The model is "not for autonomous clinical decisions," but what does this mean operationally? Will every high-risk patient be reviewed by a clinician before care coordination outreach is triggered? Or only the top 10%? Document the specific review protocol.