In this study, the primary objective was to externally validate the HFRS to accurately predict in-hospital mortality in a large, nationally representative cohort of mechanically ventilated older adults. In its current form, the HFRS could not be successfully validated for use in this population. As expected, we found that hospitalizations of mechanically ventilated patients at intermediate and high risk of frailty, as categorized by the HFRS, were associated with increased risks of prolonged hospitalization and emergency hospital readmissions. 30 days, compared to low-risk hospitalizations. Counterintuitively, they were inversely associated with in-hospital mortality compared to low-risk hospitalizations, suggesting a potentially spurious relationship. Regardless, the HFRS had only moderate discrimination and accuracy in predicting any of these outcomes. Using the HFRS as a continuous variable or with splines did not provide additional value over using the low, intermediate, and high risk HFRS subcategories. Our results would suggest that clinicians and researchers should avoid using HFRS when conducting big data research with critically ill patient administrative datasets.
Comparison with previous studies
Previous studies of the HFRS have focused on its validation in general hospitalizations, including non-ICU and ICU patients17,18,19,36,37,38,39. Recently, there has been interest in external validation of HFRS in USI administrative databases, as research interest in the fragility of big data increases.21,40,41. A German intensive care study of 1,498 patients assessed HFRS to predict a combined outcome of mortality and risk of readmission and found no association after adjusting for disease severity21. In a large population-based study in Wales, the HFRS had only a moderate ability to predict inpatient, 6-month and 1-year inpatient and ICU mortality41. Conversely, a US study of 12,854 patients, using the single-center MIMIC-III (Medical Information Mart for Intensive Care) database, found that higher HFRS was associated with increased risk of mortality. at 28 days40.42.
In our study, we found that critically ill older adults hospitalized on mechanical ventilation were at high risk for poor outcomes, including prolonged hospitalization (41%), 30-day in-hospital mortality (44%), to hospital (45%) and readmission to emergency hospital within 30 days (20%). Unsurprisingly, the use of palliative care was very high at 26.8%, with higher use in high-risk frailty groups. The overall readmission rate was high among patients in this study, suggesting the current difficulties in transitioning care for these patients and the potential scope for quality improvement.
Previous studies of critically ill patients have established that frailty is associated with increased risks of mortality3.4. Counterintuitively, we found that HFRS was inversely associated with mortality in the NRD (ie, lower HFRS was associated with the highest risks of in-hospital mortality). To verify this surprising and unnatural finding, we performed a post hoc analysis on the entire NRD population of older adults, regardless of receipt of mechanical ventilation, and found that HFRS performed well on the together population (i.e., higher HFRS was associated with the highest risks of in-hospital mortality in everything the elderly) (ESM eTable 13).
There may be possible explanations for this unusual phenomenon, including selection bias and coding bias. In the original study by Gilbert et al., they validated the HFRS in a general hospitalized population to predict in-hospital mortality16. In general, critically ill patients have a higher risk of death than the general hospitalized population, representing a surrogate endpoint. Therefore, by limiting our population to mechanically ventilated patients, a selection bias may have been introduced, potentially altering the true association of FHSR and mortality. Coding biases may also arise, as critically ill patients who have been hospitalized for a prolonged period and/or who have survived their hospitalization may appear more “frail” as they accumulate more secondary diagnoses according to ICD-10-CM recorded in their medical records. In BDNI, most hospitalizations of mechanically ventilated older adults were in the intermediate-risk frailty group, and most hospitalizations in the high-risk group had significantly more ICD-10-CM codes entered compared to hospitalizations in the low risk group. band. Finally, frail patients with more severe disease or those with treatment limitations may choose less invasive treatments, introducing additional selection bias. We have adjusted the non-resuscitation status; however, this may not fully capture all processing limitations.
These biases and differences in the ICU patient population compared to the original development cohort could potentially explain why the HFRS had mixed performance in predicting in-hospital mortality in an ICU patient population, as shown. seen in this study and others.
Strengths and limitations
Our study had several strengths, including the use of a large multicenter dataset, including nearly 650,000 weighted hospitalizations. To our knowledge, our study represents one of the largest studies of critically ill patients examining the use of HFRS, allowing generalization of our findings to critically ill older adults receiving mechanical ventilation. Unlike previous external validation studies in administrative databases of critically ill patients, we assessed the HFRS to predict prolonged hospitalization and 30-day emergency hospital readmissions. In addition, we assessed both the discrimination and the calibration of the model, which provides confidence in the results presented. Finally, our study performed several sensitivity analyzes to verify our results.
However, our study has limitations. As discussed previously, selection bias may have occurred in our selection of a mechanically ventilated population. The BDNI was not designed specifically to report ICU admissions. Therefore, identification of critically ill patients was done using ICD-10 codes, specific to mechanical ventilation. Other codes, such as use of vasopressors, are known to be significantly undercoded in administrative databases43. Because the HFRS is derived from a composite of ICD-10 codes, coding practices and biases may affect the relative prevalence of admission comorbidities, diagnoses, and treatments. Some codes important for determining HFRS, such as dementia in Alzheimer’s disease (F00) or care involving the use of rehabilitation procedures (Z50), were undercoded (ESM eTable 3). This is also seen in other databases including the Centers for Medicare & Medicaid Services and the National Inpatient Sample Databases36.37. Other critically ill patient databases may operate differently, depending on their coding practices.
In addition, the NRD does not have enough information to determine the severity of illness in the intensive care unit, such as Sequential Organ Failure Assessment (SOFA) scores or acute physiological assessment and d Chronic Health Assessment II (APACHE II). We are therefore unable to verify whether the HFRS would perform better after adequate adjustment for disease severity; however, other studies would suggest that the HFRS does not work well even after adjusting for disease severity21. Similarly, the BDNI does not capture detailed clinical information (i.e. patient weight, vasopressor dosage) and, although it does capture information on the duration of mechanical ventilation, this information are often incomplete. In addition, it does not record out-of-hospital deaths, which limits our ability to assess only in-hospital mortality. Finally, we did not assess other scores because this was outside the scope of our study. These limitations highlight the difficulty of applying HFRS to critically ill patient datasets and reinforce our caution about using HFRS to predict these outcomes.
Clinical Implications, Research Implications and Future Directions
Clinicians need to have accurate predictions of frailty and outcomes to identify patients who would benefit from early referral to geriatric medicine, as well as to engage with patients and their families in shared decision-making, discussion about goals of care and end-of-life planning, and/or referral to palliative care. Similarly, health care administrators must have accurate estimates of the number of frail patients to plan and allocate health care services. Big data researchers need accurate scores to correctly classify patients.
While the HFRS may have utility in non-ICU databases, our study demonstrates its limitations in critically ill patients. The mFI is a promising alternative; however, it must be developed and validated for use with ICD-10-CM codes15.44. Perhaps the best solution for clinicians, researchers and administrators would be to adapt and transform existing databases for frailty research. Along with other well-validated frailty scores such as the CFS, there is a compelling case for its integration into routine clinical practice and inclusion in data capture. Future research should be done to redevelop the HFRS or other scores with different weighting specifically for critically ill patients.