Results of a Computerized Screening of Stroke Patients for Unjustified Hospital Stay
Background and Purpose Effective methods to monitor length of stay can help reduce unnecessary hospital stay without adversely affecting the quality of care. In this study a clinical algorithm for assessing unjustified hospital stay in stroke patients was computerized and tested.
Methods An algorithm was developed by the authors to estimate the number of medically justified and unjustified hospital days for patients admitted with a primary diagnosis of ischemic stroke. Data for the algorithm were obtained from 177 stroke patients from an acute-care teaching hospital. The performance of the algorithm was evaluated on a subset of 46 patients by comparing the number of medically unjustified hospital days determined by the algorithm with the consensus determination of two neurologists.
Results The algorithm classified 68% of the 177 patients as having some unjustified hospital days and 41% of all hospital days as unjustified. With the neurologists as the gold standard, the sensitivity of the algorithm was .89 and the specificity was .91. The correlation between the number of unjustified days determined by the algorithm and the neurologists was .76.
Conclusions There is considerable unjustified length of stay for stroke patients. Physicians can develop simple clinical algorithms for detecting unjustified hospital stay in stroke patients that provide a reasonable approximation of complex clinical judgment.
Hospitals have become increasingly involved in monitoring care to reduce unjustified stay.1 Unfortunately, unjustified LOS monitoring can create a medical care atmosphere that emphasizes cost reduction at the expense of quality. Standards of care such as those developed by the firm of Milliman & Robertson, Inc2 will have the most adverse effects on quality when they do not discriminate well between unjustified stay and long stay. Methods that discriminate poorly are likely to create pressure to prematurely discharge patients who stay beyond a threshold but may not affect patients with shorter stays who could have had an earlier discharge.
There are two approaches often used for careful monitoring of unjustified stay. One is to use risk-adjusted outcomes. This method classifies as unjustified a stay longer than expected for a patient with a given presentation.3 The other approach is to use clinical judgment to determine whether the stay is too long. The judgment may be implicit, ie, it depends entirely on a single clinician, or it may be explicit, ie, it depends entirely or in part on rules that have been derived by many clinicians. A comprehensive explicit review tool has been developed for LOS.4 Although procedures involving clinical judgment can discriminate between long and unjustified stays, they may be expensive if they require extensive data collection or the time of a clinical reviewer. The purpose of the present study was to develop and evaluate a computerized procedure that incorporates some of the philosophy of comprehensive clinical review systems but is less expensive to use. The procedure was developed for stroke patients, but similar procedures could be developed for other diseases and conditions.
Subjects and Methods
Source and Type of Data
The patients for this study were hospitalized for stroke at Froedtert Memorial Lutheran Hospital, a private nonprofit hospital affiliated with the Medical College of Wisconsin. Medical records were selected from a computer listing of patients with discharge dates from January 1, 1992, to July 14, 1993, with a primary discharge diagnosis of ischemic stroke (ICD-9-CM codes follow in parentheses): cerebral thrombosis (434.0); cerebral embolism (434.1); cerebral artery occlusion, unspecified (434.9); and acute but ill-defined cerebrovascular disease (436). Records were reviewed in 1993 and 1994. Records were not reviewed if they were not available or if they had ICD-9-CM codes for subarachnoid hemorrhage, cerebral hemorrhage, transient ischemic attack, or any surgical procedure. There were 419 records searched; 218 (52%) were excluded because the patients had hemorrhagic stroke, precerebral occlusion, or surgery, and 24 (12% of the charts with an appropriate diagnosis) were excluded because the charts were not available.
The nurse reviewers entered detailed information from the medical records into a computerized database. The types of information entered included diagnosis codes, procedures codes, demographic information, administrative information, vital signs, neurological findings on the physical examination (eg, level of consciousness, motor deficits, aphasia), activities of daily living (eg, mobility and feeding), medical history, laboratory studies, radiology results, interventions (eg, ICU, telemetry, all lines and tubes, medications, occupational therapy), complications (eg, pneumonia, deep vein thrombosis, pressure ulcers), and discharge disposition. All information that was entered on the patient was printed out on an abstract that facilitated the review of the medical record by the neurologist.
Description of the Algorithm
An algorithm was developed by the authors to estimate the number of medically justified and unjustified hospital days for patients admitted with a primary diagnosis of ischemic stroke. Although it would have been better to use generally accepted guidelines for the development of the algorithm, no such guidelines existed, and we used our own clinical judgment to draft the algorithm. The algorithm was revised after feedback from the neurological faculty at the Medical College of Wisconsin and after the algorithm was tested on charts that are not discussed in this report. The algorithm did not evaluate the necessity of hospital admission, which has been addressed elsewhere.5
The justifications for hospitalization for stroke patients included in the algorithm are summarized in Table 1⇓. Since the first 3 hospital days usually involve close observation and performance of various routine diagnostic tests, they were always considered to be justified by the algorithm. Only the remainder of the hospital stay was evaluated by the algorithm.
The justifications for hospital stay are shown in Table 1⇑. Ideally, the justifications should only include the medical needs of the patient, such as stroke progression (defined as a specific indication of stroke progression in the chart or any deterioration in motor or mental function) or impaired oral intake (including orders to give nothing orally). However, an algorithm based only on patient characteristics would be too complex to implement economically. For that reason most justifications of hospital stay were physician decisions that depended on the medical judgment of the physician treating the patient as well as patient characteristics. These judgments included the need for the patient to be in the ICU or to require procedures such as cardiac telemetry or cerebral angiography. Since basic laboratory investigations (eg, blood work, electrocardiography, imaging studies, carotid ultrasonography, or echocardiography) are routinely completed in the first few days of hospitalization, specific allowance was not made for these tests. A fewer number of additional days than shown in Table 1⇑ may be justified for some patients. However, we were liberal in our allowances of the number of additional days to reduce the false-positive rate for the algorithm.
The patient conditions and medical interventions listed in Table 1⇑ not only justify the hospital day on which they occur but also a number of subsequent hospital days, as shown in the table. For example, stroke progression justifies all hospital days while the stroke is progressing and 3 additional days after the patient stabilizes. The patient’s stay is also justified for 3 days after discharge from the ICU, 2 days after oral feeding was impaired, and 1 day after most of the other testing, monitoring, and treatment procedures. For this retrospective review, death automatically justified the entire LOS. Death was included as part of the algorithm because it is a marker of severe disease that might not be otherwise recognized by the algorithm.
Computerization of the stroke algorithm used CARS, a data application written by Kenneth Goldberg as part of his Howard Hughes fellowship in 1991 to 1992. The application was written in the Paradox Application Language6 and linked to the VP-Expert system7 for execution of the algorithm. With CARS, nurses who have little computer experience can create computerized algorithms and data entry screens.
Evaluation of the Algorithm
The reliability of the abstracted data was evaluated by comparing algorithmic results on 38 randomly selected charts that were abstracted by the two nurses who performed most of the data abstraction. The reliability of the data for determining the presence of any unjustified stay was measured by the κ statistic, and the reliability of the data for determining the number of unjustified hospital days was measured by the Pearson correlation coefficient.
The performance of the algorithm was evaluated by comparing unjustified stay as determined by the algorithm with unjustified stay as determined by each of two board-certified neurologists (R.S.G., D.J.L.). Of the 177 charts abstracted for the computerized review, 50 were selected to be reviewed by the neurologists. Two of the selected charts could not be located, and two of the cases did not have a primary diagnosis of acute ischemic stroke. It is not known how many of the 129 unreviewed charts involved misdiagnoses. When reviewing the records, neither neurologist knew the results of the algorithm or the results of the review by the other neurologist. Each neurologist used his own judgment to determine the number of medically unjustified days that the patient stayed in the hospital. No explicit criteria were used for review. After independently reviewing each of the cases for unjustified stay, the neurologists discussed the cases in which they disagreed on the number of unjustified days to arrive at a consensus. This consensus was used as the gold standard for whether there was unjustified stay and for the number of unjustified days.
The two neurologists who performed the chart review were involved in the creation of the algorithm. Thus, the study tested the question of whether physicians could create an algorithm that reflected their own clinical judgment. The answer to this question was not evident a priori since clinical judgment requires a great deal of information that cannot be captured in a simple algorithm.
Based on clinical experience and a preliminary review of the charts, a stratified sample was used for chart review. The three strata were determined by length of hospital stay: 3 days or less, 3 to 15 days, and greater than 15 days. Charts with stays of 3 days or less were undersampled since almost all of these stays were entirely justified (five of five by physician review). Charts with stays of more than 15 days were also undersampled because almost all of these charts had unjustified LOS (12 of 13 by physician review). Since the physicians were only able to review a limited number of cases, we oversampled those cases that most required review because their status could not easily be determined by LOS alone. The reason for this type of sampling was to determine the effectiveness of the algorithm for the most difficult cases. For each year of hospitalization, the charts in a given strata were sampled consecutively by medical record number. Since requested charts were often not available on the first search, a high percentage of the charts were ordered.
The performance of the algorithm relative to the review of each neurologist was expressed in three ways: (1) The sensitivity of the algorithm was the percentage of cases determined by the neurologist to have any unjustified stay that were also found by the algorithm to have unjustified stay. (2) The specificity of the algorithm was the percentage of cases determined by the neurologist to have no unjustified stay that were also considered to have no unjustified stay by the algorithm. (3) The ability of the algorithm to determine the number of unjustified days was expressed as the correlation between the number of unjustified days determined by the algorithm and the number of unjustified days determined by the consensus of the neurologists. Because the sampling procedure selected cases that were likely to be most difficult for the algorithm, the performance of the algorithm was probably better in the complete data set than in the sample of reviewed cases.
The medical charts of 177 patients admitted with acute stroke were abstracted. An average of 7 minutes was required to abstract the information necessary to run the algorithm. The most time-consuming data element was the determination of whether there was a progression in the stroke during the period of hospitalization. To obtain all of the data elements necessary for a comprehensive study of resource utilization, however, was much more difficult. The abstraction of the data elements described in “Subjects and Methods” required an average of 45 minutes per patient.
A characterization of patients abstracted for the study is shown in Table 2⇓. The patients are divided into whether the charts were reviewed by the neurologists. As expected for stroke, the patients were generally elderly. A high percentage of the patients were nonwhite, and less than 30% of the patients had private insurance. The average LOS was 10.4 days for all patients, 11.1 days for the patients whose charts were not reviewed, and 8.4 days for the charts of the patients who were reviewed. The difference in LOS for the reviewed and unreviewed charts occurred because of undersampling of charts with extreme LOS, as described in “Subjects and Methods.” The reviewed charts with a positive algorithm (ie, indicating some unjustified stay) had a shorter average LOS than the unreviewed charts because fewer charts with very long stays were reviewed; the reviewed charts with a negative algorithm (ie, indicating no unjustified stay) had a longer average LOS than the unreviewed charts because fewer charts with stays of 3 days or less were reviewed.
The distribution of the number of unjustified days as determined by the algorithm is presented in Fig 1⇓. For 57 patients (32%) there were no unjustified days (negative algorithm). For 67 patients (38%) there were between 1 and 4 unjustified days, and for another 53 patients (30%) there were 5 or more unjustified days. For the 57 patients with no unjustified stay, the average LOS was 5.6 days. The average LOS for the remaining 120 patients was 12.6 days. Since the average medically justified LOS was 6.4 days for these patients, the average unjustified LOS was 6.2 days. For the entire group of 177 patients, 747 hospital days were judged by the algorithm to be medically unjustified. This represented 41% of total hospital days. Half of the unjustified days were accounted for by patients with more than 8 unjustified days, although this group included only 15% of the patients.
The distribution of the number of unjustified days differed for patients on the neurology service and those admitted to other services. Eighty-two percent of the patients were admitted to the neurology service. The average LOS for these patients was 9.6 compared with 14.1 for those with attending physicians on other services (P=.03). The average number of unjustified days for the stroke patients on the neurology service as determined by the algorithm was 3.5 compared with 7.7 for the patients with attending physicians on other services (P=.01).
The reasons for justified stay beyond 3 days are shown in Table 3⇓. Of the 146 patients who stayed in the hospital longer than 3 days, 74% had at least 1 day that was justified by the algorithm beyond the 3 initial days. Impaired oral intake was the most commonly noted reason that justified some LOS greater than 3 days. The use of the ICU and administration of heparin and other intravenous medications were other frequently noted reasons for extended stay, although they rarely justified the entire LOS.
For this retrospective review the algorithm considered death to justify the entire LOS. However, 85% of the days for the 10 subjects who died would have been justified by other components of the algorithm. This suggests that including death in the algorithm does not greatly affect the accuracy of the algorithm. An algorithm without death may provide an accurate method for concurrent review of hospitalized cases.
Evaluation of the Algorithm
For the 38 charts abstracted by two nurses, the determination by the algorithm of whether or not the entire hospital stay was medically justified was the same in 33 of the cases (κ=.72, P<.0001), indicating a moderate degree of interrater agreement beyond chance. Disagreement in the 5 cases was attributed to one abstraction error (failure to record the change from intravenous to oral anticonvulsants), disagreement as to the day that oral intake became adequate, and disagreement on whether there was progression of the stroke (3 cases). The correlation between the number of unjustified days determined by the algorithm from the data of the two abstractors was .92 (P<.001).
For the 46 charts reviewed by both neurologists, there was agreement concerning whether the entire hospital stay was medically justified in 38 cases (κ=.59, P<.0001). Of the remaining 8 cases, each neurologist rated 4 as justified, while the other neurologist rated them with some unjustified stay. In 5 of these 8 cases the neurologist who noted unjustified LOS noted only 1 day of unjustified stay. The correlation between neurologists with respect to the number of unjustified days was .78. After independent review, the cases were discussed in an attempt to attain a consensus review. The neurologists arrived at a consensus except for 2 cases. For each of these 2 cases, one neurologist thought there was 1 unjustified day and the other thought there were no unjustified days.
The agreement of the algorithm with each of the neurologists was comparable to the agreement of the neurologists with each other (Table 4⇓). The results of the algorithm were also compared with the consensus of the two neurologists. Of the 33 cases that had unjustified stay by consensus, 29 had unjustified stay by the algorithm, ie, the sensitivity was .89. Only 1 of the 11 cases that the neurologists classified as having no unjustified stay had unjustified stay by the algorithm, ie, the specificity was .91. The algorithm was negative for both cases on which the neurologists disagreed.
The agreement between the algorithm and the consensus of the neurologists for 44 cases is also shown in Fig 2⇓. In approximately half of the reviews the algorithm and neurologists agreed exactly on the number of medically justified hospital days (points falling on the 45-degree line). The slight preponderance of points falling below the line indicates that the neurologists judged fewer hospital days medically justified than did the algorithm. The correlation between the results of the algorithm and the consensus of the two neurologists was .76 for the number of unjustified days. The correlation was even greater for the number of justified days (.91) because the justified days account for most of the stay.
The reasons for the unjustified hospital stay according to a consensus of the neurologists are shown in Table 5⇓ for the 33 patients whose charts were reviewed by the neurologists and who had unjustified stay. Nearly half of the patients were awaiting placement for rehabilitation (n=15), but other reasons, such as awaiting tests (n=8) or consultations (n=5), were also important. Since some patients had more than one reason for unjustified stay, more than 33 reasons are listed.
A computerized algorithm was used to determine a profile of unjustified stay for persons hospitalized with stroke. Based on the results of the algorithm on 177 patients, 68% of those treated at the study hospital for ischemic stroke had at least 1 unjustified hospital day, and 41% of all hospital days were medically unjustified. These results are comparable to a previous study of unjustified hospital days on a neurology ward.8 We evaluated the validity of the algorithm for determining the profile of unjustified stay on a subset of 44 charts using the consensus review of two neurologists as the gold standard. To detect a case with unjustified stay, the algorithm had a sensitivity of .89 and a specificity of .91. The correlation of the number of unjustified days determined by the algorithm with the number of the consensus review was .76. Because the sampling scheme oversampled those cases that were most difficult for the algorithm to assess correctly, the estimate of the sensitivity and specificity of the algorithm for detecting unjustified stay is conservative.
The results of this study strongly support the ability of physicians to generate a simple algorithm that is easily computerized and often agrees with complex medical judgments about the appropriate time for discharging a stroke patient. The level of agreement between the algorithm and the neurologists was about the same as that between the two neurologists. There may be less agreement between the algorithm and the judgments of other neurologists who do not believe that the components used in the present algorithm justify hospital stay. However, at a minimum the results show that physicians can generate an algorithm that reflects their own medical judgment. This was a surprising finding since clinical judgment is complex and must consider many aspects of patient care. Yet the clinicians agreed with the findings of this simple algorithm as often as they agreed with the judgment of each other.
The purpose of the algorithm was to address unjustified LOS due to inefficiencies in providing services or discharging the patient. The algorithm was not designed to address other issues: (1) unnecessary hospitalization, (2) unnecessary LOS caused by errors in the quality of care (eg, errors causing stroke progression or additional ICU stay), and (3) unnecessary use of resources (eg, admission to the ICU) that obscures unnecessary LOS. Although the ability to address these issues would be useful, it would add to the complexity of an algorithm designed to monitor LOS.
The algorithm can be used in paper form as a screening tool for unjustified LOS. Cases found by the algorithm to have unjustified stay can then be reviewed by physicians to confirm that there was a problem. The algorithm will be much more useful if it is computerized, however, for the following reasons: (1) electronically stored data can be entered into the computer and may substantially reduce the time of data abstraction and calculations of unjustified stay, and (2) with computerized data it is much easier to obtain clinical practice profiles of the computerized algorithm results. Profiling of the algorithm results may be useful to suggest causes of delays (eg, practices of specific physicians or differences in care on weekends) and to monitor the effect of changes in care (eg, adding more social workers). A secondary benefit of obtaining profiles is that it may be useful to profile components of the algorithm. For example, the components include the number of days in the ICU, on telemetry, or on intravenous medication. Profiling these components will be useful to monitor resource utilization. It will also be possible with these profiles to determine whether the rates of unjustified stay may be underestimated because the use of certain resources is high.
Based on the results of the computerized algorithm, we have instituted clinical guidelines at Froedtert Hospital to help reduce LOS. The algorithm will be used to monitor the effect of these guidelines. Previous studies have found that the institution of clinical protocols can reduce hospital LOS and associated charges.9 10 11 Although the average LOS for stroke patients has significantly decreased over the past few decades, large regional differences persist. These differences are apparently due to widespread variation in patient management.12
Other methods have been used to obtain practice profiles for LOS. The least expensive and most commonly used is to compare average LOS at the index hospital with a benchmark LOS that is usually determined from other hospitals. An LOS the same or less than the benchmark would be reassuring; a longer LOS would suggest that the hospital is wasting resources. For the following reasons, however, the comparison with a benchmark may not be useful: (1) Patients at the benchmark hospitals may have different LOS than the index hospital because they differ with respect to severity of illness. This problem can be reduced by the use of methods such as the Computerized Severity Index3 to adjust for severity of illness. (2) The benchmark may not be ideal; ie, it may also have many cases with unjustified LOS. (3) Unjustified LOS at the index hospital may be partially masked by cases that are discharged prematurely. Reduction in the LOS with premature discharge may become more of a problem as pressures to reduce LOS increase.
In contrast to benchmarking, practice profiles derived from a computerized algorithm avoid these disadvantages of benchmarking since the algorithm evaluates the course of disease and treatment instead of only LOS.
The Appropriateness Evaluation Protocol4 is similar to the algorithm described in this study in that it makes use of clinical criteria to identify cases with unjustified hospital stay. With this protocol each hospital day is reviewed by explicit criteria to determine whether hospitalization for that day is medically justified. The protocol differs from the current approach in two important ways: (1) Since it is generic (ie, it uses the same criteria for all cases regardless of the patient’s diagnosis), it requires the abstraction of more data elements and does not deal with the specific issues required in decision making for stroke patients. (2) Since it is not computerized and allows the nurse reviewers to use their own judgment for overriding the criteria, it is more subjective and may be more variable than the computerized algorithm described in this report. The computerized algorithm will have more advantages as more data elements become computerized and the time for data abstraction is reduced or eliminated.
The use of computerized algorithms for reviewing care has recently been introduced as a method for monitoring health care. The first use of the method was the UCDSS, which was developed by the HCFA.13 14 15 The initial purpose of the UCDSS was to computerize generic algorithms that had been used by nurses for reviewing quality of care and the necessity of hospital admission for individual cases. Because of the low specificity of the algorithms in the first version of UCDSS and a shift toward profiling care rather than identifying errors in individual cases, HCFA has largely abandoned the UCDSS as originally conceived.14 There are several fundamental differences, however, between the algorithm tested in this study and those in the original UCDSS: (1) The approach in this study is disease specific rather than the generic approach first used by HCFA. (2) The algorithm focuses on resource utilization after hospital admission rather than the need for admission and quality of care. (3) The algorithm was developed locally for a specific hospital rather than nationally for all hospitals. For these reasons the approach to algorithm development in this study may be more successful than that first tried by HCFA.
The use of computerized clinical algorithms to monitor health care is in the early stages of development. There has been little work defining the role of algorithms in monitoring care and even less in developing specific algorithms. As extensive electronic clinical data bases become more available, the cost of using algorithms will be greatly reduced and experimentation with implementing algorithms is likely to increase. The results presented in this report suggest that algorithms can be valuable in monitoring LOS for stroke patients. Since assessment of the LOS for stroke patients is as complex as the assessment of many other aspects of medical care, it is likely that computerized algorithms to evaluate aspects of care for other diseases or conditions may also perform well.
Selected Abbreviations and Acronyms
|CARS||=||Customizable Algorithmic Review System|
|HCFA||=||Health Care Financing Administration|
|ICD-9-CM||=||International Classification of Diseases, 9th Revision, Clinical Modification|
|ICU||=||intensive care unit|
|LOS||=||length of stay|
|UCDSS||=||Uniform Clinical Data Set System|
This study was supported in part by an American Heart Association Grant-in-Aid (Dr Goldman); Clinical Investigator Development Award funding from the National Institutes of Health (K08-NS-01549 to Dr Lanska); Research Advisory Group funding from the Office of Research and Development, Department of Veterans Affairs (Dr Lanska); and a Howard Hughes Fellowship (Kenneth C. Goldberg, MD). The authors wish to acknowledge Dr Thad Hagen for providing the extensive hospital resources and support needed for this project; Kenneth C. Goldberg, MD, for writing the computer program on which the study was based; Sheri Dix, RN, for coordinating the data collection; Michele Agnello, RN, Pam Epple, RN, and Kurt Donzelli, RN, for collecting the data; and Teri Wermager for preparing the manuscript.
- Received November 13, 1995.
- Revision received January 8, 1996.
- Accepted January 8, 1996.
- Copyright © 1996 by American Heart Association
Myerson AR. Helping health insurers say no. New York Times. March 20, 1995:C1.
Horn SD, Sharkey PD, Buckle JM, Backofen JE, Averill RF, Horn RA. The relationship between severity of illness and hospital length of stay and mortality. Med Care. 1991;27:305-317.
Lanska DJ. Review criteria for hospital utilization for patients with cerebrovascular disease. Neurology. 1994;44:1531-1532.
PAL Programmer’s Guide. Scotts Valley, Calif: Borland International, Inc; 1992.
Rule Bases Expert System Development Tool VP Expert. Paperback Software International; 1989.
Wachtel T, Moulton AW, Pezzulo J, Hamolsky M. Inpatient management protocols to reduce health care costs. Med Decis Making. 1986;6:101-109.
Odderson IR, McKenna BS. A model for management of patients with stroke during the acute phase. Stroke. 1993;24:1823-1827.
Bowen J, Yaste C. Effect of a stroke protocol on hospital costs of stroke patients. Neurology. 1994;44:1961-1964.
Lanska DJ. Medicare hospital utilization review for ischemic cerebrovascular disease. Neurology. 1993;43:650-654.
Lanska D. A public/private partnership in the quest for quality: development of cerebrovascular disease practice guidelines and review criteria. Am J Med Qual. 1995;10:100-106.