Test Accuracy of Short Screening Tests for Diagnosis of Delirium or Cognitive Impairment in an Acute Stroke Unit Setting
Background and Purpose—Guidelines recommend cognitive screening in acute stroke. Various instruments are available, with no consensus on a preferred tool. We aimed to describe test accuracy of brief screening tools for diagnosis of cognitive impairment and delirium in acute stroke.
Methods—We collected data on sequential stroke unit admission in a single center. Four assessors trained in cognitive testing independently performed screening and reference tests. Brief assessments comprised the following: 10- and 4-point Abbreviated Mental Test (AMT-10; AMT-4); 4-A Test (4AT); Clock Drawing Test (CDT); Cog-4; and Glasgow Coma Scale (GCS). We also recorded the multidisciplinary team’s informal review using single question (SQ). We compared against reference standards of Montreal Cognitive Assessment (MoCA) and Confusion Assessment Method for delirium using usual diagnostic cutpoints. For MoCA, we described effects of lowering the diagnostic threshold to MoCA <24 and MoCA <20. We described sensitivity, specificity, and positive and negative predictive values.
Results—Over a 10-week period, 111 subjects had cognitive assessment data. Subjects were 50% male (n=55), and median age was 74 years (interquartile range, 64–85). AMT-4, AMT-10, and SQ all had excellent (1.00) specificity for detection of cognitive impairment, although sensitivity was poor (all <0.60). The 4AT had greatest sensitivity for detecting delirium (1.00 [confidence interval [CI], 0.74–1.00]) and reasonable specificity (0.82 [CI, 0.72–0.89]). Properties of 4AT for detection of cognitive impairment, at the traditional MoCA threshold, were also good (sensitivity, 0.86; specificity, 0.78). Using diagnostic thresholds of MoCA ≤26, <24, and <20 gave proportions with cognitive impairments of 86%, 61%, and 49%, respectively, with resulting changes in screening test properties. At lower MoCA thresholds, CDT had favorable sensitivity and specificity (MoCA <20: sensitivity, 0.93, specificity, 0.66; MoCA <24: sensitivity, 0.85, specificity, 0.77).
Conclusions—Many brief screening assessments are specific but not sensitive for detection of cognitive impairment in acute stroke. Our primary analysis suggests that 4AT is a reasonable choice for delirium and cognitive screening in this setting. However, these data are based on standard MoCA diagnostic threshold and may not be suited for an acute stroke population.
Cognitive and mood problems are prevalent in stroke survivors1 and have clinical and prognostic implications. The importance of the cognitive dimension to stroke outcomes is increasingly recognized, and many countries recommend active case finding through screening all stroke survivors for cognitive issues.2,3 The need for robust cognitive screening strategies is not limited to stroke survivors; for example, in the UK the Commissioning for Quality and Innovation (CQUIN) is implementing payment frameworks to encourage cognitive screening of all older adults admitted to hospital.
Early detection should allow for prompt intervention and tailoring of the management plan to suit the cognitive needs of the patient. Thus, screening for cognitive issues in the acute setting makes intuitive sense. However, cognitive assessment in the acute stroke setting presents several challenges. Screening strategies may be complicated by stroke-related and nonstroke-related factors; incident delirium is common in stroke4 and will impact on cognition, communication and functional impairments may impact on the ability to complete standard cognitive assessments, and, in any acutely unwell cohort, clinical condition may limit the feasibility of valid cognitive assessment.
Various instruments and scales are available for cognitive assessment. No tool specifically designed for acute stroke has been described and there is no consensus on the optimal assessment tool for this setting.5 Detailed neuropsychological assessment may not be practical in a busy acute stroke unit. In practice, a 2-stage technique is often used with initial brief screening by a nonspecialist used to select those subjects who may need more detailed assessment. Again, a variety of brief screening tools designed for acute medical wards are available, but descriptions of the properties of these tools in acute stroke are limited.6
The usual pattern for diagnostic accuracy studies is to compare index test(s) against a clinical gold standard. In cognitive studies, the reference standard is usually clinical diagnosis of dementia.7 The purpose of brief screening tests in the acute stroke unit is not to make a dementia diagnosis, but to select those subjects who may need further assessment. Thus, we chose to describe properties of brief screening assessments against a more detailed cognitive test. Delirium can also be screened for using brief cognitive assessment tools. Delirium may relate to, but is not synonymous with, poststroke cognitive impairment; therefore, we examined properties of tools for delirium diagnosis separately.
Few cognitive assessment tools have been specifically designed or validated for use in acute stroke settings. An instrument commonly used in stroke settings is the Montreal Cognitive Assessment (MoCA).8 MoCA has many properties that make it attractive for use with stroke survivors; however, normative data are derived from community-dwelling older adults and the traditional MoCA score threshold (≤26) was designed to distinguish mild cognitive impairment.9 There is no consensus on the optimal cutpoint of MoCA for distinguishing stroke survivors with clinically important cognitive problems.10,11
We sought to describe test accuracy properties of various brief (<2 minutes of testing time) screening assessments against an independent clinical diagnosis of cognitive impairment (using MoCA) and delirium in the acute stroke setting. We also described the effect of altering the screen-positive cutpoint for MoCA using differing predetermined diagnostic thresholds.
We devised the study in line with methodological guidance for diagnostic test accuracy studies (Quality Assessment of Diagnostic Accuracy Studies [QUADAS2]).12 For reporting, we used Standard for the Reporting of Diagnostic accuracy Studies (STARD)13 and the dementia-specific extension STARDdem.14
The study was conducted in the stroke units of an urban teaching hospital (Glasgow Royal Infirmary, UK). Data collection was performed over a 10-week period (April–June 2012). The units admit all suspected strokes regardless of patient age, prestroke function, or severity of stroke. The unit has a policy of cognitive screening; certain instruments are suggested but no particular tool is favored.
Assessors were a group of 4 medical students undertaking a period of elective study in stroke. All were trained in use of the brief and more detailed cognitive assessment tools. Training comprised use of scale-related educational materials, tutorials, assessment of mock cases, and initial supervision of study cases.15
Patients were consecutive, consenting inpatient stroke survivors (ischemia and hemorrhage). We were deliberately inclusive and used no specific exclusion criteria, although patients had to be medically stable to allow an attempt at a least part of a cognitive assessment. Decisions on appropriateness of assessment were made by the clinical team. Patients were assessed during the period of day 1 to day 4 after stroke unit admission.
Our index tests were a selection of brief (<2 minutes of administration time) screening tools approved for use in our hospital. Screening tools were chosen based on literature review, previous use in stroke populations, and ease of data collection.
The brief screening assessments comprised the following:
Hodkinson 10-point abbreviated mental test (AMT-10)16 and the shortened 4-item abbreviated mental test (AMT-4) were used.17 These direct cognitive tests are commonly used in UK hospital practice and have been validated in acute care settings.17
The clock drawing test (CDT) was scored using a 0- to 3-point method. This direct screening test does not require verbal output and has been suggested as suitable for use in stroke.18
The COG-4, a cognitive examination derived from National Institutes of Health Stroke Scale (NIHSS) and that is suggested as suitable for use in stroke, was used.19
The 4-A Test (4AT), a 4-component assessment based on direct assessment and observations, also was used. The 4AT has scoring rules to differentiate cognitive impairment from delirium.20
We collected the admitting clinician’s informal assessment of presence of cognitive issues; this was derived from the medical case sheet. Because clinicians infrequently recorded any cognitive assessment, we included Glasgow Coma Scale (GCS) best eye opening (<4) and verbal response (<5) as suggestive of cognitive/delirium issues.21 This approach has been suggested for use in acute care settings. We also collected a standardized single question (SQ),22 “does this patient have cognitive issues” at the daily multidisciplinary team review of patients.
To assess for prevalent cognitive issues, we used the Informant Questionnaire for Cognitive Decline in the Elderly (IQCODE)23 (Table 1). The IQCODE was not intended as a brief screening tool for incident cognitive problems but was meant to quantify the numbers with potential dementia before stroke event.
Our reference standard for diagnosis of cognitive impairment was the MoCA.8 This is a multidomain, direct assessment that has been used in stroke populations.24 We also assessed for presence of delirium using the Confusion Assessment Method (CAM).25 The CAM is the most commonly used delirium assessment system and has been validated for use in hospital inpatients26 (Table 1).
For all index tests and reference standards, we used standard cutpoints to describe test-positive and test-negative cases (Table 1). We did not modify the assessments for patients with specific impairments (eg, aphasia, hemianopia). Researchers were allowed to assist participants with motor weakness as needed. We noted the presence of impairments that may effect performance on standard cognitive assessments, giving particular attention to language and visuo-spatial impairments. We approached all stroke survivors for whom the treating clinical team felt that an attempt at even basic cognitive screening would be appropriate. We scored “unable” only when the subject could make no attempt at any of the cognitive assessments.
Researchers from the pool of 4 were randomly allocated to various tasks for each patient using a simple folded paper in a hat task allocation. Three researchers performed same-day cognitive assessments. One researcher completed the short screening assessments, another assessed for cognitive impairment using the MoCA, and the third assessed for delirium using the CAM. Researchers were blinded to each other’s scores. The assessor not performing cognitive assessments independently collected clinical and demographic details from patients’ case records. Specific metrics of interest were GCS, stroke subtype (using the Oxford Community Stroke Project classification),27 sensory impairment, and active diagnosis of depression or dementia. The assessor performing case note extraction also collected IQCODE data from informants when available and recorded SQ data from the MDT (Figure).
Analyses were specified before data were collected. Our primary analyses of interest were test accuracy of each short screening test for the dichotomous outcome cognitive impairment/no cognitive impairment or delirium/no delirium. We created standard 2×2 data tables for each index test against the reference standard of MoCA and CAM describing binary test results cross-classified with the binary reference standard. We calculated sensitivity, specificity, and positive and negative predictive values with corresponding 95% confidence intervals (95% CI). We also created 3×3 tables that incorporate data on those subjects who were unable to be scored using index test or reference standard (an intention to diagnose approach).28
To assess the effect of differing diagnostic thresholds of MoCA, we performed all analyses using the traditional cutpoint of ≤26 and repeated all analyses using prespecified cutpoints of <24 and <20, chosen based on previous literature describing use of MoCA in acute and stroke settings.9–11,24
Recognizing that the tests used are not suited for patients with aphasia or sensory problems (for example, hemianopia), we performed sensitivity analyses removing subjects with severe aphasia (NIHSS language score >2) and adjusting MoCA diagnostic thresholds for subjects with visual–spatial defects. We did not assess metrics of reproducibility (for example, interobserver variability).
Statistical analyses were performed using SPSS (version 18.0) and Statsdirect software (Stats Direct Ltd, UK). Data were collected and stored in line with Scottish Stroke Care Audit procedures, and corresponding data governance measures were followed.29 Data extraction from proformas and analyses were performed by an independent researcher who did not participate in data collection (R.L.).
Over the 10-week study period, 138 eligible subjects were admitted; cognitive data were collected from 111. Subjects with cognitive data were 50% male (n=55); median age was 74 years (interquartile range [IQR], 64–85) and 43 (38%) had a history of stroke.
Stroke classifications were lacunar stroke (n=26, 23%), partial anterior circulation stroke (n=29, 26%), posterior circulation stroke (n=9, 8%), and total anterior circulation stroke (n=20, 18%). Median NIHSS score was 3 (IQR, 1–5). Subjects had various risk factors for incident cognitive problems: 13 (12%) had a documented diagnosis of dementia, 7 (6%) had depression, and 16 (14%) had prestroke sensory impairments. There was substantial comorbidity, and median number of medications was 7 (IQR, 4–9). IQCODE data were available for 90 subjects (median, 3.2; IQR, 3.0–3.9). At a threshold of 3.4, 37 (41% of all tested) subjects had IQCODE scores, suggestive of prestroke dementia.
Using the standard diagnostic threshold for MoCA defined 86 (86%) as having cognitive impairment; using thresholds of <24 and <20 gave proportions with cognitive problems of 62 (61%) and 49 (49%), respectively (Tables 2 and 3).
For our analysis of brief screening tests against MoCA using traditional cutpoints, direct assessments (AMT-4, AMT-10) and informal staff assessments (GCS and SQ) all had excellent specificity but were limited by sensitivity (Table 2). Describing brief screening tests for diagnosis of delirium, the 4AT had excellent sensitivity and reasonable specificity (1.00 and 0.82, respectively); sensitivity and specificity of 4AT for detection of cognitive impairment (at MoCA≤26) were also favorable (0.86 and 0.78, respectively). Other tests were limited by their poor sensitivity or specificity (Table 4). The properties of screening tests differed when MoCA diagnostic thresholds were adjusted. Using lower MoCA cutpoints, the CDT seemed to perform well (MoCA <20: sensitivity, 0.93, specificity, 0.66; MoCA <24: sensitivity, 0.85, specificity, 0.77); informal assessments (SQ and GCS) still had suboptimal sensitivity (Table 3).
Short screening assessments were completed by most of the patients assessed (Table 1); we present exemplar 3×3 tables for AMT-10 assessment (cognition) and 4AT (delirium; Table 5). Seven patients had severe aphasia; removing these subjects from the analysis made little difference to the index test properties (see the online-only Data Supplement). Correcting for sensory problems, 5 patients had the MoCA score reclassified as no impairment. This correction was associated with a trend toward improved test properties (see the online-only Data Supplement).
We have demonstrated substantial differences in test properties of brief screening assessments for assessment of cognition and delirium in an acute stroke setting. We have also demonstrated that using traditional cutpoints for MoCA results in the majority of stroke survivors being labeled as having cognitive impairment.
Our basic level cognitive assessment battery reveals that confusion on the acute stroke unit will be a mix of possible prevalent dementia (often undiagnosed), incident cognitive change (not all delirium), and language and sensory disturbances. In light of this complexity, it is not surprising that unstructured clinical assessment does not perform well. Of the brief screening tests studied, using the standard MoCA threshold, the 4AT had favorable properties for delirium screening and reasonable properties as a cognitive screen. This comes with the caveat that 4AT (and all screening test) properties were altered when we adjusted the MoCA threshold. At lower thresholds, which may be more appropriate in an acute stroke setting, the CDT had favorable properties.
In any assessment of test properties, there will be a trade-off between sensitivity and specificity. Because delirium may be a signal of an intercurrent physiological stressor that necessitates a change in treatment (sepsis, metabolic derangement, or others), high sensitivity may be preferred; in this case, 4AT and the AMT-based tests performed well. The sensitivity/specificity balance when screening for general cognitive problems is less obvious and will depend on the purpose of the screening.
Our prevalence of cognitive issues (86% of subjects tested) is higher than previous estimates of cognitive problems in stroke survivors, albeit most studies have not been performed in the acute poststroke period.30 We suggest that MoCA thresholds for use in acute stroke require revision and we recommend further research to describe normative data in an acute stroke setting and to chart natural history of early cognitive problems.31 Numbers with delirium were lower than in previous stroke series; this is likely a result of the fluctuating nature of delirium and our use of a single CAM assessment, whereas other studies have used daily assessments.4 It is important to note that we can make no inference about properties of screening tests for diagnosis of poststroke dementia from our data. Testing this hypothesis would require prospective follow-up with comprehensive clinical assessments.
We acknowledge the limitations of this work. Assessors were not experienced stroke clinicians, although they were fully trained in all assessments. Not all our brief screens are standard practice and certain of these were chosen because of local availability or ease of use. There is potential for incorporation bias because the tests chosen are not mutually exclusive. For example, MoCA includes CDT data, and 4AT incorporates AMT-4. We did not measure metrics such as patient/clinician acceptability, time taken for assessment, or change in clinical management as a result of screening.
We assessed consecutive stroke unit admissions as able. Median NIHSS score is low for an unselected acute stroke cohort because we did not include those with very severe stroke. Although this exclusion limits generalizability, in practice, standard cognitive assessment in the context of severe stroke or other medical emergency is unlikely to be feasible or clinically useful. Rather, we present data on a population representative of stroke survivors who may be considered for acute cognitive screening.
Accepting these limitations, we present data on an important, topical, and relatively under-researched area of stroke care. The size of our dataset is comparable with other test accuracy studies. As best possible, we took steps to ensure internal validity (randomization, blinding, independent analysis) and followed best practice in methodology and reporting.
We have demonstrated that informal assessment for cognitive issues and delirium is insufficiently sensitive in acute stroke. In general, the use of brief screening tools was feasible in an acute stroke unit setting. We have shown that favorable diagnostic properties of traditional screening instruments should not be assumed. Of the screening tools we tested, 4AT seems a reasonable choice as a delirium and cognitive impairment screen, whereas CDT may be useful for cognitive screening. We have also demonstrated that thresholds for cognitive screening, derived from community-dwelling older adults, may not be applicable in an acute stroke setting. Given current recommendations for routine cognitive assessments in stroke, we urge further test accuracy assessments of any proposed screening instruments.
We thank Jean Wilson, Stroke Audit Coordinator Glasgow Royal Infirmary, for assisting with data. We thank Anna Noel-Storr, Cochrane Dementia and Cognitive Improvement Group, for sharing STARDdem reporting guidance.
Dr Quinn has received grant support for studies of stroke assessment and has received payment for developing educational resources for stroke researchers. He has received research grants from Chest Heart and Stroke Scotland British Geriatric Society and Royal College of Physicians and Surgeons Glasgow to pursue work relating to cognitive screening in stroke. R. Lees work in cognitive assessment in stroke has been supported by a Chest Heart and Stroke Scotland project award. The other authors report no conflicts.
The online-only Data Supplement is available with this article at http://stroke.ahajournals.org/lookup/suppl/doi:10.1161/STROKEAHA.113.001724/-/DC1.
- Received April 15, 2013.
- Accepted July 17, 2013.
- © 2013 American Heart Association, Inc.
- Hachinski V,
- Iadecola C,
- Petersen RC,
- Breteler MM,
- Nyenhuis DL,
- Black SE,
- et al
- 3.↵Scottish Intercollegiate Guidelines Network (SIGN). Management of patients with stroke. Rehabilitation, prevention and management of complications and discharge planning. SIGN Publication Guideline 118;2010.
- Shi Q,
- Presutti R,
- Selchen D,
- Saposnik G
- Lees R,
- Fearon P,
- Harrison JK,
- Broomfield NM,
- Quinn TJ
- Blake H,
- McKinney M,
- Treece K,
- Lee E,
- Lincoln NB
- Davis DHJ,
- Creavin ST,
- Noel-Storr A,
- Quinn TJ,
- Smailagic N,
- Hyde C,
- et al
- Godefroy O,
- Fickl A,
- Roussel M,
- Auribault C,
- Bugnicourt JM,
- Lamy C,
- et al
- Bossuyt PM,
- Reitsma JB,
- Bruns DE,
- Gatsonis CA,
- Glasziou PP,
- Irwig LM,
- et al
- 14.↵STARDdem Initiative. STARDdem Web site. 2002. www.starddem.org.Accessed May 2013.
- Inouye SK
- Hodkinson HM
- Schofield I,
- Stott DJ,
- Tolson D,
- McFadyen A,
- Monaghan J,
- Nelson D
- 20.↵The 4AT test group. 4AT test materials and scoring. 2011. http://www.the4at.com/the-4at. Accessed May 2013.
- Sands MB,
- Dantoc BP,
- Hartshorn A,
- Ryan CJ,
- Lujic S
- Cumming TB,
- Bernhardt J,
- Linden T
- Schuetz GM,
- Schlattmann P,
- Dewey M
- Douiri A,
- Rudd AG,
- Wolfe CD
- Pendlebury ST,
- Wadling S,
- Silver LE,
- Mehta Z,
- Rothwell PM