Validation of the Telephone Interview of Cognitive Status and Telephone Montreal Cognitive Assessment Against Detailed Cognitive Testing and Clinical Diagnosis of Mild Cognitive Impairment After Stroke
Background and Purpose—Assessment of cognitive status poststroke is recommended by guidelines but follow-up can often not be done in person. The Telephone Interview of Cognitive Status (TICS) and the Telephone Montreal Cognitive Assessment (T-MoCA) are considered useful screening instruments. Yet, evidence to define optimal cut-offs for mild cognitive impairment (MCI) after stroke is limited.
Methods—We studied 105 patients enrolled in the prospective DEDEMAS study (Determinants of Dementia After Stroke; NCT01334749). Follow-up visits at 6, 12, 36, and 60 months included comprehensive neuropsychological testing and the Clinical Dementia Rating scale, both of which served as reference standards. The original TICS and T-MoCA were obtained in 2 separate telephone interviews each separated from the personal visits by 1 week (1 before and 1 after the visit) with the order of interviews (TICS versus T-MoCA) alternating between subjects. Area under the receiver-operating characteristic curves was determined.
Results—Ninety-six patients completed both the face-to-face visits and the 2 interviews. Area under the receiver-operating characteristic curves ranged between 0.76 and 0.83 for TICS and between 0.73 and 0.94 for T-MoCA depending on MCI definition. For multidomain MCI defined by multiple-tests definition derived from comprehensive neuropsychological testing optimal sensitivities and specificities were achieved at cut-offs <36 (TICS) and <18 (T-MoCA). Validity was lower using single-test definition, and cut-offs were higher compared with multiple-test definitions. Using Clinical Dementia Rating as the reference, optimal cut-offs for MCI were <36 (TICS) and approximately 19 (T-MoCA).
Conclusions—Both the TICS and T-MoCA are valid screening tools poststroke, particularly for multidomain MCI using multiple-test definition.
- mild cognitive impairment
- ROC curve
- Telephone Interview of Cognitive Status
- Telephone Montreal Cognitive Assessment
See related article, p 2919
Stroke survivors are at high risk of cognitive impairment and dementia.1–5 The Montreal Cognitive Assessment (MoCA) is a recommended screening tool6,7 but follow-up can often not be done in person. The Telephone Interview of Cognitive Status (TICS) and the Telephone MoCA (T-MoCA) are commonly used screening instruments. However, only a limited number of studies have adequately investigated the sensitivity and specificity of telephone interviews after stroke with regard to assessing cognitive impairment.6,8
A methodological challenge for telephone screening is the existence of multiple versions of these test instruments. The original TICS was developed from the Mini-Mental State Examination and has a maximum of 41 points.9 It has been validated in patients with stroke for the detection of dementia with proposed cut-offs of ≤2410 and ≤28,11 but there is no validation for mild cognitive impairment (MCI). Several modified versions of the TICS (TICSm) have been introduced by adding items particularly on delayed recall and by omitting other items. Pendlebury et al8 suggested an optimal cut-off at 24 to 25 for MCI using a modified version of the TICS with a maximum of 39 points. However, there are other versions of the modified TICS with maximum scores ranging from 37 to 50 points and results cannot be transferred to the original TICS or other modified versions of the TICS. The more recently developed T-MoCA has been shown to be feasible and valid with an optimal cut-off at approximately 18 to 19 for MCI.8 Yet, results were obtained in community-dwelling subjects and sample size was limited (n=68) thus necessitating additional data in other samples.
Another challenge in classifying cognitive function poststroke relates to multiple definitions of MCI as determined by comprehensive neuropsychological testing (CNT), which is the reference standard for diagnosis.7,12 Variations include single-domain versus multidomain MCI and single-test versus multiple-tests definitions.13 There is no consensus on how cognitive dysfunction after stroke should be defined, and there are considerable differences in terms of validity and cut-offs for screening instruments depending on definitions.13
In the current study, we investigated the effect of both single-test and multiple-tests definitions on test accuracy of the original TICS and T-MoCA for any MCI (single- and multidomain MCI) and multidomain MCI using both CNT and the Clinical Dementia Rating (CDR) scale as reference standards.
Study Design and Study Population
Patients were drawn from the observational prospective DEDEMAS study (Determinants of Dementia After Stroke; NCT01334749).14 The main inclusion criterion was an acute stroke defined by an acute focal neurological deficit in combination with a corresponding lesion on magnetic resonance imaging. Patients were excluded if they had a previous diagnosis of dementia or a summed score of >64 in the short version of the Informant Questionnaire on Cognitive Decline in the Elderly15 at baseline, known diseases of the CNS other than stroke, or a condition interfering with follow-up such as end-stage malignancy or missing language skills. Further details are given in Table I in the online-only Data Supplement.
Between October 2015 and October 2016, patients were invited to undergo 2 separate telephone interviews in addition to their face-to-face visits at 6, 12, 36, or 60 months, all of which included CNT. The telephone interviews were scheduled within 2 weeks before (interview 1) and 2 weeks after (interview 2) the regular face-to-face visits. To control for learning effects, half of the patients received the TICS at interview 1 and the T-MoCA at interview 2 whereas the other half received the 2 tests in reversed order (Figure 1). The median time interval between the telephone interviews and the face-to-face visits was 5 days (Q1–Q3, 3–6 days) for interview 1 and 6 days (Q1–Q3, 3–8 days) for interview 2. None of the patients included into the current study met the criteria for dementia.
Until October 2016, 255 patients enrolled in the DEDEMAS study had reached at least 6-month follow-up time point (Figure I in the online-only Data Supplement). Some of them did not have additional telephone interviews because they died (n=18), withdraw study participation or declined additional telephone interview (n=38), or had no face-to-face visit scheduled between October 2015 and October 2016 (n=94) leaving 105 individuals for analysis. A comparison of characteristics of participants and nonparticipants is provided in the online-only Supplement.
Telephone Assessments (TICS and T-MoCA)
We used the original TICS.9 MoCA items that did not require the use of a pencil and paper or a visual stimulus were used for the T-MoCA, which has a maximum score of 22.8 Details on subtests from the 2 telephone assessments are given in Table II in the online-only Data Supplement.
Reference Standards for Validation and Determination of Cut-Offs
CNT was done using 18 cognitive tests covering the following 4 domains: executive function/attention, memory, language, and visuospatial function (Table III in the online-only Data Supplement). We used a standard threshold of 1.5 SD below published norms to define impairment on a given test corrected for age, sex, and education.7,13,16 The primary analysis is based on a multiple-tests definition for impairment in a given domain.13 Analyses were done both for any MCI (defined as impairment in a single domain or in multidomains) and multidomain MCI (defined by impairment in at least 2 domains).13 We further present results for the single-test definition. The main analysis was repeated using a SD of 1.645.7 Results are given in the Results and Tables IV and V in the online-only Data Supplement. Details on multiple-tests and single-tests definitions are provided in the online-only Data Supplement.
The global CDR score is a commonly used measure for identifying MCI or dementia and composed of 3 domains of cognition (memory, judgment and problem solving, and orientation) and 3 domains of function (community affairs, home and hobbies, and personal care). Interviews were conducted with both the patient and an informant, and sum of boxes were computed. The most heavily weighted component is memory as the CDR was developed to rate clinical dementia in elderly. We used the Washington University online algorithm (http://www.biostat.wustl.edu/~adrc/cdrpgm/index.html) to calculate the global CDR and a score of 0.5 for MCI definition.17–19
Standard Protocol Approvals, Registrations, and Patient Consent
The study was conducted according to the Declaration of Helsinki and approved by the local ethics committee. Written informed consent was obtained by the patient or legal guardian before study participation.
Receiver operating characteristic (ROC) curve analysis was performed to assess discrimination validity of the TICS and T-MoCA. We report area under the ROC curve (AUC). For each cut-off, we computed sensitivity, specificity, positive predictive value, and negative predictive value together with 95% confidence intervals (CIs). ROC curves of TICS and T-MoCA were compared using the nonparametric approach implemented in the ROCCONTRAST statement in SAS (http://support.sas.com/kb/45/339.html). Scores for the TICS and T-MoCA are given as median and interquartile range. The discriminability index d′ is provided in Table VI in the online-only Data Supplement.
Following common practice to add 1 point to the face-to-face MoCA for patients with low education, we performed sensitivity analyses by adding up to 2 points for patients with low education (<12 years) to the TICS (maximum, 43 points) and up to 1 point to the T-MoCA (maximum score, 23 points). We further did additional sensitivity analyses excluding patients with mood disorders (n=8). All analyses were performed using SAS version 9.4 (SAS Institute Inc, Cary, NC).
One hundred five patients (mean age, 72±9 years; 31% women; 36% with education <12 years; Table 1) agreed to undergo the face-to-face visit with CNT and at least 1 of the 2 telephone interviews (Figure 1). Details on stroke classification, cause, and stroke severity are provided in Table 1. Details on Instrumental Activities of Daily Living, mood disorders, and domain specific results at the time of testing are given in Tables VII and VIII in the online-only Data Supplement. Using the multiple tests definition (primary analysis), 14 (13%) had single-domain MCI, 8 (8%) had multidomain MCI, 83 (79%) had no MCI (for single-test definitions, see the online-only Data Supplement), and 16 (15%) had MCI as defined by the CDR.
One hundred one patients received both the TICS and CNT, 100 patients received both the T-MoCA and CNT, and 96 patients completed all assessments (Figure 1). Time intervals after stroke were well balanced between assessments (Figure II in the online-only Data Supplement).
MCI Defined by CNT and Multiple-Tests Definition
Using CNT and the multiple-tests definition as a reference, the median TICS scores (101 patients) were 36 (Q1–Q3, 35–37), 34 (33–36), and 32 (31.5–34.5) in patients with no MCI, single-domain MCI, and multidomain MCI, respectively. AUC was 0.76 for any MCI (95% CI, 0.63–0.88) and 0.83 for multidomain MCI (95% CI, 0.62–1.00; Figure 2). Optimal sensitivities and specificities were achieved at approximately 36 to 37 for any MCI and a cut-off of <36 for multidomain MCI (Table 2). Adjustment for education and exclusion of patients with mood disorders did not improve the AUC (Table IX and X in the online-only Data Supplement).
Corresponding numbers for the T-MoCA scores (100 patients) were 20 (18–21), 17 (16–19), and 15 (11.5–26.5) in patients with no MCI, single-domain MCI, and multidomain MCI, respectively. AUC was 0.82 for any MCI (95% CI, 0.71–0.94) and 0.94 for multidomain MCI (95% CI, 0.88–1.00; Figure 2). Optimal sensitivities and specificities were achieved at a cut-off of <19 for any MCI and a cut-off of <18 for multidomain MCI (Table 2). Again, adjustment for education and exclusion of patients with mood disorders did not improve the AUC (Tables IX and X in the online-only Data Supplement).
MCI Defined by CNT and Single-Test Definition
The number of patients meeting individual MCI definitions and the distribution of TICS and T-MoCA scores obtained using single-test definition are detailed in the online-only Data Supplement. Validity was lower using single-test compared with multiple-tests definition (Figure III in the online-only Data Supplement). For TICS, optimal cut-offs were achieved for any MCI at approximately 37 to 38 and for multidomain MCI at a cut-off of <37. For T-MoCA, optimal cut-offs were achieved at approximately 20 to 21 for both any and multidomain MCI (Table XI in the online-only Data Supplement).
MCI Defined by Clinical Dementia Rating
Using the CDR scores as a reference for MCI definition, the median TICS scores (101 subjects) were 33 (32–35) and 36 (35–37) in patients with and without MCI, respectively. AUC was 0.78 (95% CI, 0.64–0.92; Figure 3). Optimal sensitivities and specificities were achieved at a cut-off of <36 (Table 3). Adjustment for education and exclusion of patients with mood disorders did not improve the AUC (Tables IX and X in the online-only Data Supplement).
Corresponding numbers for the T-MoCA scores (100 subjects) were 18 (14–19) and 20 (17–21) in patients with and without MCI, respectively. AUC was 0.73 (95% CI, 0.59–0.87; Figure 3). Optimal cut-offs were achieved at approximately 19 (Table 3). Again, adjustment for education and exclusion of patients with mood disorders did not improve the AUC (Tables IX and X in the online-only Data Supplement).
Comparison of TICS and T-MoCA
To determine whether any of the 2 screening instruments is superior to the other, we compared ROC curves of the TICS and T-MoCA in the 96 patients who completed both tests. There was no significant difference in the ROC curves irrespective of MCI definitions and reference standards (CNT: multiple-tests definition: any MCI: P=0.11, multidomain MCI: P=0.22; single-test definition: any MCI: P=0.45, multidomain MCI: P=0.58; MCI defined by CDR: P=0.92).
We found both the original TICS and T-MoCA to perform well in detecting MCI after stroke and present cut-off values for future use. Sensitivities and specificities were higher in multidomain MCI than in single-domain MCI particularly when using multiple-tests definition. T-MoCA corresponded slightly better with CNT (multiple-tests definition), whereas TICS corresponded slightly better with the CDR as shown by the AUC analyses. Validity was not improved by adjustment for education.
There is considerable uncertainty about optimal criteria and methods to define MCI.7 This includes the definition of impairment in a given cognitive domain13 as well as the appropriateness of single-domain MCI versus multidomain MCI, as some investigators consider single-domain MCI to be overinclusive and thus subject to false-positive classification.6,13,20 The current study accounted for different criteria and methods to define MCI, thus providing a reference for the definition of cut-offs for both the original TICS and T-MoCA. Considering different MCI definitions, we suggest using a cut-off of <36 for the TICS and <19 for the T-MoCA (Table 2). Our recommendation for the T-MoCA is in line with a previous suggestion from Pendelbury et al,8 whereas the current study is the first to provide a cut-off for the original TICS in patients poststroke.
Our study has several strengths. First, the design was prospective and telephone intervals were validated against multiple reference standards for CNT and against CDR. Second, we included patients at different time points after stroke, and the results can thus be considered representative of a large time interval after stroke.7 Third, the order of interviews and intervals from the face-to-face visits were balanced between the T-MoCA and TICS (Figure 1), thus enabling direct comparisons between the 2 telephone interviews. Fourth, time intervals between the telephone assessments and both CNT and CDR were well defined thus facilitating reliable assessment of sensitivities and specificities.7 Also, our study design accounts for the known overlap of some of the subtests between the TICS and T-MoCA (Table II in the online-only Data Supplement).
We found the T-MoCA to perform slightly better than the original TICS when using CNT as a reference. This applies to both the AUCs and the ability to derive cut-offs for any MCI. A possible explanation might be the composition of test items representing individual domains (Table III in the online-only Data Supplement). Thus, for example, the TICS is weighted toward testing memory function as it was derived from the Mini-Mental State Examination.9 This might also explain why the TICS was slightly superior to detect MCI as defined by the CDR, which was developed for use in patients with Alzheimer disease.21 Overall, both telephone interviews seem similarly suited for use in patients after stroke consistent with a recent meta-analysis of face-to-face interviews that found no single screening test to be clearly superior to others.6
Our study also has limitations. Our sample was relatively small and restricted to less severely affected patients because of the requirement to attend face-to-face visits with 2 additional telephone interviews. Hence, difficulties in conducting telephone interviews in real life might not be adequately reflected by our data. Also, participants were on average highly educated and motivated, which might have contributed to the relatively low number of patients with MCI. Consequently, our suggested cut-points will have to be validated in samples representing different educational levels, race, and ethnicity, as well as in stroke-free samples. The low positive predictive values and high negative predictive values obtained for the proposed cut-offs in our sample emphasize that the T-MoCA and TICS must be viewed as screening tools. Although both tests seem suited to exclude MCI, the proposed cut-offs cannot be used to diagnose MCI. Instead, patients positively screened for MCI with the proposed cut-offs should undergo detailed cognitive testing before making a diagnosis of MCI.
Finally, we did not incorporate recent definitions of vascular MCI as proposed by the American Heart Association22 and of mild neurocognitive disorder as defined in DSM-V (www.dsm5.org). Thus, similar studies using these new categories seem necessary.
Adjustment for education had no substantial influence on the AUCs (Table IX in the online-only Data Supplement) in our sample. This seems to justify common practice to abstain from adding points to scores in telephone screening tests based on educational status. However, the proportion of poorly educated patients was low in the current study, which might have limited our ability to capture an effect of educational adjustment.
In conclusion, both the TICS and T-MoCA are valid screening tools for cognitive impairment poststroke, particularly for multidomain MCI using multiple-test definition. Our data provide cut-offs for both tests for use in clinical practice and in research studies.
We thank Margit Deschner, Brigitte Huber, Julia Hill, Rupert Aigner, and Antonia Weingart for their support. Drs Zietemann and Dichgans designed the study, developed the protocol, obtained ethical approval, and wrote the article. Drs Wollenweber and Müller were involved in patient recruitment, acquisition of data, and revising the manuscript. Dr Zietemann performed the data analysis and Drs Zietemann, Dichgans, and Kopczak interpreted the data. All authors reviewed and edited the article and approved the final version of the article
Sources of Funding
This study was supported by the Deutsche Forschungsgemeinschaft (German Research Foundation) within the framework of the Munich Cluster for Systems Neurology (EXC 1010 SyNergy), the German Center for Neurodegnerative Diseases (DZNE), Munich, the European Union’s Horizon 2020 research and innovation programme under grant agreement No 666881, SVDs@target, and the Vascular Dementia Research Foundation.
The online-only Data Supplement is available with this article at http://stroke.ahajournals.org/lookup/suppl/doi:10.1161/STROKEAHA.117.017519/-/DC1.
- Received April 5, 2017.
- Revision received August 9, 2017.
- Accepted August 31, 2017.
- © 2017 American Heart Association, Inc.
- Qu Y,
- Zhuo L,
- Li N,
- Hu Y,
- Chen W,
- Zhou Y,
- et al
- Dichgans M,
- Leys D
- Moulin S,
- Labreuche J,
- Bombois S,
- Rossi C,
- Boulouis G,
- Hénon H,
- et al
- Lees R,
- Selvarajah J,
- Fenton C,
- Pendlebury ST,
- Langhorne P,
- Stott DJ,
- et al
- Stolwyk RJ,
- O’Neill MH,
- McKay AJ,
- Wong DK
- Pendlebury ST,
- Welch SJ,
- Cuthbertson FC,
- Mariz J,
- Mehta Z,
- Rothwell PM
- Brandt J,
- Spencer M,
- Folstein M
- Hachinski V,
- Iadecola C,
- Petersen RC,
- Breteler MM,
- Nyenhuis DL,
- Black SE,
- et al
- Pendlebury ST,
- Mariz J,
- Bull L,
- Mehta Z,
- Rothwell PM
- Woolf C,
- Slavin MJ,
- Draper B,
- Thomassen F,
- Kochan NA,
- Reppermund S,
- et al
- Jak AJ,
- Preis SR,
- Beiser AS,
- Seshadri S,
- Wolf PA,
- Bondi MW,
- et al
- Gorelick PB,
- Scuteri A,
- Black SE,
- Decarli C,
- Greenberg SM,
- Iadecola C,
- et al