PREDICTING PROGRESSION FROM NORMAL TO MCI AND FROM MCI TO AD USING CLINICAL VARIABLES IN THE NATIONAL ALZHEIMER’S COORDINATING CENTER UNIFORM DATA SET VERSION 3: APPLICATION OF MACHINE LEARNING MODELS AND A PROBABILITY CALCULATOR

 

Y. Pang¹, W. Kukull², M. Sano³, R.L. Albin^4,5, C. Shen⁶, J. Zhou¹, H.H. Dodge^7,8

 

1. Department of Computer Science and Engineering, Michigan State University, East Lansing, Michigan, USA; 2. National Alzheimer’s Coordinating Center, University of Washington, Seattle, Washington, USA; 3. Department of Psychiatry, Alzheimer Disease Research Center, Icahn School of Medicine at Mount Sinai, New York, New York, USA; 4. Department of Neurology, Michigan Alzheimer’s Disease Center, University of Michigan, Ann Arbor, Michigan, USA; 5. Neurology Service & GRECC, VAAAHS, Ann Arbor, MI, USA; 6. Biogen, Inc, Cambridge, Massachusetts, USA; 7. Department of Neurology, Layton Aging and Alzheimer’s Disease Center, Oregon Health & Science University, Portland, Oregon, USA; 8. Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, USA

Corresponding Author: Hiroko H. Dodge, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, USA, hdodge@mgh.harvard.edu

J Prev Alz Dis 2023;2(10):301-313
Published online January 19, 2023, http://dx.doi.org/10.14283/jpad.2023.10

 

---

Abstract

Clinical trials are increasingly focused on pre-manifest and early Alzheimer’s disease (AD). Accurately predicting clinical progressions from normal to MCI or from MCI to dementia/AD versus non-progression is challenging. Accurate identification of symptomatic progressors is important to avoid unnecessary treatment and improve trial efficiency. Due to large inter-individual variability, biomarker positivity and comorbidity information are often insufficient to identify those destined to have symptomatic progressions. Using only clinical variables, we aimed to predict clinical progressions, estimating probabilities of progressions with a small set of variables selected by machine learning approaches. This work updates our previous work that was applied to the National Alzheimer’s Coordinating Center (NACC) Uniform Data Set Version 2 (V2), by using the most recent version (V3) with additional analyses. We generated a user-friendly conversion probability calculator which can be used for effectively pre-screening trial participants.

Key words: National Alzheimer’s Coordinating Center (NACC), Alzheimer’s disease, MCI, dementia, MCI conversion, AD, PET amyloid , machine learning.

---

 

Introduction

Clinical trials are increasingly focused on treating Alzheimer’s disease (AD) during early, asymptomatic, or pre-dementia stages. Differentiating those likely to progress in clinical diagnosis from normal to MCI or from MCI to dementia/AD from those unlikely to progress is challenging. Recruiting non-progressors into trials leads to reduced statistical power for detecting efficacy (1-3). As biomarkers are not necessarily correlated with clinical outcomes, using biomarkers alone as screening tools might not be optimal for trial enrichments, may be costly, and is susceptible to sample selection bias because of their invasive nature. The importance of accurate identification of progressors is expected to increase because of the recent FDA approval of aducanumab for MCI and mild dementia due to Alzheimer disease. The Centers for Medicare and Medicaid Services (CMS) made a decision to cover treatment costs with this monoclonal antibody only for Medicare beneficiaries enrolled in approved studies (https://www.cms.gov/medicare-coverage-database/view/ncacal-decision-memo.aspx?proposed=N&ncaid=305). Since this agent has a relatively high risk of amyloid-related imaging abnormalities (ARIA) (4), accurately predicting and enrolling those at higher risk of clinical progression into trials is critical for avoiding unnecessary treatments.
In the United States, National Institute of Health (NIH)-funded Alzheimer’s Disease Centers are required to use an uniform assessment approach (Uniform Data Set [UDS]) and upload this data to the National Alzheimer’s Coordinating Center (NACC) (https://www.naccdata.org). We previously used clinical variables from NACC UDS Version 2 (V2) and a machine learning approach to identify a set of variables predicting normal to MCI conversions within 4 years (5). UDS Version 3 (V3) replaced V2 in 2015 (6, 7). In the current study, we updated our analyses by using V3 with additional analyses. These additional analyses include: 1) using shorter durations of follow-ups to predict conversions from MCI to clinical diagnosis of AD (2 years and 3 years) and 2) generating a user-friendly calculator which estimates the probability of progression by entering subject-specific values of the small set of selected variables. Since patients interested in enrolling in anti-amyloid and other drug trials are likely to be referred to Alzheimer’s Disease Centers (ADCs) for in-depth diagnosis, pragmatic trials can be proposed by including participants who visited ADCs. The current study aims to provide the probability of conversion in clinical diagnosis with high accuracy using the NACC UDS V3 data. The calculator may be useful for pre-screening potential trial participants with high probabilities of clinical progressions before pursuing higher-cost biomarker assessments, such as an assessment of amyloid burden by a positron emission tomography (PET). The calculator can also be used for study enrichment for other trials and to strategize patients’ follow-up plans.

 

Methods

Data

NACC data

The NACC at the University of Washington maintains a repository of the UDS collected from participants in all of the National Institute on Aging (NIA)-funded Alzheimer’s Disease Centers (ADC) in the United States. There are over 30 past and present ADCs. The UDS consists of data collection protocols administered systematically to participants enrolled in each ADC (8-16). Participants are recruited, enrolled, and followed on an annual basis, generating center-specific longitudinal cohorts. These participants include individuals with clinical syndromic diagnoses of normal cognition (NC), MCI or cognitive impairments not meeting clinical MCI criteria, and dementia of various etiologies, including AD. Each AD Center enrolls participants in a NACC research cohort according to center-specific priorities. In general, most participants come from clinician referrals, self-referral by patients or family members, active recruitment through community organizations, and volunteers who wish to contribute to research. Most centers also enroll volunteers with normal cognition. NACC participants are not an epidemiologically based sample of the U.S. population with or without dementia. They are best regarded as a referral-based or volunteer case series selected based on each center’s research focus. Consent is obtained at the individual ADCs, as approved by their Institutional Review Boards (IRBs). The UDS data includes demographics, medical history, medication use, physical and neurological exam findings, clinical ratings of dementia severity [Clinical Dementia Rating (CDR®) Dementia Staging Instrument] (17), and neuropsychological test scores. Systematic guidelines for clinical diagnosis are based on the most up-to-date published diagnostic research criteria (8, 18-20). Further information on the NACC database may be found at: https://www.alz.washington.edu. Current analyses use data downloaded on March 12, 2021.

Analytical Approaches

We have two independent objectives in this study: MCI progression prediction and AD progression prediction. In the MCI progression prediction, we aimed to differentiate those who converted to clinical diagnosis of MCI (or AD without having a diagnosis of MCI) within 4 years from baseline normal cognition and those maintaining normal cognition for at least 4 years. For this prediction task, a cognitively normal subject converting to MCI at any time within the 4-year observation window is a positive case, and a cognitively normal subject remaining normal for at least 4 years is used as a negative case. Cognitively normal subjects remaining normal by the time of loss to follow-up before year 4 were excluded. We also examined the transition to a specific subtype of MCI, amnestic MCI (aMCI), again with a 4-year observation window. Due to the relatively small sample size of non-amnestic MCI subjects in this cohort, we did not examine the transition to non-amnestic MCI (naMCI). In predicting the progression to clinical diagnosis of Alzheimer’s disease (AD), we aimed to differentiate between those converting to AD from MCI and those maintaining MCI status, using two follow-up durations: observation windows of 2 years and 3 years. We used these durations because the majority of clinical trials recruiting MCI participants complete follow-ups within 36 months.

Data preparation

Considerable time and effort were devoted to making the data appropriate for the supervised modeling process. Baseline numerical patient characteristics are included and missing indicators that indicate reasons for missingness (e.g., 99, -4) were coded appropriately by creating a dummy variable that indicates each type of missingness if they are considered to contribute to the prediction. After data clean-up and preparation procedures and determination of cognitive status at each assessment time point (see Supplement for more information), we applied the models listed below.

Feature selection method & Classification method

We examined the sensitivity, specificity, and accuracy of predictions using Receiver Operating Characteristics Area Under Curve (AUC). In this study, classifiers included Support Vector Machine (SVM), Logistic Regression (LR), and Random Forests. We compared the performance of univariate feature selection methods, including Information Gain (InfoGain), chi-squared test (Chi2), Fisher Score, and Analysis of variance (ANOVA). Additionally, embedded methods, including LASSO and Decision Tree, which incorporate feature selection within the classifier construction, and determine the feature importance simultaneously (joint feature selection property), were also used.
Different numbers of clinical variables (henceforth we call “features”) were examined (2, 5, 10, 15, 20, 25, 30, 35, and 40) to compare model performance. The number of features is determined when AUC does not improve significantly after including additional features. In our pipeline of model performance evaluation, datasets were divided into test and training datasets. We examined each combination independently by 5-fold cross-validation on the training datasets to automatically select the proper combination of our final model according to average validation performance. The whole assessment process was repeated 10 times to compute the average performance across the 10 repetitions. The final model was tested on the test datasets and corresponding performance metrics were reported.

Diagnosis

Cognitive statuses assigned during annual visits included normal cognition, MCI, aMCI, and AD. This information was extracted from the NACC data and used to determine progressions (i.e., from normal to MCI and from MCI to AD). The variables used to determine the cognitive status including etiology are described in detail in Supplement A and Supplement Table 1.

 

Results

Table 1 summarizes the sample sizes used for each prediction. The largest sample size (number of unique participants) was for examining predictors of MCI to AD transition with 2 years (MCI2AD_2) because we could use samples with only 2 years of follow-up. The smallest sample size was for examining predictors of normal cognition to aMCI. Those who progressed to MCI within 4 years were reduced from 408 to 183 once we limited MCI incidence to aMCI.
Figure 1 shows that 20 selected clinical variables were the best candidate parameter for our prediction model in the cross-validation process. That is, further adding features beyond 20 did not improve performance significantly. Therefore, the performance metrics were assessed using 20 clinical variables in the following model constructions.

*: Need to remain the normal cognitive status for either 4 years (NC2MCI, NC2aMCI), MCI for either 3 years (MCI2AD_3) or 2 years (MCI2AD_2).

 

Prediction model performance

Tables 2 and 3 show results of the models’ test performances for each progression, including from normal cognition to MCI (NC2MCI) within 4 years, normal cognition to amnestic MCI within 4 years (NC2aMCI), MCI to AD within 2 years (MCI2AD_2) and MCI to AD within 3 years (MCI2AD_3). The AUC was lower for predicting MCI transition which was around 70%, compared with predicting AD transition, which was around 80%. For example, for the NC2aMCI transition, the combination using Random Forest as a classifier and ANOVA as a feature selection method reached the highest AUC with average 74.6% accuracy, 76.4% specificity, 71.6% sensitivity, and 74.0% AUC. For the MCI2AD_3 transition, the combination using Random Forest as a classifier and ANOVA as a feature selection method reached the best performance with an average of 82.1% accuracy, 75.9% specificity, 85.1% sensitivity, and 80.5% AUC. Overall, accuracy and AUC were the highest for predicting MCI to AD transition within 2 years (85% for accuracy as well as sensitivity and specificity).

Note: mean and std (standard deviation) are calculated based on the repeated assessment process; The model which had the highest AUC was bolded.

 

Selected Variables

Table 3 shows variables selected for transition models. Clinician’s judgment (the variable named COGSTAT in the NACC UDS V3 Data Dictionary) based on the neuropsychological examination was selected for all 4 transitions as one of the 20 variables. Supplemental Table 2 shows the distribution of responses for each variable by the transited vs. non-transited participants. Supplemental Table 2 can be read as follows: for example, the variable, COGSTAT (clinician’s judgement), was more likely to be coded as “1: better than normal for age” or “2: normal for age” at baseline for those who remained normal and the proportion of “3: one or two test scores abnormal” or “4: three or more scores are abnormal or lower than expected” increases as the duration of transition to conversion reduces from 3 years to 2 years, implying that those who converted to AD within a short duration showed more impairment at baseline, as expected. Transition to AD within 3 years was predicted mostly by neuropsychological tests and CDR (memory, community affairs and judgement subitems) while a transition to AD within 2 years was predicted more by variables indicating whether biomarkers including FDG-PET, amyloid PET, and CSF were assessed. We included a response variable that indicates these assessments were not conducted or unknown as a potential predictor variable, instead of removing it since the missing of this variable itself would be informative for predictions. 16% of those who progressed from MCI to AD within 2 years received FDG-PET assessment, while only 4% of those who did not transition to AD received the assessment (Supplemental Table 2.d). Interestingly, for the MCI2AD_2 transition, whether smoked a cigarette in the last 30 days was selected as one of the predictors, showing that 96% of those who remained MCI did not smoke while a lower % (72%) of those who transited to AD indicated they did not smoke.

Probability threshold which optimizes prediction for pre-screening or study enrichment

Although the higher the probability, the more likely the subject is going to transit, fewer subjects are selected as the probability increases. For example, based on our results, subjects with an estimated transitional probability of 0.9 (90%) for NC2MCI transition have a positive predictive value of 96.6%, indicating that they are almost guaranteed to transit to MCI within 4 years. However, a small fraction of our sample – 25.3% among the total samples used for NC2MCI prediction – has an estimated transitional probability of 0.9 or above. It is often a balance of how sure we would like to be in selecting those who progress in clinical symptoms vs. what proportion of subjects among the sampling pool can be selected for pre-screening or trial enrichment. We provided examples of sensitivity/specificity associated with each probability and how much % of subjects are selected (i.e., prevalence) in our total sampling pool in Figure. 2 (a-d). Examples of how to interpret the figures are described in the footnote.

Note: For example in (A), if we select subjects with predicted probability ranging from 0.7 to 1.0, we can achieve a sensitivity of 87.3%, a positive predictive value of 93.7%, and 41.4 % of participants in the test datasets have this probability. On the other hand, if we select subjects with a predicted probability of 0.9 and above, we can achieve a sensitivity of 99.1%, a positive predictive value of 96.6%, but only 25.3% of the test data sets have this probability.

 

Calculator

We developed a calculator that provides conversion probabilities by entering the person-specific information on the selected (in this study, 20) variables used in the NACC UDS V3. For each transition model, the hyper-parameters validated by the previous cross-validation were used to generate the prediction model. In order to generate probability output, we only used logistic regression as our classifier in the calculator. We applied bootstrap along with the classifier and dataset to estimate the mean value of the feature weights (i.e., the coefficient for each feature). And with the weights, we were able to estimate the probability of each new subject by using his or her information for the selected variables. The excel calculator can be downloaded at: https://www.shorturl.at/nHOY2. To aid in how to use the calculator, Figure 3 shows a screenshot of the calculator using the excel sheet used to estimate the probability of conversion from normal cognition to MCI within 4 years as an example. Please note that the pull-down menu functionality is available once it’s downloaded.

In the above example (screenshot), the excel sheet to estimate the probability of conversion from normal cognition to MCI within 4 years was selected. In this example, NACCCOGF variable (Clinician Judgment of Symptoms) was highlighted, and its response categories were shown in the pulldown menu. The variable names used in the calculator are the same as those used in the NACC code book so that the study coordinator can easily identify the targeted variable. The code book is located at: https://files.alz.washington.edu/documentation/uds3-rdd.pdf. In this example, given the data entered into this calculator, the subject is likely to convert to MCI within 4 years with 89.1% (the probability of the conversion). This probability is accurate with 95% of the simulated samples, implying that we have high confidence that this subject will convert to MCI within 4 years.

 

Discussion

The main aim of this study is to generate prediction models (prediction in terms of clinical diagnosis) and a calculator to identify those at high risk of progressing from normal cognition to MCI, and from MCI to AD within relatively short follow-up durations. We used an observation window of 4 years from normal to MCI conversion and 2 and 3 years for MCI to AD conversions, i.e., a common duration of follow-up in pharmacological trials. Our user-friendly calculator can be used for pre-selecting subjects for further assessment of in-vivo biomarkers or for enriching study participants in various trials. The clinical variables selected are all derived from the data collected uniformly across all NIH-funded Alzheimer’s Disease Centers in the United States (UDS V3). In tables and supplemental materials, we included variable names used in the NACC Data Dictionary which is downloadable from the website (https://www.naccdata.org) so that researchers can replicate or expand our studies. Although the subjects used for this study are not from an epidemiological cohort and not are representative of the general elderly population in the US, these subjects are likely to represent those interested in participating in AD and related dementia (ADRD) clinical trials.
Similar to our previous study using an older UDS version (UDS V2) (5), neuropsychological test results played important roles in predictions: MoCA (global cognition), Craft story (learning and delayed memory), Animal Fluency (language, language-based executive functions), Trail Making A (psychomotor speed), Multilingual Naming Test (naming) and Benton Figure Delayed Drawing (visuospatial) are important predictors. Detailed descriptions of these tests are explained previously (29). As expected, participants’ reports of their subjective cognitive decline (DECSUB) played roles in predicting transitions to MCI, but informants’ reports were more important for predicting transitions to AD, consistent with prior studies of these transitions (21).
An intriguing finding is that the variables consistently predicting transitions involved clinicians’ judgements of participants’ degree of cognitive (either memory or overall) impairments. This variable was consistently selected as an important predictor, both in our previous study (2), and in other studies (22). Clinicians’ judgements synthesize information that might not be necessarily reported in the UDS, such as the way respondents answer questions, facial expressions and speech utterances, as well as detailed information collected from informants/partners. Although we do not believe that machine learning-based approaches can fully replace clinical judgements, it might be possible to develop algorithms that resemble clinical judgement using information not reported in standard paper-pencil formats. For example, a large number of studies using speech characteristics (linguistic and acoustic) showed promising results in differentiating early-stage MCI subjects from those with normal cognition (23-27). Additionally, information about longitudinal medical history free from recall bias might be supplemented by Electronic Health Records (EHR), using modeling approaches similar to those used in this and other studies (28).
Many approaches were proposed to predict conversions using only clinical variables, just like our current study (30-34). The analytical approaches and the model performance have been extensively discussed in each study. We would like to emphasize that the choice of a prediction model also depends on the nature of the data the developed model will be applied to in practice, and its usefulness depends on the available data. For example, Bernier et al. used longitudinal “slopes” of cognitive decline and its deviation from the normative decline to predict conversions (33, 34). Using only one global cognitive test is advantageous, but their chart requires at least two data points to obtain the slopes. Our calculator uses data obtained in NACC UDS V3, which takes about 90 minutes or longer to complete and requires trained assessors and clinicians, i.e., is tasking. Therefore, our approach is practical when patients come from ADCs or memory clinics where NACC UDS is routinely used. We anticipate that patients who are interested in enrolling in anti-amyloid and other drug trials are likely to be referred to memory clinics. The trial study coordinator can enter the data into the calculator to estimate the likelihood of clinical progression within a short interval for the study enrichment and/or pre-screening for biomarker assessments.
There are limitations to this study. Each AD Center enrolls participants in the NACC research cohort according to center-specific priorities. The cohort used in this study is not a representative group of the elderly population in the USA. The participants are skewed towards higher income and education strata. Certain racial and ethnic groups in the general population at risk of MCI or AD were also underrepresented. The generalizability of our results is limited and the probability calculator may not be applicable to other cohorts. We used linear models and thus captured only the linear relationships between the predictor variables and target variables in this study.

 

Conclusion

Drawing on the most recent version (UDS V3) of uniformly collected data from all NIH-funded Alzheimer’s Disease Centers in the United States, we identified a small set of clinical variables which strongly predict transitions from normal cognition to MCI within 4 years and from MCI to AD within 2 and 3 years, updating our previous work (5). We developed a user-friendly conversion probability calculator that can be used for clinical trial enrichment and/or pre-screening for subsequent assessment of biological markers.

 

Acknowledgments: The NACC database is funded by NIA/NIH Grant U24 AG072122. NACC data are contributed by the NIA-funded ADRCs: P30 AG062429 (PI James Brewer, MD, PhD), P30 AG066468 (PI Oscar Lopez, MD), P30 AG062421 (PI Bradley Hyman, MD, PhD), P30 AG066509 (PI Thomas Grabowski, MD), P30 AG066514 (PI Mary Sano, PhD), P30 AG066530 (PI Helena Chui, MD), P30 AG066507 (PI Marilyn Albert, PhD), P30 AG066444 (PI John Morris, MD), P30 AG066518 (PI Jeffrey Kaye, MD), P30 AG066512 (PI Thomas Wisniewski, MD), P30 AG066462 (PI Scott Small, MD), P30 AG072979 (PI David Wolk, MD), P30 AG072972 (PI Charles DeCarli, MD), P30 AG072976 (PI Andrew Saykin, PsyD), P30 AG072975 (PI David Bennett, MD), P30 AG072978 (PI Neil Kowall, MD), P30 AG072977 (PI Robert Vassar, PhD), P30 AG066519 (PI Frank LaFerla, PhD), P30 AG062677 (PI Ronald Petersen, MD, PhD), P30 AG079280 (PI Eric Reiman, MD), P30 AG062422 (PI Gil Rabinovici, MD), P30 AG066511 (PI Allan Levey, MD, PhD), P30 AG072946 (PI Linda Van Eldik, PhD), P30 AG062715 (PI Sanjay Asthana, MD, FRCP), P30 AG072973 (PI Russell Swerdlow, MD), P30 AG066506 (PI Todd Golde, MD, PhD), P30 AG066508 (PI Stephen Strittmatter, MD, PhD), P30 AG066515 (PI Victor Henderson, MD, MS), P30 AG072947 (PI Suzanne Craft, PhD), P30 AG072931 (PI Henry Paulson, MD, PhD), P30 AG066546 (PI Sudha Seshadri, MD), P20 AG068024 (PI Erik Roberson, MD, PhD), P20 AG068053 (PI Justin Miller, PhD), P20 AG068077 (PI Gary Rosenberg, MD), P20 AG068082 (PI Angela Jefferson, PhD), P30 AG072958 (PI Heather Whitson, MD), P30 AG072959 (PI James Leverenz, MD). Additionally, the following specific NIH grant supported this study: RF1AG072449.

Author Contributions: Study concept and design: Dodge, Pang, Zhou. Acquisition of subjects and/or data: Kukull, Albin, Sano, Dodge. Analysis and interpretation of data: Pang, Zhou, Dodge. Preparation of manuscript: Dodge, Pang, Zhou. Critical revision of the manuscript: Sano, Albin, Kukull, Shen.

Conflict of interest: No author has a conflict of interest in reporting this study.

Open Access: This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

 

SUPPLEMENTARY MATERIAL1

 

References

1. Breitner JC. How can we really improve screening methods for AD prevention trials?. Alzheimer’s & Dementia: Translational Research & Clinical Interventions. 2016 Jan;2(1):45.
2. Kent DM, Hayward RA. Limitations of applying summary results of clinical trials to individual patients: the need for risk stratification. Jama. 2007 Sep 12;298(10):1209-12.
3. Angus DC, Chang CC. Heterogeneity of Treatment Effect: Estimating How the Effects of Interventions Vary Across Individuals. JAMA. 2021 Dec 14;326(22):2312-3.
4. Budd Haeberlein S, Aisen PS, Barkhof F, Chalkias S, Chen T, Cohen S, Dent G, Hansson O, Harrison K, von Hehn C, Iwatsubo T. Two randomized phase 3 studies of aducanumab in early Alzheimer’s disease. The Journal of Prevention of Alzheimer’s Disease. 2022 Apr;9(2):197-210.
5. Lin M, Gong P, Yang T, Ye J, Albin RL, Dodge HH. Big data analytical approaches to the NACC dataset: aiding preclinical trial enrichment. Alzheimer disease and associated disorders. 2018 Jan;32(1):18.
6. Dodge HH, Goldstein FC, Wakim NI, Gefen T, Teylan M, Chan KC, Kukull WA, Barnes LL, Giordani B, Hughes TM, Kramer JH. Differentiating among stages of cognitive impairment in aging: Version 3 of the Uniform Data Set (UDS) neuropsychological test battery and MoCA index scores. Alzheimer’s & Dementia: Translational Research & Clinical Interventions. 2020;6(1):e12103.
7. Sachs BC, Steenland K, Zhao L, Hughes TM, Weintraub S, Dodge HH, Barnes LL, Craft S, Parker ML, Goldstein FC. Expanded demographic norms for version 3 of the Alzheimer disease centers’ neuropsychological test battery in the uniform data set. Alzheimer disease and associated disorders. 2020 Jul;34(3):191.
8. Morris JC. The Clinical Dementia Rating (CDR): Current version and scoring rules. Neurology, 43(11):2412-2414, 1993. DOI: 10.1212/WNL.43.11.2412-a
9. Beekly DL, Ramos EM, Lee WW, Deitrich WD, Jacka ME, Wu J, Hubbard JL, Koepsell TD, Morris JC, Kukull WA. The National Alzheimer’s Coordinating Center (NACC) database: the uniform data set. Alzheimer Disease & Associated Disorders. 2007 Jul 1;21(3):249-58.
10. Weintraub S, Salmon D, Mercaldo N, Ferris S, Graff-Radford NR, Chui H, Cummings J, DeCarli C, Foster NL, Galasko D, Peskind E. The Alzheimer’s disease centers’ uniform data set (UDS): The neuropsychological test battery. Alzheimer disease and associated disorders. 2009 Apr;23(2):91.
11. Morris JC, Weintraub S, Chui HC, Cummings J, DeCarli C, Ferris S, Foster NL, Galasko D, Graff-Radford N, Peskind ER, Beekly D. The Uniform Data Set (UDS): clinical and cognitive variables and descriptive data from Alzheimer Disease Centers. Alzheimer Disease & Associated Disorders. 2006 Oct 1;20(4):210-6.
12. Jack Jr CR, Albert MS, Knopman DS, McKhann GM, Sperling RA, Carrillo MC, Thies B, Phelps CH. Introduction to the recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimer’s & Dementia. 2011 May 1;7(3):257-62.
13. McKhann GM, Knopman DS, Chertkow H, Hyman BT, Jack Jr CR, Kawas CH, Klunk WE, Koroshetz WJ, Manly JJ, Mayeux R, Mohs RC. The diagnosis of dementia due to Alzheimer’s disease: Recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimer’s & Dementia. 2011 May 1;7(3):263-9.
14. Albert MS, DeKosky ST, Dickson D, Dubois B, Feldman HH, Fox NC, Gamst A, Holtzman DM, Jagust WJ, Petersen RC, Snyder PJ. The diagnosis of mild cognitive impairment due to Alzheimer’s disease: recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimer’s & Dementia. 2011 May;7(3):270-9.
15. Sperling RA, Aisen PS, Beckett LA, Bennett DA, Craft S, Fagan AM, Iwatsubo T, Jack Jr CR, Kaye J, Montine TJ, Park DC. Toward defining the preclinical stages of Alzheimer’s disease: Recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimer’s & Dementia. 2011 May 1;7(3):280-92.
16. Beekly DL, Ramos EM, Lee WW, Deitrich WD, Jacka ME, Wu J, Hubbard JL, Koepsell TD, Morris JC, Kukull WA. The National Alzheimer’s Coordinating Center (NACC) database: the uniform data set. Alzheimer Disease & Associated Disorders. 2007 Jul 1;21(3):249-58.
17. Shirk SD, Mitchell MB, Shaughnessy LW, et al. A web-based normative calculator for the Uniform Data Set (UDS) neuropsychological test battery. Alzheimers Res Ther, 3(6):32, 2011. DOI: 10.1186/alzrt94
18. National Alzheimer’s Coordinating Center. NACC Uniform Data Set Data Element Dictionary For Initial Visit Packet. 2015. Available at: www.alz.washington.edu/NONMEMBER/ UDS/DOCS/VER3/ivp_ded.pdf. Accessed April 15, 2018. 11.
19. National Alzheimer’s Coordinating Center. NACC Uniform Data Set Coding Guidebook For Initial Visit Packet. 2015. Available at: www.alz.washington.edu/NONMEMBER/UDS/DOCS/VER3/ UDS3_ivp_guidebook.pdf. Accessed April 14, 2018. 12.
20. National Alzheimer’s Coordinating Center. NACC Uniform Data Set Instructions for the Neuropsychological Battery (Form C2). 2015. Available at: www.alz.washington.edu/NONMEMBER/ UDS/DOCS/VER3/UDS3_npsych_instructions_C2.pdf. Accessed April 14, 2018.
21. Caselli RJ, Chen K, Locke DE, Lee W, Roontiva A, Bandy D, Fleisher AS, Reiman EM. Subjective cognitive decline: self and informant comparisons. Alzheimers Dement. 2014 Jan;10(1):93-8. doi: 10.1016/j.jalz.2013.01.003. Epub 2013 Apr 3. PMID: 23562429; PMCID: PMC3732500.
22. James C, Ranson JM, Everson R, Llewellyn DJ. Performance of Machine Learning Algorithms for Predicting Progression to Dementia in Memory Clinic Patients. JAMA Netw Open. 2021;4(12):e2136553. doi:10.1001/jamanetworkopen.2021.36553
23. Tang F, Xiao C, Wang F, Zhou J. Predictive modeling in urgent care: a comparative study of machine learning approaches. Jamia Open. 2018 Jul;1(1):87-98
24. Tang F, Uchendu I, Wang F, Dodge HH, Zhou J. Scalable diagnostic screening of mild cognitive impairment using AI dialogue agent. Scientific Reports, 2020; 10(1): 5732. doi: 10.1038/s41598-020-61994-0. PMCID: PMC7109153
25. Asgari M, Kaye JA, Dodge HH: Predicting Mild Cognitive Impairment (MCI) from Spontaneous Spoken Utterances Alzheimer’s & Dementia: Translational Research & Clinical Interventions 3: 219-228, 2017. PMCID: PMC5651423
26. Chen L, Asgari M, Gale R, Wild K, Dodge H, Jeffrey Kaye. Improving the Assessment of Mild Cognitive Impairment in Advanced Age with a Novel Multi-Feature Automated Speech and Language Analysis of Verbal Fluency. Frontiers in Psychology, section Psychology for Clinical Settings 2020 Apr 9;11:535. doi: 10.3389/fpsyg.2020.00535. eCollection 2020 (PMC7160369).
27. Chen, L, Dodge, HH, Asgari M. (2020). Topic-Based Measures of Conversation for Detecting Mild Cognitive Impairment. Proceedings of the First Workshop on Natural Language Processing for Medical Conversations, 2020, 63. 10.18653/v1/2020.nlpmc-1.9. PMCID: PMC7909094
28. Zhang XS, Tang F, Dodge HH, Zhou d J, Wang F. MetaPred: Meta-Learning for Clinical Risk Prediction with Limited Patient Electronic Health Records. KDD ‘19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining July 2019 Pages 2487–2495 https://doi.org/10.1145/3292500.3330779 PMCID: PMC8046258
29. Weintraub S, Besser L, Dodge HH, Teylan M, Ferris S, Goldstein FC, Giordani B, Kramer J, Loewenstein D, Marson D, Mungas D, Salmon D, Welsh-Bohmer K, Zhou XH, Shirk SD, Atri A, Kukull WA, Phelps C, Morris JC: Version 3 of the Alzheimer Disease Centers’ Neuropsychological Test Battery in the Uniform Data Set (UDS): Alzheimer disease and associated disorders 32(1)10-17, 2018. PMCID:(PMC5821520.
30. Reinke, C, Doblhammer, G, Schmid, M, Welchowski, T. Dementia risk predictions from German claims data using methods of machine learning. Alzheimer’s & Dementia. 2022; 1– 10. https://doi.org/10.1002/alz.12663
31. Guest, F., Kuzma, E., Everson, R. and Llewellyn, D.J. (2020), Identifying key features for dementia diagnosis using machine learning. Alzheimer’s & Dementia, 16: e046092. https://doi.org/10.1002/alz.046092
32. Albright, J., Ashford, M.T., Jin, C., Neuhaus, J., Camacho, M.R., Rabinovici, G.D., Truran-Sacrey, D., Maruff, P.T., Mackin, R.S., Nosheny, R.L. and Weiner, M.W. (2021), Machine learning approaches to predicting amyloid status using data from an online research and recruitment registry: The Brain Health Registry. Alzheimer’s & Dementia., 17: e053330. https://doi.org/10.1002/alz.053330
33. Bernier, PJ, Gourdeau, C, Carmichael, P-H, et al. It’s all about cognitive trajectory: accuracy of the cognitive charts–MoCA in normal aging, MCI, and dementia. J Am Geriatr Soc. 2022; 1- 7. doi:10.1111/jgs.18029
34. Bernier PJ, Gourdeau C, Carmichael PH, Beauchemin JP, Verreault R, Bouchard RW, Kröger E, Laforce R Jr. Validation and diagnostic accuracy of predictive curves for age-associated longitudinal cognitive decline in older adults. CMAJ. 2017 Dec 4;189(48):E1472-E1480. doi: 10.1503/cmaj.160792. PMID: 29203616; PMCID: PMC5714699.

© The Authors 2023