Main Outcome Measures
Abbreviations and Acronyms:AUC (area under the receiver operating characteristic curve), CI (confidence interval), DL (deep learning), DR (diabetic retinopathy), GON (glaucomatous optic neuropathy), ICD (International Classification of Diseases), ISNT (inferior > superior > nasal > temporal), ONH (optic nerve head), PPA (parapapillary atrophy), RNFL (retinal nerve fiber layer), VA (Veterans Affairs), VF (visual field)
Development Dataset Grading
Referable Glaucomatous Optic Neuropathy Grading
Development of the Algorithm
- Szegedy C.
- Vanhouke V.
- Ioffe S.
- et al.
Abadi M, Agarwal A, Barham P, et al. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Available at: tensorflow.org.
|Training Dataset||Tuning Dataset||Validation Dataset A||Validation Dataset B (Veterans Affairs Atlanta)||Validation Dataset C (Dr. Shroff)|
|No. of images (1 image per patient)||86 618||1508||1205||9642||346|
|No. of graders||43 graders: 14 glaucoma specialists, 26 ophthalmologists, and 3 optometrists||11 glaucoma specialists||12 glaucoma specialists||6 ophthalmologists||4 glaucoma specialists|
|No. of grades per image||1–2||3||3–6||1||1|
|Grades per grader, median (IQR)||861 (225–2952)||388 (96–553)||185 (93–731)||N/A||NA|
|Data used to define the reference standard||45° fundus photograph||45° fundus photograph||45° fundus photograph||Health factors and ICD codes||Full glaucoma workup including VF and OCT|
|No. of patients||86 618||1508||1205||9642||346|
|Age (yrs), median (IQR)||57 (49–65)||57 (49–64.8)||57 (49.5–64)||64 (68.7–57.5)||N/A|
|No. of images for which age was available||64 861||1386||1115||9642|
|Female gender, no./total no. (%) of images for which gender was known||32 413/62 178 (52.1)||731/1349 (54.2)||571/1075 (53.2)||458/9642 (4.7)||N/A|
|Glaucoma/GON gradability distribution|
|Images gradable for glaucoma, no./total (%) among images for which glaucoma gradability was assessed||74 973/86 127 (87.0)||1451/1508 (96.2)||1171/1204 (97.3)||9642/9642 (100)|
|Glaucoma/GON risk distribution|
|No. nonglaucomatous (%)||35 877 (47.8)||849 (57.1)||687 (57.5)||8753 (90.8)||63 (18.2)|
|No. low-risk glaucoma suspect (%)||20 740 (27.6)||259 (17.4)||290 (24.3)||N/A||N/A|
|No. high-risk glaucoma suspect (%)||13 180 (17.5)||268 (18.0)||170 (14.1)||N/A||175 (50.6)|
|No. likely glaucoma (%)||5307 (7.1)||110 (7.4)||48 (4.0)||890 (9.2)||108 (31.2)|
|No. referable glaucoma (%)||18 487 (24.6)||378 (25.4)||218 (18.1)||890 (9.2)||283 (81.7)|
Clinical Validation Datasets
|Feature||Area under the Receiver Operating Characteristic Curve (95% Confidence Interval)||No. of Labeled Images||Prevalence (%)||Binary Cutoffs|
|I vs. S||0.661 (0.594–0.722)||1162||8.2||I < S vs. I > S or I ≅ S|
|S vs. T||0.946 (0.897–0.981)||1156||1.6||S < T vs. S > T or S ≅ T|
|Notch||0.908 (0.852–0.956)||1162||2.6||Yes/possible vs. no|
|Laminar dot sign||0.950 (0.937–0.963)||1013||24||Yes/possible vs. no|
|Emerging||0.973 (0.954–0.987)||1166||4.7||Yes vs. possible/no|
|Directed||0.957 (0.944–0.969)||1167||15.9||Yes vs. possible/no|
|Baring of circumlinear vessels||0.723 (0.688–0.755)||1154||22.7||Present and clearly bared vs. all else|
|Disc hemorrhage||0.758 (0.666–0.844)||1173||2.1||Yes/possible vs. no|
|β PPA||0.933 (0.914–0.948)||1170||16.9||Yes/possible vs. no|
|RNFL defect||0.778 (0.706–0.843)||973||6.5||Yes/possible vs. no|
|Vertical CDR||0.922 (0.869-0.963)||1154||4.6||≥0.7 vs. <0.7|
Evaluating the Algorithm
Evaluating Optic Nerve Head Feature Importance
|Feature||Reference Standard||Algorithm Predictions||Round 1 Majority|
|Odds Ratio||P Value||Rank||Odds Ratio||P Value||Rank||Odds Ratio||P Value||Rank|
|Vertical CDR ≥0.7||581.671||<0.001||1||347.861||<0.001||1||475.757||<0.001||1|
|Notch: possible or yes||29.438||<0.001||2||9.564||0.021||3||4.158||0.218||4|
|RNFL defect: possible or yes||10.740||<0.001||3||13.098||<0.001||2||12.946||<0.001||2|
|Circumlinear vessels: present + bared||4.728||<0.001||4||6.241||<0.001||4||4.852||<0.001||3|
|Laminar dot: possible or yes||3.594||<0.001||5||3.320||<0.001||7||3.882||<0.001||6|
|Disc hemorrhage: possible or yes||3.221||0.043||6||2.178||0.369||9||1.649||0.508||9|
|Nasalization emerging: yes||3.162||0.008||7||4.253||0.001||5||4.014||<0.001||5|
|Rim comparison: I < S||2.560||0.004||8||3.512||<0.001||6||3.033||<0.001||7|
|Nasalization directed: yes||2.230||0.002||9||3.010||<0.001||8||2.239||0.001||8|
|Rim comparison: S < T||1.461||0.799||10||1.257||0.894||11||1.175||0.919||11|
|β PPA: possible or yes||1.319||0.226||11||1.584||0.076||10||1.357||0.192||10|
Performance of the Algorithm
Optic Nerve Head Features Analysis
Limitations and Future Work
- Table S1
- Table S2
- Table S3
- Table S4
- Table S5
- Table S6
- Fig S1
- Fig S2
- Fig S3
- Supplemental Material
- The number of people with glaucoma worldwide in 2010 and 2020.Br J Ophthalmol. 2006; 90: 262-267
- Global prevalence of glaucoma and projections of glaucoma burden through 2040: a systematic review and meta-analysis.Ophthalmology. 2014; 121: 2081-2090
- Managing glaucoma in developing countries.Arq Bras Oftalmol. 2011; 74: 83-84
- Temba Glaucoma Study: a population-based cross-sectional survey in urban South Africa.Ophthalmology. 2003; 110: 376-382
- Awareness of incident open-angle glaucoma in a population study.Ophthalmology. 2007; 114: 1816-1821
- Primary Open-Angle Glaucoma Suspect Preferred Practice Pattern(®) Guidelines.Ophthalmology. 2016; 123: P112-P151
- Glaucoma Screening.Kugler Publications, Amsterdam, The Netherlands2008
- Gaps in glaucoma care: a systematic review of monoscopic disc photos to screen for glaucoma.Expert Rev Ophthalmol. 2014; 9: 467-474
- Digital ocular fundus imaging: a review.Ophthalmologica. 2011; 226: 161-181
- Telemedicine for detecting diabetic retinopathy: a systematic review and meta-analysis.Br J Ophthalmol. 2015; 99: 823-831
- Primary open-angle glaucoma.Lancet. 2004; 363: 1711-1720
- Evaluating the optic disc and retinal nerve fiber layer in glaucoma. I: clinical examination and photographic methods.Semin Ophthalmol. 2000; 15: 194-205
- The pathophysiology and treatment of glaucoma.JAMA. 2014; 311: 1901-1911
- Primary Open-Angle Glaucoma Preferred Practice Pattern(®) Guidelines.Ophthalmology. 2016; 123: P41-P111
- Do findings on routine examination identify patients at risk for primary open-angle glaucoma?.JAMA. 2013; 309: 2035
- [Healthy optic discs with large cups—a diagnostic challenge in glaucoma].Klin Monbl Augenheilkd. 2006; 223: 308-314
- Shape of the neuroretinal rim and position of the central retinal vessels in glaucoma.Br J Ophthalmol. 1994; 78: 99-102
- Localized retinal nerve fiber layer defects in nonglaucomatous optic nerve atrophy.Graefes Arch Clin Exp Ophthalmol. 1994; 232: 759-760
- Retinal nerve fiber layer defect as an early manifestation of diabetic retinopathy.Ophthalmology. 1993; 100: 1147-1151
- Nerve fiber bundle visual field defect resulting from a giant peripapillary cotton-wool spot.J Neuroophthalmol. 2001; 21: 276-277
- Baring of a circumlinear vessel in glaucoma.Arch Ophthalmol. 1983; 101: 739-744
- Five rules to evaluate the optic disc and retinal nerve fiber layer for glaucoma.Optometry. 2005; 76: 661-668
- Deep learning.Nature. 2015; 521: 436-444
- Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.JAMA. 2016; 316: 2402-2410
- Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy.Ophthalmology. 2018; 125: 1264-1272
- Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes.JAMA. 2017; 318: 2211-2223
- Using a deep learning algorithm and integrated gradients explanation to assist grading for diabetic retinopathy.Ophthalmology. 2019; 126: 552-564
- A deep learning-based algorithm identifies glaucomatous discs using monoscopic fundus photographs.Ophthalmol Glaucoma. 2018; 1: 15-22
- Efficacy of a deep learning system for detecting glaucomatous optic neuropathy based on color fundus photographs.Ophthalmology. 2018; 125: 1199-1206
- Development of a deep residual learning algorithm to screen for glaucoma from fundus photography.Sci Rep. 2018; 8: 14665
- Performance of deep learning architectures and transfer learning for detecting glaucomatous optic neuropathy in fundus photographs.Sci Rep. 2018; 8: 16685
- The effectiveness of teleglaucoma versus in-patient examination for glaucoma screening: a systematic review and meta-analysis.PLoS One. 2014; 9e113779
- Welcome to EyePACS.2018. Accessed 5.12.18)
- Home page.Accessed 5.12.18)
- The Age-Related Eye Disease Study (AREDS): design implications. AREDS report no. 1.Control Clin Trials. 1999; 20: 573-600
- About UK Biobank.Accessed 5.12.18)
- Analysis of neuroretinal rim distribution and vascular pattern in eyes with presumed large physiological cupping: a comparative study.BMC Ophthalmol. 2014; 14: 72
- The lamina cribrosa and visual field defects in open-angle glaucoma.Can J Ophthalmol. 1983; 18: 124-126
- The ISNT rule: how often does it apply to disc photographs and retinal nerve fiber layer measurements in the normal population?.Am J Ophthalmol. 2017; 184: 19-27
- Rethinking the inception architecture for computer vision.2015. Accessed 3.12.18)
Abadi M, Agarwal A, Barham P, et al. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Available at: tensorflow.org.
- Imagenet classification with deep convolutional neural networks.in: Bartlett P. Advances in Neural Information Processing Systems. Curran Associates, Inc, RedHook, NY2012: 1097-1105
- Synthesis Lectures on Artificial Intelligence and Machine Learning.. 2012; 6: 1-114
- Popular ensemble methods: an empirical study.J Artif Intell Res. 1999; 11: 169-198
- Mathematical statistics with resampling and R.Wiley, Hoboken, NJ2018
- The use of confidence or fiducial limits illustrated in the case of the binomial.Biometrika. 1934; 26: 404
- The Kolmogorov-Smirnov test for goodness of fit.J Am Stat Assoc. 1951; 46: 68
- Content Analysis: An Introduction to Its Methodology.SAGE Publications, Thousand Oaks, CA2018
- Baring of the circumlinear vessel. An early sign of optic nerve damage.Arch Ophthalmol. 1980; 98: 865-869
Susanna R Jr, Medeiros FA. The Optic Nerve in Glaucoma. 2nd ed. Amsterdam, the Netherlands; 2006.
- Intraobserver and interobserver agreement in measurement of optic disc characteristics.Ophthalmology. 1988; 95: 350-356
- Expert agreement in evaluating the optic disc for glaucoma.Ophthalmology. 1992; 99: 215-221
- Racial differences in optic disc topography: baseline results from the confocal scanning laser ophthalmoscopy ancillary study to the ocular hypertension treatment study.Arch Ophthalmol. 2004; 122: 22-28
- Ethnic variation in optic disc size by fundus photography.Curr Eye Res. 2013; 38: 1142-1147
- Detecting structural progression in glaucoma with optical coherence tomography.Ophthalmology. 2017; 124: S57-S65
- Hybrid deep learning on single wide-field optical coherence tomography scans accurately classifies glaucoma suspects.J Glaucoma. 2017; 26: 1086-1094
- Using deep learning and transfer learning to accurately diagnose early-onset glaucoma from macular optical coherence tomography images.Am J Ophthalmol. 2019; 198: 136-145
Supplemental material available at www.aaojournal.org.
Financial Disclosure(s): The author(s) have made the following disclosure(s): S.P.: Employee and equity owner – Google LLC (Mountain View, CA).
R.C.D.: Employee and equity owner – Google LLC (Mountain View, CA).
N.H.: Employee and equity owner – Google LLC (Mountain View, CA).
Y.L.: Employee and Equity owner – Google LLC (Mountain View, CA); Patent – WO2018035473A3 (Processing fundus images using machine learning models).
J.K.: Employee and equity owner – Google LLC (Mountain View, CA).
N.K.: Employee and equity owner – Google LLC (Mountain View, CA).
M.S.: Employee – Google LLC (Mountain View, CA).
R.S.: Employee and equity owner – Google LLC (Mountain View, CA).
D.J.W.: Employee and equity owner – Google LLC (Mountain View, CA).
A.B.: Employee and equity owner – Google LLC (Mountain View, CA).
C.S.: Employee and equity owner – Google LLC (Mountain View, CA).
A.M.: Employee and equity owner – Google LLC (Mountain View, CA).
A.E.H.: Consultant – Google LLC (Mountain View, CA); Employee – Advanced Clinical (Deerfield, IL).
F.A.M.: Consultant – Carl-Zeiss Meditec, Inc., Reichert, Inc., Allergan, Novartis, Quark Pharmaceuticals, Stealth Biotherapeutics, Galimedix Therapeutics, Inc., Biozeus, Inc., Ngoggle, Inc.; Financial support – Carl-Zeiss Meditec, Heidelberg Engineering, Reichert, Dyopsys, Inc., Google LLC (Mountain View, CA)
G.S.C.: Employee and equity owner – Google LLC (Mountain View, CA); Patent – WO2018035473A3 (Processing fundus images using machine learning models).
L.P.: Employee and equity owner – Google LLC (Mountain View, CA); Patent – WO2018035473A3 (Processing fundus images using machine learning models).
D.R.W.: Employee and equity owner – Google LLC (Mountain View, CA); Patent – WO2018035473A3 (Processing fundus images using machine learning models).
Google LLC, Mountain View, California, funded this study and had a role in its approval for publication. This research was conducted using the United Kingdom Biobank Resource under application number 17643. Some images used for the analyses described in this manuscript were obtained from the National Eye Institute Study of Age-Related Macular Degeneration (NEI-AMD) database (https://www.ncbi.nlm.nih.gov/projects/gap/cgi-bin/study.cgi?study_id=phs000001.v3.p1) through dbGaP accession number phs000001.v3.p1.c1. Funding support for the NEI-AMD was provided by the National Eye Institute , National Institutes of Health , Bethesda, Maryland (grant no.: N01-EY-0-2127 ).
HUMAN SUBJECTS: Human subjects were included in this study. All images were de-identified according to HIPAA Safe Harbor prior to transfer to the study investigators. Ethics review and Institutional Review Board exemption were obtained using Quorum Review Institutional Review Board. All research complied with the Health Insurance Portability and Accountability Act of 1996.
No animal subjects were included in this study.
Conception and design: Phene, Dunn, Hammel, Liu, Krause, Huang, Spitze
Analysis and interpretation: Phene, Dunn, Hammel, Liu, Krause, Kitade, Sayres, Wu, Bora, Semturs, Schaekermann, Huang, Medeiros
Data collection: Phene, Dunn, Hammel, Liu, Kitade, Schaekermann, Misra, Huang, Spitze, Maa, Gandhi, Corrado, Peng, Webster
Obtained funding: Phene, Dunn, Hammel, Liu, Krause, Kitade, Schaekermann, Sayres, Wu, Bora, Semturs, Misra, Medeiros, Corrado, Peng, Webster
Overall responsibility: Phene, Dunn, Hammel, Liu, Krause, Kitade, Schaekermann, Sayres, Wu, Bora, Semturs, Misra, Huang, Spitze, Medeiros, Maa, Gandhi, Corrado, Peng, Webster
User LicenseCreative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0) |
For non-commercial purposes:
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article (private use only, not for distribution)
- Reuse portions or extracts from the article in other works
- Sell or re-use for commercial purposes
- Distribute translations or adaptations of the article
Elsevier's open access license policy