When we, a 71-year-old white woman and a 60-year-old white man, both nonsmokers, have our eyes examined, we will want to know whether we classify as Age-Related Eye Disease Study (AREDS) category 3, and therefore would likely benefit from taking the AREDS supplement, which contains high-dose antioxidants (β-carotene, vitamin C, and vitamin E) plus high-dose zinc. As a statistician (J.W.) who chaired the AREDS and AREDS2 Data Safety Monitoring Committee and an epidemiologist (D.C.M.) who recently wrote an editorial about the results of the AREDS2 trial,
1
we are interested in the interchange between Awh et al2
, 3
(see article on page 162) and Chew et al4
, 5
(see article on page 212) not only professionally but also personally. If we indeed are at risk for development of neovascular age-related macular degeneration (AMD), we want to know whether we should follow the recommendations of Awh et al3
and undergo genetic testing to determine what supplements we should take or whether we should follow the guidance of Chew et al5
and take AREDS supplements without engaging in genetic testing.We rely on 5 articles to inform our decision. All use data from the AREDS, a randomized, masked, clinical trial with 4757 participants assigned to 1 of 4 treatment groups: antioxidants and zinc placebo, zinc and antioxidant placebo, both antioxidants and zinc, or double placebo. First, Klein et al
6
showed data from the AREDS suggesting that genetic variants may have differential response to treatments with antioxidants and zinc; however, the authors concluded that all variants of the genotypes they studied showed some benefit of AREDS supplements. Therefore, they did not recommend genetic screening. Their analysis was based on genetic data from 876 white participants of the AREDS trial who at baseline had intermediate AMD or unilateral advanced AMD.Five years later, Awh et al
2
analyzed AREDS data but reached different conclusions. These authors studied 995 white AREDS participants who had category 3 AMD in 1 eye and category 1 to 4 AMD in the fellow eye, and for whom peripheral blood-derived DNA was available. The authors considered 9 sets of risk alleles defined by complement factor H and age-related maculopathy sensitivity 2. They concluded that the best supplement (antioxidants, zinc, both, or neither) depends on the individual's genotype (Table 1).Table 1Risk Alleles and Recommended Treatment by Genotype Group
2
, 3
| Risk Alleles | % of Sample | No. in Sample | Best AREDS Treatment (Awh et al 2 ) | Awh et al 3 Genotype Group | |
|---|---|---|---|---|---|
| CHF | ARMS2 | ||||
| 0 | 0 | 5.9 | 58 | No supplement | 1 |
| 0 | 1 | 5.3 | 52 | Zinc | 3 |
| 0 | 2 | 1.0 | 10 | Zinc | 3 |
| 1 | 0 | 22.5 | 224 | Antioxidants | 1 |
| 1 | 1 | 22.6 | 225 | Antioxidants + zinc | 3 |
| 1 | 2 | 6.6 | 65 | Zinc | 3 |
| 2 | 0 | 13.3 | 132 | Antioxidants | 2 |
| 2 | 1 | 16.4 | 163 | No supplement | 4 |
| 2 | 2 | 6.7 | 66 | No supplement | 4 |
AREDS = Age-Related Eye Disease Study; ARMS2 = age-related maculopathy sensitivity 2; CHF = complement factor H.
In an article countering those findings, Chew et al
4
tested for evidence of a difference in the effect of treatment by genotype. The interaction P value for testing whether people's response to the AREDS supplement differs by genotype was not significant when they looked at the genotypes described by Klein et al6
(P = 0.06) or those described by Awh et al2
(P = 0.45). Therefore, they concluded that AREDS supplementation reduces the rate of progression of AMD across all genotype groups.Awh et al
3
disputed the conclusions of Chew et al,4
criticizing their division of the AREDS population. Moreover, Awh et al3
presented data that the authors claimed show clinically important differences by genotype in the effect of antioxidants and zinc.Now, Chew et al
5
have responded, criticizing Awh et al3
for presenting results that are biased and therefore difficult to interpret. The most important contribution of the article by Chew et al5
is the use of newly available genetic data from the AREDS. In applying grouping of genotypes in the study by Awh et al,3
Chew et al5
found that every group shows benefit of AREDS supplements.In sorting out these conflicting articles, we are not going to dwell on some issues we have with them. We, as do Awh et al,
3
find the presentation by Chew et al4
of 27 comparisons, many with small sample sizes, not clinically interpretable. Our more important agreement is with Chew et al,4
who point out that Awh et al2
do not correct adequately for multiple statistical testing and use post hoc exploration of subgroups. Correcting for multiplicity is necessary in dealing with subgroups7
—shooting many arrows at a target improves the chances for even a poor archer; however, it is not clear how to deal with multiplicity in this case because Awh et al2
did not report how many subgroups were examined or which genotypes, if any, were prespecified and which were post hoc. Post hoc exploration of subgroups is treacherous because one never knows whether the results are due to inadvertent selection bias or if they are “true.”8
Many people search post hoc for subgroup effects. Such “data dredging” rewards the searcher with a high probability of finding differences between subgroups, but the probability is high that the finding is spurious. Replication is the only reliable way of assessing the validity of a post hoc finding.Awh et al
2
develop regression models to identify predictive variables; however, retrospective fitting of data is, at best, hypothesis-generating; the real test of a post hoc exploratory analysis is to find an independent data set to serve to validate the finding. As noted by Awh et al,2
“Given the absence (to our knowledge) of an existing dataset appropriate for validation studies, confirmation of our findings in a different cohort is unlikely in the near future.”Fortunately, such a dataset has emerged from the AREDS itself. Awh et al
2
, 3
used a dataset of 995 white subjects with AREDS category 3 disease in at least 1 eye who had available DNA. Chew et al5
had access to an additional 526 patients from the AREDS with the same qualifications whose DNA has recently become available. The demographics of the test dataset (Awh et al2
, 3
) and the validation set (Chew et al5
) are nearly the same. Chew et al5
looked at the 4 genetic groups developed by Awh et al.3
Had the validation sample by Chew et al5
replicated the findings by Awh et al,3
then despite all the potential biases inherent in the analyses by Awh et al, we would have considered the recommendation to have genetic testing reasonable. In contrast to the finding by Awh et al,3
however, Chew et al5
found that the AREDS formulation (antioxidants plus zinc) showed benefit in delaying the development of neovascular AMD in all 4 genetic subgroups.To understand what happened, consider Table 1. The first 3 columns and the fifth column are from Table 6 in the article by Awh et al,
2
the fourth column is calculated, and the last column presents the genetic groups assigned by Awh et al.3
Awh et al
3
took the 9 genotypes identified in their earlier article,2
defined 4 categories on the basis of outcome (Table 1, last column), and fit statistical models that identify which supplements (antioxidants, zinc, or both) each of the 4 groups should use. It does not take a statistician or epidemiologist to forecast that the model will predict the results shown in Table 2. Yusuf et al7
point to subgroups defined by post-randomization variables as “improper” and not amenable to unbiased statistical testing. We distinguish between subgroups identified post hoc, but defined on the basis of baseline (i.e., pre-randomization) variables from the more problematic improper subgroups, whether defined a priori or post hoc, that are based on post-randomization variables. The genetic subgroups in the report by Awh et al3
are both post hoc and improper. The approach that Awh et al3
used in defining their genotype subgroups is circular: After selecting post-randomization outcomes observed by Awh et al2
to define subgroups in the study by Awh et al,3
they then tested those subgroups with the very same data.Table 2Best Treatment
3
by Genetic Group as Deduced from Examination of Table 1 and Chosen by Statistical Modeling2
| Genotype Group | Best Treatment | |
|---|---|---|
| Awh et al's 2 Conclusion | Awh et al's 3 Conclusion | |
| 1 | Antioxidants | Antioxidants only |
| 2 | Antioxidants | Antioxidants only; avoid zinc |
| 3 | Zinc (perhaps with antioxidants) | Zinc plus antioxidants (the AREDS supplement) |
| 4 | No supplement | No supplement |
AREDS = Age-Related Eye Disease Study.
The theoretical physicist Eugene Wigner describes a class Einstein taught in 1928. “He told us once: ‘Life is finite. Time is infinite. The probability that I am alive today is zero. In spite of this, I am now alive. Now how is that?’ None of his students had an answer. After a pause, Einstein said, ‘Well, after the fact, one should not ask for probabilities.’”
8
Calculating probabilities after the fact is exactly what Awh et al3
did. Awh et al2
identified 9 genotypes defined by 0, 1, or 2 risk alleles in each of 2 genes. After the fact, they performed modeling, looking for P values, which are probabilities. Awh et al2
raised some hypotheses about genotype and treatment with the AREDS supplement. The validation study by Chew et al5
showed no evidence that those hypotheses were true. Thus, this statistician and this epidemiologist, if classified as having an AREDS class 3 risk status, hope that our ophthalmologists will not recommend genotyping us when deciding whether to recommend AREDS supplementation. Indeed, should our ophthalmologist send us for genotyping, we will use the recommendation as an opportunity for warning about the inferential perils of subgroup analyses, especially those that (similar to the study by Awh et al3
) are based on outcomes.References
- Editorial: evidence for including lutein and zeaxanthin in oral supplements for age-related macular degeneration.JAMA Ophthalmol. 2014; 132: 139-141
- CFH and ARMS2 genetic polymorphism predict response to antioxidants and zinc in patients with age-related macular degeneration.Ophthalmology. 2013; 120: 2317-2323
- Treatment response to antioxidants and zinc based on CFH and ARMS2 genetic risk allele number in the Age-Related Eye Disease Study.Ophthalmology. 2015; 122: 162-169
- No clinically significant association between CFH and ARMS2 genotypes and response to nutritional supplements: AREDS Report Number 38.Ophthalmology. 2014; 121: 2173-2180
- Genetic testing in persons with age-related macular degeneration and the use of AREDS supplements: to test or not to test?.Ophthalmology. 2015; 122: 212-214
- CFH and LOC3877151/ARMS2 genotypes and treatment with antioxidants and zinc for age-related macular degeneration.Ophthalmology. 2008; 115: 1019-1025
- Analysis and interpretation of treatment effects in subgroups of patients in randomized clinical trials.JAMA. 1991; 266: 93-98
- The Recollections of Eugene P. Wigner as Told to Andrew Szanton.Plenum, New York1992
Article Info
Footnotes
Should our ophthalmologist send us for genotyping, we will use the recommendation as an opportunity for warning about the inferential perils of subgroup analyses…
Financial Disclosure(s): The author(s) have made the following disclosure(s): J.W.: Chaired – Data and Safety Monitoring Committees for the AREDS and AREDS2 trials.
D.C.M.: Honoraria – Data and Safety Monitoring Committee participation for the AREDS2 sirolimus clinical trial from EMMES Corp.; Data Monitoring Committee participation for the clinical trial of emixustat HCL treatment for geographic atrophy associated with AMD from Acucela Inc.; Data and Safety Monitoring Board participation for the clinical trial of radiation treatment for neovascular AMD from Oraya Therapeutics Inc.
Identification
Copyright
© 2015 American Academy of Ophthalmology. Published by Elsevier Inc. All rights reserved.
ScienceDirect
Access this article on ScienceDirectLinked Article
- Genetic Testing in Persons with Age-Related Macular Degeneration and the Use of the AREDS Supplements: To Test or Not to Test?OphthalmologyVol. 122Issue 1
- PreviewThe controversy surrounding the use of genetic testing to guide the treatment of persons with age-related macular degeneration (AMD) continues. In 2001, the results of the Age-Related Eye Disease Study (AREDS), a placebo-controlled trial, demonstrated that oral supplementation with a combination of antioxidant vitamins and zinc reduced the risk of progression to late AMD by 25% in persons with intermediate AMD in ≥1 eye.1 Klein et al2 evaluated the influence of the genotypes complement factor H (CFH) (Y402H, rs1061170) and LOC387715/age-related maculopathy susceptibility 2 (ARMS2) (A69S, rs10490924) on the response to treatment with AREDS supplements (combination of antioxidants and zinc), zinc alone, or antioxidants alone in 876 AREDS participants who had available DNA and who were at high risk of developing advanced AMD.
- Full-Text
- Preview
- Treatment Response to Antioxidants and Zinc Based on CFH and ARMS2 Genetic Risk Allele Number in the Age-Related Eye Disease StudyOphthalmologyVol. 122Issue 1
