Skip Navigation


Carcinogenesis Advance Access originally published online on May 14, 2008
Carcinogenesis 2008 29(6):1215-1218; doi:10.1093/carcin/bgn120
This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow All Versions of this Article:
29/6/1215    most recent
bgn120v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by Milne, R. L.
Right arrow Articles by Benítez, J.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Milne, R. L.
Right arrow Articles by Benítez, J.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?

© The Author 2008. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oxfordjournals.org

The importance of replication in gene–gene interaction studies: multifactor dimensionality reduction applied to a two-stage breast cancer case–control study

Roger L. Milne1,*, Rainer Fagerholm2, Heli Nevanlinna2 and Javier Benítez1,3

1 National Genotyping Centre, Spanish National Cancer Research Centre, Madrid 28029, Spain
2 Department of Obstetrics and Gynecology, Helsinki University Central Hospital, Helsinki 00290, Finland
3 Human Cancer Genetics Program, Spanish National Cancer Research Centre, Madrid 28029, Spain

* To whom correspondence should be addressed. Tel: +34 91 224 6900; Fax: +34 91 224 6980; Email: rmilne{at}cnio.es


   Abstract
Top
 Abstract
Introduction
 Materials and methods
 Results
 Discussion
 Funding
 References
 
A polygenic model has been proposed to explain the bulk of the genetic component of breast cancer aetiology and this is probably to include both main effects and interactions between multiple loci. However, the power to detect the interactions using traditional analytical methods is very limited for most studies. Multifactor dimensionality reduction (MDR) has been suggested to have increased power to detect interactions and is increasing being used in published studies. We applied MDR to a two-stage case–control breast cancer study conducted in Spain and Finland. In the stage 1 Spanish study of 864 cases and 845 controls, we evaluated interaction between 474 single-nucleotide polymorphisms in 120 cancer-related genes, subdivided into 34 genetic pathways and found evidence of a four-way interaction between genes in the FatiGO-defined B-cell receptor-signalling pathway (P < 0.006). However, this result was not replicated in the stage 2 Finnish study of 580 cases and 920 controls (P = 0.99). A number of technical issues in applying MDR to case–control data were identified and discussed. One of these is that the estimated sign test P-value can vary substantially at random, which raises doubts about its reliability. More generally, the present study serves as an important caution in the interpretation of results from single studies of gene–gene or gene–environment interaction in complex diseases. Just as for genetic main effects, the replication of positive findings in additional independent series is essential.

Abbreviations: LD, linkage disequilibrium; MDR, multifactor dimensionality reduction; SNP, single-nucleotide polymorphism


   Introduction
Top
 Abstract
 Introduction
Materials and methods
 Results
 Discussion
 Funding
 References
 
Breast cancer is a complex disease, with multiple genetic and environmental factors involved in its aetiology. A polygenic model has been proposed as the most likely to explain the bulk of the genetic component. Under this model, multiple loci across the genome each contribute to disease susceptibility. These main effects have been the focus of most genetic association studies conducted to date, with a number of recent studies reporting unequivocally positive findings for single-nucleotide polymorphisms (SNPs) (1–4). These SNPs have been identified predominantly via multistage designs beginning with genome-wide association studies that are then replicated in very large case–control series.

It is also probably that multiple loci interact to increase breast cancer risk and this may include epistatic interactions, where their combined effect is greater (or less) than that expected by multiplying their individual main effects. However, statistical power to detect such genetic interactions is a serious limitation for most studies, particularly for hypothesis-free indirect association studies where the number of possible interactions is extremely large. Different statistical methods, including logistic regression, classification and regression trees and multifactor dimensionality reduction (MDR), have been applied to large-scale SNP association studies to detect gene–gene interaction in a number of complex diseases, with some promising results (5–8). However, few, if any, of these interactions have been replicated in independent studies.

MDR is a non-parametric data mining approach that converts multiple variables to a single attribute, thereby changing the representation space of the data, in order to facilitate the identification of non-additive interactions. It has been suggested to have increased power, relative to traditional statistical methods such as logistic regression, to detect both gene–gene and gene–environment interaction (9,10), and a number of publications have emerged claiming to have demonstrated this in studies of relatively modest sample size (9) (see also http://compgen.blogspot.com/2006/05/mdr-applications.html). We applied this method to data on 474 SNPs in 120 cancer-related genes in a case–control series from Spain and then sought to validate positive findings in second stage Finnish replication series, with the aim of detecting novel gene–gene interactions in breast cancer susceptibility.


   Materials and methods
Top
 Abstract
 Introduction
 Materials and methods
Results
 Discussion
 Funding
 References
 
We carried out a two-stage study of gene–gene interactions in breast cancer in a Spanish and a Finnish case–control series.

Study participants
The study participants for both the stage 1 Spanish study and the Finnish replication study have been described previously (11). Briefly, Spanish cases were 864 women with breast cancer recruited between 2000 and 2004. Of these, 574 were a consecutive series of patients attending three public hospitals and the remainder were familial cases (with at least one first-degree relative also affected with breast cancer) attending the Spanish National Cancer Research Centre Family Cancer Clinic for genetic testing. All familial cases were tested for mutations in BRCA1 and BRCA2 [see (11) for details] and none were found to be carriers. Controls were 845 Spanish women free of breast cancer, recruited from multiple health care centres between 2000 and 2005 and with an age distribution comparable with cases (mean age of 53 years versus 50 years for cases). Informed consent was obtained from all participants and the study was approved by the institutional review board of La Paz Hospital in Madrid.

Eligible Finnish cases were 884 consecutive, newly diagnosed female breast cancer patients recruited from 1997 to 1998 and in 2000 at the Helsinki University Central Hospital. Their mean age was 57 years and none were found to carry mutations in BRCA1 or BRCA2. These included 214 cases with a family history of breast cancer. Eligible controls were a random sample of 1104 female blood donors attending blood banks in Helsinki in 2003. Their mean age was 41 years and all were free of breast cancer. The study was carried out with the informed consent of all patients and approval from the Ethics Committees of the Departments of Oncology and Obstetrics and Gynecology as well as from the Ministry of Social Affairs and Health in Finland.

Gene and SNP selection
A total of 121 genes were selected, including the 112 described previously [involved in cancer, the cell cycle pathway, DNA repair, cell communication, hormone metabolism, apoptosis, carcinogen metabolism, cell adhesion and/or signal transmission, as described previously (11)] plus 9 drug metabolism genes (CYP11B1, CYP11B2, CYP17A1, CYP1A1, CYP2A7, CYP2D6, CYP2E1, NAT1 and NAT2). A full list of genes is available online at http://bioinfo.cnio.es/cgi-bin/cegen/frequencies.cgi. SNPs were selected in these genes according to previously described criteria (11), based on NCBI dbSNP build 120 http://www.ncbi.nlm.nih.gov/entrez/query.fcgi. A final total of 769 SNPs (the 710 previously reported plus 59 in the drug metabolism genes) were selected.

For the present study, we reduced the number of SNPs included in analyses by excluding SNPs that were tagged by others. The program Tagger (12) was used to identify genotyped SNPs in linkage disequilibrium (LD) and a cut-off of r2 > 0.80 applied.

Genotyping
DNA extraction and quantification methods at both study centres are described in Milne et al. (11). Spanish subjects were genotyped for all 769 SNPs using the Illumina platform [Illumina, San Diego, CA (13)], with at least one duplicate and one negative control per 96-well plate and six duplicates between plates. Finnish subjects were genotyped for four SNPs by the KBiosciences genotyping service using the custom-developed ‘Kaspar’ assay (http://www.kbioscience.co.uk), with 184 duplicated samples in total.

Statistical analyses
For the stage 1 Spanish study, genes were non-exclusively grouped into genetic pathways defined by the Kyoto Encyclopedia of Genes and Genomes using the program FatiGO+ (http://fatigo.bioinfo.cipf.es/) (14). A total of 31 pathways containing at least two candidate genes were identified (see Table I). Additional pathways not included in the Kyoto Encyclopedia of Genes and Genomes that were considered were the mismatch repair (MLH1, MSH2, MSH3 and MSH6), nucleotide excision repair (XPA, XPC, ERCC1, ERCC2, ERCC4 and ERCC6) and double-strand break repair pathways (BRCA1, BRCA2, XRCC1, XRCC1, XRCC2, XRCC3, XRCC4 and FANCA). Interaction between all non-tagged SNPs in the genes pertaining to each pathway was then assessed using the program MDR v1.1.0 (9).


View this table:
[in this window]
[in a new window]

  		Table I. Genetic pathways considered and genes therein for which SNPs were genotyped in the stage 1 Spanish study

 
The MDR method is described in detail elsewhere (10,15). Briefly, in an analysis of n-way interactions, the n-dimensional space formed by all possible combinations of values (classes) of a given set of n variables (in this case, SNPs) is reduced to a single dimension by reclassifying each class as either high risk or low risk according to the relative proportion of cases to controls in that class. This is done for all possible combinations of n variables among the total N variables considered, and the combination with the lowest misclassification error is reported. The MDR software gives a number of output parameters to assess each interaction reported. The cross-validation consistency score is a measure of the degree of consistency with which the reported interaction is identified as the most evident among all possibilities of that order considered. The sign test purportedly gives P-values for that interaction, adjusted for multiple testing for the full set of SNPs included in a single analysis. Finally, the testing balanced accuracy is a measure of the degree to which the interaction accurately predicts case–control status, taking into account the ratio of cases to controls, with scores between 0.50 (indicating that the model predicts no better than chance) and 1.00 (indicating perfect prediction). Scores of at least 0.55 are considered ‘interesting’ (http://compgen.blogspot.com/2006/12/mdr-101-part-4-results.html).

In order to remove the possibility of spurious associations due to missing genotypes, only cases and controls for which genotypes were called for all SNPs in genes in a given pathway were included. All possible two-, three- and four-way SNP interactions were tested using 10-fold cross-validation in an exhaustive search (considering all possible SNP combinations). The testing balanced accuracy for the best-fitting model was estimated in an unbiased way by rerunning a forced analysis for that particular combination of SNPs. Analyses were repeated for any pathway for which the best of any of these interactions had an associated sign test P-value < 0.05, each time changing the random seed from the default of zero to a different value and varying the number of cross-validations carried out.

For the stage 2 Finnish study, all possible SNP interactions were tested using the default MDR parameters in an exhaustive search. For both studies, departure from Hardy–Weinberg equilibrium was tested in controls for individual SNPs using the genhwi command in Stata version 8 (16).


   Results
Top
 Abstract
 Introduction
 Materials and methods
 Results
Discussion
 Funding
 References
 
All duplicates both within and between plates genotyped in the Spanish samples were concordant for all SNPs. Of the 769 SNP assays, 80 either failed genotyping (no polymerase chain reaction amplification, insufficient intensity for cluster separation or no or poor cluster definition) were monomorphic or violated Hardy–Weinberg equilibrium (n = 5, with P-value < 0.0001), leaving 689 SNPs successfully genotyped in 120 genes (all three SNPs selected in MAP2K3 failed genotyping). A full list of these SNPs, including estimated minor allele frequencies for controls, can be found at our Bioinformatics website (http://bioinfo.cnio.es/cgi-bin/cegen/frequencies.cgi). These 689 SNPs were tagged with r2 > 0.80 by a subset of 474 SNPs that were included in all subsequent analyses.

Of the 34 pathways evaluated using MDR, seven had combinations of SNPs with associated sign test P-values < 0.05. The results of analyses for these pathways are summarized in Table II. On rerunning analyses with different random seeds, it was noted that the estimated sign test P-value can be highly unstable from one run to the next. The most extreme example of this was observed for the analysis of the ‘Regulation of actin cytoskeleton’ pathway with values ranging from 0.0002 to 0.39 between reruns. We therefore repeated the analyses for these seven pathways (see Table II). For all but one of these, at least one in four repeats gave a sign test estimated P-value ≥ 0.19. The exception was the result for a four-way interaction between PIK3R1(rs40419), PIK3R2(rs2267922), AKT1(rs2498804) and NFKB1(rs93059) in the B-cell receptor-signalling pathway, for which all four additional P-values were ≤ 0.01. The cross-validation consistency was moderate to high, with values ranging from 0.67 to 0.95, and the testing balanced accuracy scores promising, ranging between 0.56 and 0.58. This was based on complete genotypes for all 27 SNPs in genes pertaining to this pathway, which were available for 698 (81%) cases and 760 (90%) controls. Results were consistent when only participants missing at least one genotype for the four SNPs (rs40419, rs2267922, rs2498804 and rs93059) were excluded, leaving 855 (99%) cases and 839 (99%) controls testing balanced accuracy = 0.54, (P-value = 0.01). This interaction was also consistently observed after stratifying the sample at age 50 (testing balanced accuracy = 0.58 and 0.54, respectively and P-value = 0.01 and 0.01, respectively).


View this table:
[in this window]
[in a new window]

  		Table II. Summary of results from repeated MDR analyses of SNPs in seven pathways with evidence of interaction in the Spanish case–control seriesa

 
Given this positive result for the four-way interaction between PIK3R1(rs40419), PIK3R2(rs2267922), AKT1(rs2498804) and NFKB1(rs93059), we carried out a validation study in the Finnish case–control series described previously (11). This was considered necessary because, whereas the sign test P-value adjusts for multiple comparisons for a given set of SNPs, it does not adjust for multiple comparisons across distinct pathways. Therefore, despite the consistency in estimated P-values and relatively high testing balanced accuracy and model consistency across repeated analyses of genes in the B-cell receptor-signalling pathway, we could not be sure that this was not a false positive in light of the 34 pathways tested.

Of the 1988 eligible Finnish participants, 185 cases and 111 controls were excluded due to DNA samples being depleted since the genotyping was carried out for the previous study. Genotypes between duplicates were concordant for all four SNPs, and no evidence of departure from Hardy–Weinberg equilibrium was observed. A further 119 cases and 73 controls were excluded from the analysis of interactions due to missing genotypes for at least one SNP, leaving 580 cases and 920 controls in the final analysis. The minor allele frequencies were similar to those observed in the Spanish sample for all four SNPs (0.41 versus 0.40 for rs40419, 0.48 versus 0.42 for rs2267922, 0.35 versus 0.33 for rs2498804 and 0.47 versus 0.48 for rs93059).

The Finnish validation study gave no evidence of interaction between rs40419, rs2267922, rs2498804 and rs93059, with a testing balanced accuracy of 0.48 and a sign test P-value of 0.99. This result remained unchanged when cases with a family history were excluded and when controls under age 40 and cases over age 70 were excluded. It was also consistent for women under age 50 and over age 50 (testing balanced accuracy = 0.46 and 0.47, respectively and P-value = 0.94 and 0.87, respectively).


   Discussion
Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
Funding
 References
 
MDR is a novel analytical tool for the detection of gene–gene and gene–environment interactions in complex diseases. However, few, if any, positive findings have been replicated in independent studies to confirm that they are true associations, rather than due to chance, as appears to have been the case for most single variant main effects identified for the first time, at least in breast cancer (17).

We have carried out two-stage study of gene–gene interaction in breast cancer using MDR. Stage 1 included an exhaustive search for multilocus epistatic interactions between SNPs in 121 cancer-related genes. Based on a relatively large sample of Spanish breast cancer cases and controls, we found evidence of a four-way interaction between SNPs in genes involved in the FatiGO-defined B-cell receptor-signalling pathway. However, that this was a false-positive result could not be ruled out after consideration of the multiple pathways tested. Consistent with what has now become standard practice in association studies evaluating main effects, we sought to replicate our finding in an independent case–control series and found no evidence of the putative interaction.

On possible explanation for this finding is that the Finnish study lacked statistical power to detect the interaction observed in the Spanish series. While it is difficult to quantify the power, or for that matter an effect size, the sample sizes of the two studies were quite similar and the P-value and testing balanced accuracy estimated from the latter suggest no effect at all. An alternative explanation for the failure to replicate the four-way interaction is that LD patterns may be different in the Finnish population and therefore, the markers selected may not adequately capture the variation in the truly causal variants. Again, this appears unlikely since LD patterns have been found to be generally similar across Europe (18,19) and any differences are probably to be reflected in reduced power rather than the absence of any evidence of an effect. Of course, this may be an oversimplification in the presence of a four-way interaction and so, this possible explanation cannot be entirely ruled out. Stratified and sensitivity analyses suggested that differences between the two studies in the age and menopausal status distributions of cases and controls are unlikely to explain the lack of replication. The other explanation for the lack of replication is that there is no interaction at all between these SNPs.

A number of limitations of the MDR method were identified in this study. The first was that the sign test P-value estimator is imprecise and can vary substantially at random. Requiring consistency in results across repeated analyses appears to address this problem, although this should be evaluated systematically via simulation of a known effect. The other is that missing genotypes were a major source of noise when included in the analysis. It has been recommended that missing genotypes be included, either as an additional category for each genotype or by imputing them. We believe that bias was minimized in our study by excluding subjects with missing genotypes. Analyses of the data including a missing category gave artefactual associations related to the greater tendency to be missing among cases due to reduced sample quantity, particularly among cases with a family history who had been tested for mutations in BRCA1 and BRCA2. That results were consistent in the Spanish series when 99% of cases and controls were included (based on analyses of the four SNPs in question, rather than all SNPs in the pathway) confirms that any bias was minimal. Finally, we cannot exclude the possibility that epistatic interactions, or even main effects, exist between/for SNPs in the genes studied, both due to limited power and because the SNPs genotyped do not capture all the common variation in these genes. The latter is the case because SNP selection was based on data from Phase I of HapMap (20), which included a lower density of polymorphisms.

An increasing number of gene–gene interactions are being reported in the literature based on analyses using MDR (see http://compgen.blogspot.com/2006/05/mdr-applications.html). This is encouraging since it originally appeared that unrealistic sample sizes would be required to have sufficient power to detect multilocus interaction using traditional methods such as logistic regression. However, the present study serves as an important caution in the interpretation of results. We recommend that the MDR sign test P-value estimator be more extensively evaluated. In the meantime, conclusions regarding statistical significance of putative interactions identified using MDR should not be based on a single analysis, but rather on consistent results from repeated analyses. In addition, careful consideration should be given to how missing data is treated in the analysis, particularly when they are not missing at random, such as when DNA quality differs between cases and controls. Finally, just as for main effects, replication in additional independent samples of observed interaction in a single study is essential. It is also more complicated. Differences in LD patterns between study populations may influence the detection of higher order interactions in unknown ways, and even when significant effects are observed in replication studies, it remains to be clarified exactly what constitutes a consistent finding, particularly for results from methods such as MDR where biological meaning is difficult to assign.


   Funding
Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 Funding
References
 
Genome Spain Foundation and Fundación Marató to the Spanish component. Helsinki University Central Hospital Research Fund, Academy of Finland (110663), Finnish Cancer Society and Sigrid Juselius Foundation to the Finnish study.


   Acknowledgments
 
We thank José Ignacio Arias (Hospital Monte Naranco), Pilar Zamora (La Paz), Álvaro Ruibal (Fundación Jiménez Díaz), Santiago Palacios (Instituto Palacios), Silvia de Sanjose (Institut Català d'Oncologia) and Rogelio González Sarmiento (Centro Investigación del Cancer) for the use of Spanish samples of cases and controls and Kirsimari Aaltonen, Hannaleena Eerola and Carl Blomqvist, as well as RN Nina Puolakka for their kind help with patient contacts, data management and sample collection in Finland.

Conflict of Interest Statement: None declared.


   References
Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 Funding
 References
 

  1. Cox A, et al. A common coding variant in CASP8 is associated with breast cancer risk. Nat. Genet. (2007) 39:352–358.[CrossRef][Web of Science][Medline]
  2. Easton DF, et al. Genome-wide association study identifies novel breast cancer susceptibility loci. Nature (2007) 447:1087–1093.[CrossRef][Medline]
  3. Hunter DJ, et al. A genome-wide association study identifies alleles in FGFR2 associated with risk of sporadic postmenopausal breast cancer. Nat. Genet. (2007) 39:870–874.[CrossRef][Web of Science][Medline]
  4. Stacey SN, et al. Common variants on chromosomes 2q35 and 16q12 confer susceptibility to estrogen receptor-positive breast cancer. Nat. Genet. (2007) 39:865–869.[CrossRef][Medline]
  5. Andrew AS, et al. Concordance of multiple analytical approaches demonstrates a complex relationship between DNA repair gene SNPs, smoking and bladder cancer susceptibility. Carcinogenesis (2006) 27:1030–1037.[Abstract/Free Full Text]
  6. Briollais L, et al. Methodological issues in detecting gene-gene interactions in breast cancer susceptibility: a population-based study in Ontario. BMC Med. (2007) 5:22.[CrossRef][Medline]
  7. Chen M, et al. High-order interactions among genetic polymorphisms in nucleotide excision repair pathway genes and smoking in modulating bladder cancer risk. Carcinogenesis (2007) 28:2160–2165.[Abstract/Free Full Text]
  8. Tsai CT, et al. Renin-angiotensin system gene polymorphisms and coronary artery disease in a large angiographic cohort: detection of high order gene-gene interaction. Atherosclerosis (2007) 195:172–180.[CrossRef][Web of Science][Medline]
  9. Moore JH, et al. A flexible computational framework for detecting, characterizing, and interpreting statistical patterns of epistasis in genetic studies of human disease susceptibility. J. Theor. Biol. (2006) 241:252–261.[CrossRef][Web of Science][Medline]
  10. Ritchie MD, et al. Power of multifactor dimensionality reduction for detecting gene-gene interactions in the presence of genotyping error, missing data, phenocopy, and genetic heterogeneity. Genet. Epidemiol. (2003) 24:150–157.[CrossRef][Web of Science][Medline]
  11. Milne RL, et al. ERCC4 associated with breast cancer risk: a two-stage case-control study using high-throughput genotyping. Cancer Res. (2006) 66:9420–9427.[Abstract/Free Full Text]
  12. Barrett JC, et al. Haploview: analysis and visualization of LD and haplotype maps. Bioinformatics (2005) 21:263–265.[Abstract/Free Full Text]
  13. Oliphant A, et al. BeadArray technology: enabling an accurate, cost-effective approach to high-throughput genotyping. Biotechniques (2002) (suppl):56–58. 60–61.
  14. Al-Shahrour F, et al. FatiGO: a web tool for finding significant associations of Gene Ontology terms with groups of genes. Bioinformatics (2004) 20:578–580.[Abstract/Free Full Text]
  15. Ritchie MD, et al. Multifactor-dimensionality reduction reveals high-order interactions among estrogen-metabolism genes in sporadic breast cancer. Am. J. Hum. Genet. (2001) 69:138–147.[CrossRef][Web of Science][Medline]
  16. StataCorp. Stat Statistical Software: Release 8.0 (2003) College Station, TX: Stata Corporation.
  17. Breast Cancer Association Consortium. Commonly studied single-nucleotide polymorphisms and breast cancer: results from the Breast Cancer Association Consortium. J. Natl Cancer Inst. (2006) 98:1382–1396.[Abstract/Free Full Text]
  18. Mueller JC, et al. Linkage disequilibrium patterns and tagSNP transferability among European populations. Am. J. Hum. Genet. (2005) 76:387–398.[CrossRef][Web of Science][Medline]
  19. Ribas G, et al. Evaluating HapMap SNP data transferability in a large-scale genotyping project involving 175 cancer-associated genes. Hum. Genet. (2006) 118:669–679.[CrossRef][Web of Science][Medline]
  20. The International HapMap Consortium. A haplotype map of the human genome. Nature (2005) 437:1299–1320.[CrossRef][Medline]
Received February 25, 2008; revised April 15, 2008; accepted May 8, 2008.


Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us    What's this?



This Article
Right arrow Abstract Freely available
Right arrow FREE Full Text (PDF) Freely available
Right arrow All Versions of this Article:
29/6/1215    most recent
bgn120v1
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to My Personal Archive
Right arrow Download to citation manager
Right arrowRequest Permissions
Google Scholar
Right arrow Articles by Milne, R. L.
Right arrow Articles by Benítez, J.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Milne, R. L.
Right arrow Articles by Benítez, J.
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us  
What's this?