Evaluating the performance of the breast cancer genetic risk models BOADICEA, IBIS, BRCAPRO and Claus for predicting BRCA1/2 mutation carrier probabilities: a study based on 7352 families from the German Hereditary Breast and Ovarian Cancer Consortium
- Christine Fischer1,
- Karoline Kuchenbäcker2,
- Christoph Engel3,
- Silke Zachariae3,
- Kerstin Rhiem4,
- Alfons Meindl5,
- Nils Rahner6,
- Nicola Dikow1,
- Hansjörg Plendl7,
- Irmgard Debatin8,
- Tiemo Grimm9,
- Dorothea Gadzicki10,
- Ricarda Flöttmann11,
- Judit Horvath12,
- Evelin Schröck13,
- Friedrich Stock14,
- Dieter Schäfer15,
- Ira Schwaab16,
- Christiana Kartsonaki2,17,
- Nasim Mavaddat2,
- Brigitte Schlegelberger10,
- Antonis C Antoniou2,
- Rita Schmutzler4,
- on behalf of the German Consortium for Hereditary Breast and Ovarian Cancer
- 1Institute of Human Genetics, University of Heidelberg, Heidelberg, Germany
- 2Department of Public Health and Primary Care, Centre for Cancer Genetic Epidemiology, University of Cambridge, Cambridge, UK
- 3Institute for Medical Informatics, Statistics and Epidemiology, University of Leipzig, Leipzig, Germany
- 4Department of Gynaecology and Obstetrics, Center for Familiar Breast and Ovarian Cancer, University Hospital of Cologne, Köln, Germany
- 5Department of Obstetrics and Gynecology, Division of Tumor Genetics, Klinikum rechts der Isar der Technischen Universität Muenchen, Munich, Germany
- 6Institute of Human Genetics, Heinrich-Heine-University Düsseldorf, Düsseldorf, Germany
- 7Institute of Human Genetics, University Hospital of Schleswig-Holstein, Campus Kiel, Christian-Albrechts-University of Kiel, Kiel, Germany
- 8Institute of Human Genetics, University of Ulm, Ulm, Germany
- 9Institute of Human Genetics, Biozentrum University of Würzburg, Würzburg, Germany
- 10Institute of Cell and Molecular Pathology, Medical School Hannover, Hannover, Germany
- 11Institute of Medical Genetics and Human Genetics, Charite’-Universitätsmedizin Berlin, Berlin, Germany
- 12Institute of Human Genetics, University of Münster, Münster, Germany
- 13Institute of Clinical Genetics, Medical School, Technical University of Dresden, Dresden, Germany
- 14Institute of Human Genetics, University of Leipzig, Leipzig, Germany
- 15Institute of Human Genetics, University of Frankfurt/Main, Frankfurt/Main, Germany
- 16Praxis für Humangenetik, Wiesbaden, Germany
- 17Department of Statistics, University of Oxford, Oxford, UK
- Correspondence to Dr Christine Fischer, Institute of Human Genetics, University of Heidelberg, Im Neuenheimer Feld 366, Heidelberg 69120, Germany;
- Received 14 October 2012
- Revised 6 February 2013
- Accepted 21 February 2013
- Published Online First 6 April 2013
Background Risk prediction models are widely used in clinical genetic counselling. Despite their frequent use, the genetic risk models BOADICEA, BRCAPRO, IBIS and extended Claus model (eCLAUS), used to estimate BRCA1/2 mutation carrier probabilities, have never been comparatively evaluated in a large sample from central Europe. Additionally, a novel version of BOADICEA that incorporates tumour pathology information has not yet been validated.
Patients and methods Using data from 7352 German families we estimated BRCA1/2 carrier probabilities under each model and compared their discrimination and calibration. The incremental value of using pathology information in BOADICEA was assessed in a subsample of 4928 pedigrees with available data on breast tumour molecular markers oestrogen receptor, progesterone receptor and human epidermal growth factor 2.
Results BRCAPRO (area under receiver operating characteristic curve (AUC)=0.80 (95% CI 0.78 to 0.81)) and BOADICEA (AUC=0.79 (0.78–0.80)), had significantly higher diagnostic accuracy than IBIS and eCLAUS (p<0.001). The AUC increased when pathology information was used in BOADICEA: AUC=0.81 (95% CI 0.80 to 0.83, p<0.001). At carrier thresholds of 10% and 15%, the net reclassification index was +3.9% and +5.4%, respectively, when pathology was included in the model. Overall, calibration was best for BOADICEA and worst for eCLAUS. With eCLAUS, twice as many mutation carriers were predicted than observed.
Conclusions Our results support the use of BRCAPRO and BOADICEA for decision making regarding genetic testing for BRCA1/2 mutations. However, model calibration has to be improved for this population. eCLAUS should not be used for estimating mutation carrier probabilities in clinical settings. Whenever possible, breast tumour molecular marker information should be taken into account.
Carriers of mutations in BRCA1 or BRCA2 are at increased risk of developing cancer and may require specific clinical management, such as intensified surveillance or preventive surgery. Genetic screening is available for mutations in these genes. However, it cannot be offered routinely to patients presenting with breast or ovarian cancer because of the associated costs and possible adverse psychosocial effects for patients.1 Genetic counselling and molecular screening need to be targeted at families in which a BRCA1 or BRCA2 mutation is likely to be segregating. In order to identify such families, rule-based methods or more sophisticated risk prediction models are used. It is important that the models are evaluated within the populations in which they are applied.
A number of mutation carrier prediction algorithms are widely used in clinical practice. These include BOADICEA,2 BRCAPRO,3 IBIS4 and the extended Claus (eCLAUS) models.5 ,6 These models assume an underlying genetic model for breast cancer susceptibility and can thus fully use individual pedigree patterns.
Only a few studies have investigated the performance of genetic risk models in large samples of families.7–15 Parmigiani et al12 compared BRCAPRO and a modified Claus model based on 3342 families from the USA. Kurian et al9 compared BRCAPRO and BOADICEA using 1437 families from three subpopulations in California, and Antoniou et al7 analysed BOADICEA, BRCAPRO and IBIS based on 1934 families seen in UK genetics clinics. Both BOADICEA and BRCAPRO performed well in these evaluations,7 ,9 but in the UK sample, only BOADICEA was well calibrated. Although these risk models are frequently used in different countries,16 a comparative validation in a large sample from continental Europe has yet to be carried out.
Recently, BOADICEA was extended to incorporate information of breast tumour molecular markers on affected probands and family members.17 Information on oestrogen receptor (OR), progesterone receptor (PR), human epidermal growth factor 2 (HER2), and expression of basal cytokeratins CK5/6 and CK14 can be taken into account.
However, the extended version of BOADICEA (BOADICEA-Path) has not been validated. The aims of the present study were twofold: (1) to compare the diagnostic accuracy and calibration of the available genetic risk prediction models BRCAPRO, BOADICEA, IBIS and eCLAUS using 7352 families from the German Consortium for Hereditary Breast and Ovarian Cancer (GC-HBOC); (2) to compare the incremental predictive value of incorporating tumour molecular marker information into the BOADICEA model.
Materials and methods
Families were recruited consecutively between January 1997 and November 2011 through 12 centres participating in the GC-HBOC. Selection was based on aggregation of early onset of breast or ovarian cancer cases. A family was eligible if at least one of the following criteria was fulfilled: (1) ≥3 female family members diagnosed with breast cancer; (2) ≥2 women diagnosed with breast cancer, one of whom is diagnosed at ≤50 years of age; (3) families with ≥1 female diagnosed breast cancer and one diagnosed with ovarian cancer (or in the same woman); (4) ≥1 woman with breast cancer diagnosed before 36 years; (5) ≥1 female family member diagnosed with bilateral breast cancer (first cancer diagnosed before 51 years); (6) ≥1 female family member diagnosed with ovarian cancer before 41 years; (7) ≥2 female family members diagnosed with ovarian cancer; (8) ≥1 male and ≥1 female diagnosed with breast cancer. Families were offered genetic counselling, BRCA1/2 mutation screening, and depending on their screening results a specific breast cancer surveillance programme.
The family member with the most severe diagnosis (defined as bilateral breast cancer, breast and ovarian cancer, or early age of onset) was chosen as the index patient and was screened for BRCA1 and BRCA2 mutations. Mutations were classified as pathogenic, according to internationally recognised criteria.18 Comprehensive data on family history of cancer including a detailed pedigree, pathology reports and results of genetic testing were documented in a central database. Data quality was routinely checked for completeness and consistency. All patients gave their written informed consent, and the study was approved by local ethics committees. Data quality control and detailed imputation rules for missing data are described in the online supplement.
All participating centres performed mutation screening in a standardised way. High performance liquid chromatography (dHPLC) of PCR products encompassing all coding exons of the BRCA1 and BRCA2 genes and subsequent sequencing of conspicuous amplicons or direct sequencing of all BRCA amplicons was performed.19 Sequences of both genes were evaluated based on the NCBI (National Center for Biotechnology Information) cDNA reference sequences U14680.1 (BRCA1 gene) and U43746.1 (BRCA2 gene). In case of negative sequencing results, analysis for deletions or duplications of the BRCA1 gene was carried out by multiplex ligation-dependent probe amplification. During the first years of the study, for a minority of the probands, a different or more restricted mutation screening took place. However, the average mutation screening sensitivity was high due to the introduction of the standardised screening protocol. The mutation screening sensitivity was assumed to be 100% for all those screened under the standard protocol. For BRCA1, 90% of the probands were tested for a mutation based on a technique with sensitivity of 1.0 and techniques with sensitivity <0.90 were used for 1.4% of the index patients. For BRCA2, in 91% of the cases, the mutation testing sensitivity was 1.0, and 99% were tested with a sensitivity ≥0.90. The probability that a mutation remained undetected was taken into account in our analyses by using appropriately reduced screening sensitivity depending on the methods used.7 Variants of unknown significance (VUS) were not considered as mutations. In regular intervals, VUS classification was revised according to the current literature.
BRCAPRO, BOADICEA and IBIS can be used to compute BRCA1 and BRCA2 mutation carrier probabilities and age-specific risks of developing breast and ovarian cancer. BOADICEA assumes that genetic susceptibility to breast cancer is due to BRCA1 and BRCA2 mutations and a separate polygenic component.2 ,20 ,22 The extended version, BOADICEA-Path version, can include information on several tumour molecular markers.17 BRCAPRO V.2.03 models genetic susceptibility in terms of BRCA1 and BRCA2 only. IBIS V.6a assumes effects of BRCA1 and BRCA2, and models the residual genetic variability in terms of a third dominantly inherited common gene conferring moderate risks.4 IBIS additionally includes environmental factors such as parity, which here was obtained from the reported family history. IBIS can only handle female index patients. Claus models breast cancer in terms of a single, rare, highly penetrant gene. We used an extended version of the Claus model (eCLAUS) as implemented in Cyrillic 2.12 that includes ovarian cancer occurrence in families.6 For our purposes, we programmed an identical batch version using the MLINK software.23 BOADICEA and BRCAPRO and eCLAUS allow for arbitrary pedigrees without loops, but IBIS only incorporates information from female first-degree and second-degree relatives, affected cousins and half-sisters.
We used a recently updated version of BOADICEA-Path, in which the age-specific distributions of ER, PR and HER2 tumours were derived from a large dataset of BRCA1 and BRCA2 mutation carriers from the Consortium of Investigators of Modifiers of BRCA1/2,24–26 and data on pathology of tumours arising in the general population from 42 768 breast cancer cases from the Breast Cancer Association Consortium (Antoniou AC and Easton DF, personal communication, 2012). The BOADICEA-path can incorporate information on all the following markers: ER, PR, HER2, CK5/6 and CK14. However, as a minimum, for tumour pathology information to be used in the model, information on OR-status should be available. Therefore, the evaluation of BOADICEA-path was carried out with a subset of 4927 pedigrees from the current sample in which at least one family member had data on tumour markers that could be used in carrier probability estimation using BOADICEA-Path. Available data on molecular markers comprised ER, PR and HER2 status of breast tumours. Our dataset included 6531 breast cancers with at least ER status information in 4927 families, and 3680 tumours with ER, PR and Her2 status in 3158 families.
Missing data on year of birth and/or ages at cancer diagnosis were imputed as described in the supplementary material.
Model calibration was evaluated by comparing the ratio of observed to expected numbers of mutation carriers at different categories of predicted mutation carrier probability. The corresponding 95% CIs were calculated under a Poisson approximation.27 Model discrimination was assessed via receiver operating characteristic-curve (ROC) analysis.28 We tested whether the corresponding areas under the ROC curves were different from each other.29 We compared the sensitivity, specificity, positive and negative predictive values of the models at carrier-probability thresholds commonly used for selecting families for clinical genetic testing. BOADICEA and BOADICEA-Path were compared using the same methods. Additionally, reclassification analysis was performed to obtain the number of correctly and incorrectly reclassified individuals and the Net Reclassification Index (NRI).30
A total of 7352 families were eligible for analysis. Table 1 summarises the characteristics of index patients and families. In 1774 families (24.1%), the proband had a mutation in BRCA1 or BRCA2. In 588 families (8.0%), the proband had no breast or ovarian cancer, and in 88 families (1.2%), the proband was male. A table showing sample characteristics for families with and families without pathology information is given in the online supplement (see online supplementary table S1).
The results are shown in table 2. The calibration of BOADICEA with regard to the total number of mutations was better than any of the other models. Overall, 1774 mutations were observed, and BOADICEA predicted 1744.2 mutations. However, the number of BRCA1 mutations was underpredicted (1243 observed and 977.9 expected) and the number of BRCA2 mutations was overpredicted (531 observed and 766.2 expected). For BRCAPRO, the total number of predicted mutations was higher than observed mainly due to an overprediction of BRCA2 carriers. IBIS predicted 1660.3 mutations (1744 observed). The number of BRCA1 mutations was underpredicted (observed 1228 and expected 976.9), and the number of BRCA2 mutations was overpredicted (observed 516 and 684.1 expected). eCLAUS overpredicted the total number of mutations substantially (1753 observed and 3938.8 expected). For the eight categories shown in table 2, all models except eCLAUS tended to underestimate the number of mutations in low carrier-probability classes, and to overestimate them in high carrier-probability classes.
The ROC curves showed similar patterns for BOADICEA and BRCAPRO, and were simultaneously larger than those of IBIS and eCLAUS (figure 1). The area under receiver operating characteristic curve (AUC) was 0.796 for BRCAPRO and 0.791 for BOADICEA. Both AUCs were significantly larger (p<0.001) than the AUCs of IBIS (0.749) and eCLAUS (0.745) (table 3). When predicting specifically BRCA1 or BRCA2 mutations, all models showed better discrimination for BRCA1 than BRCA2. BOADICEA and BRCAPRO had AUCs of 0.83, for IBIS AUC=0.79 (table 3). The ability of the models to discriminate between BRCA2 mutation carriers and those without BRCA2 mutation was weaker, with AUC values ranging from 0.62 to 0.68 across the different models. While BOADICEA, IBIS and BRCAPRO had similar specificity (at a 10% cut-off: 55.5% for BRCAPRO, 56.5% for IBIS, 56.8 for BOADICEA), BOADICEA and BRCAPRO had higher sensitivity (82.1% and 84.3%, respectively) than IBIS (77.0%). In line with the overprediction, eCLAUS showed high sensitivity (98.0%) but very low specificity (9.6%). At a 20% threshold, the sensitivities were 74.3% for BRCAPRO and 68.7% for BOADICEA. Positive predictive values were similar for BOADICEA, IBIS and BRCAPRO, and lowest for eCLAUS at both thresholds.
Information on tumour molecular markers
Data from 4927 families were used in this analysis (see online supplementary table S1). Incorporating information on OR, PR or HER2 into BOADICEA resulted in significant improvement in both model calibration and discrimination (table 4). BOADICEA-Path led to a significant increase in AUC over BOADICEA (81.2% vs 78.7%, p<0.001). There was also improved discrimination for BRCA1 carriers (AUC=88.3% vs 83.1%, p<0.001) and BRCA2 carriers (AUC=68.2% vs 66.4%, p<0.001) individually. At the 10% threshold, BOADICEA-Path provided a higher specificity (62.2% vs 57.3%) but similar sensitivity to BOADICEA.
Predicted carrier probabilities tended to increase for carriers, and decrease for non-carriers, when information on tumour molecular markers was incorporated into BOADICEA (see online supplementary figure S1). Reclassification analysis showed that BOADICEA-Path improved classification of non-carriers at all thresholds (table 5). In addition, more carriers were classified correctly than were misclassified for carrier prediction thresholds of 10% and all higher thresholds. For instance, at the 10% threshold, the net reclassification of carriers was 14 under BOADICEA-Path, and the net reclassification of non-carriers was 101, implying a Net Reclassification Index (NRI) of 3.9%. The NRI was highest (5.4%) at the 15% carrier prediction threshold.
We carried out the largest study to date, evaluating the performance of four widely used genetic models for estimation of BRCA1 and BRCA2 mutation carrier probabilities. Our study sample was based on 7352 ‘high-risk’ families from genetic counselling centres, as defined by the clinical inclusion criteria within GC-HBOC. The present study is unique in that these criteria were applied consistently across all centres, and mutation screening was standardised for all individuals screened at these centres. Furthermore, contrasting with previous model evaluation studies, 99% of index patients in our sample were screened for BRCA1 and BRCA2 mutations using highly sensitive techniques.7 ,15
In our sample, BOADICEA and IBIS provided the best fit for the total number of BRCA1 and BRCA2 mutations. However, none of the models were simultaneously well calibrated when predicting BRCA1 or BRCA2 mutation status for several carrier probability categories. The number of mutation carriers predicted by eCLAUS was much higher than the number observed. With eCLAUS, twice as many individuals were assigned to carrier probabilities >20% compared with the other models. BOADICEA, BRCAPRO and IBIS predicted too many mutations in the two lowest risk classes. If BOADICEA was used with a 10% threshold as the eligibility criterion for genetic testing, 3866 persons (52% of all probands) in this sample would have been screened to detect 1452 BRCA1/2 mutations (82%).
Underprediction in lower risk groups was also reported for UK families by Antoniou et al,7 but otherwise these investigators found that BOADICEA had good fit overall, even for predicting the number of BRCA1 and BRCA2 mutation carriers specifically. Several reasons may explain the differences between this study and ours. BOADICEA was primarily developed using data from the UK, BRCAPRO using data from the USA. To our knowledge, there are currently no estimates of BRCA1 and BRCA2 mutation frequencies based on large German series, or penetrance estimates, specifically for the German mutation spectrum. Individual BRCA1 and BRCA2 mutation frequencies may differ from mutation frequencies in the populations used to develop the models. Population-specific and sample-specific differences in model performance have been demonstrated previously for different samples worldwide.9 ,12 ,31 Our results strengthen the requirement of customising risk prediction models to specific populations. For the genetic risk models we considered here, this could be achieved through the use of population-specific disease allele frequency and penetrance estimates.
We found that BRCAPRO and BOADICEA discriminate well between mutation carriers and non-carriers, while IBIS and eCLAUS had significantly smaller AUCs. Our results for eCLAUS are in line with two other studies which included a modified CLAUS model (the Yale model) in their comparisons. In a study of 568 Italian families,10 the Yale model had the smallest AUC among all models considered, as well as the lowest sensitivity and specificity. Similar to our analysis, the Yale model overestimated the number of carriers significantly. In a separate study of 3342 US families12 consisting of population-based samples, as well as samples recruited through genetic clinics, all the models had similar abilities to discriminate between carriers and non-carriers, and all performed better in the population-based samples.12 In their sample from a high-risk population, the predictive performance of BRCAPRO was consistent with our results. Another study that evaluated BOADICEA and BRCAPRO in the USA found somewhat higher AUCs in a sample of non-Hispanic whites compared with our study.9
We found that the AUCs for predicting BRCA1 carrier status (AUCs for all models 0.786–0.830, 10% cut-off) were higher than the AUCs for predicting BRCA2 (AUCs for all models 0.623–0.683, 10% cut-off). This is likely to be due to the fact that a number of factors considered by the current models are better predictors of BRCA1 mutations. For example, BRCA1 mutations are associated with an earlier age at diagnosis of breast cancer and much higher ovarian cancer risks compared with BRCA2 mutations.32 In addition, tumour molecular markers, especially ER and PR status, are more predictive of BRCA1 mutations. On the other hand, BRCA2 tumours have been shown to have more similar characteristics to those from the general population.26 ,33 Therefore, the current models, both with and without molecular tumour marker information, cannot differentiate as well between BRCA2 mutation carriers and non-BRCA2 carriers. However, in clinical practice, discrimination between carriers of either mutation and non-carriers is more important since the overall BRCA1 and BRCA2 carrier probability will generally influence the decision to screen for mutations.
Tumour molecular markers
This is the first study that has validated the extended version of BOADICEA which incorporates tumour pathology information. Our results indicate that the inclusion of molecular tumour markers ER, PR and HER2 improves the performance of BOADICEA across all measures. There was a significant increase in AUC, and improvements were also seen for BRCA1 and BRCA2 mutations specifically. Overall model calibration was improved, although underprediction in low-risk groups remained. The specificity of BOADICEA increased while the sensitivity remained almost identical. Reclassification analysis showed overall net improvement at all probability thresholds. BOADICEA-Path can also take into account information on the tumour markers CK5/6 and CK14. The distributions of CK5/6 and CK14 in BRCA1-, BRCA2- and non-carrier tumours in BOADICEA-Path were based on data from the Breast Cancer Linkage Consortium Analyses.17 ,33 Those analyses had demonstrated that expression of basal cytokeratin markers in triple negative tumours is predictive of BRCA1 mutation status, but not BRCA2 mutation status. We would therefore expect further improvement in the BRCA1 predicted carrier probabilities if information on CK5/6 and CK14 is available. However, these markers were not available in the current dataset. Our results are in line with smaller studies that showed improvement for BRCAPRO model prediction when incorporating ER, PR and HER2 marker information.34 ,35 Our study adds to the evidence that information on tumour characteristics from both the index patient and other family members should be used for carrier probability assessment in clinical practice. Additional improvements in the models’ predictive ability would be expected as information on other tumour characteristics becomes available and are incorporated into risk prediction models. For instance, tumours arising in BRCA1 and BRCA2 mutation carriers have been shown to differ from each other and from sporadic cancers in their histopathological appearance, cytological and architectural features and tumour morphology characteristics.36 ,37 Future model extensions should aim to incorporate such predictors.
It is noteworthy that BOADICEA is the model that most closely mimics genetic susceptibility to breast cancer, as it specifically includes a polygenic component. The polygenic model for breast cancer is supported by the discovery of a growing list of common breast cancer susceptibility loci.38 Some of these variants are more strongly associated with particular subtypes of breast cancer. The explicit inclusion of these genetic variations in the model should further improve the performance of BOADICEA.
Using the largest sample of families to date, we found that BRCAPRO and BOADICEA had the highest discriminatory ability, and BOADICEA had the best calibration. The models provide useful tools for deciding whether to offer genetic testing for BRCA1 and BRCA2 mutations. The models could potentially be improved by adapting model parameters to individual populations. Our study demonstrates that eCLAUS should not be used for estimating carrier probabilities. Information on tumour molecular markers improves the predictive value of the model and should, therefore, be taken into account for selecting families for BRCA1/2 mutation screening.
CF, KK, CE, SZ, ACA and RS contributed equally
Contributors All authors of this study made important contributions to one ore more of the activities: conception and design, statistical analysis and interpretation, data collection and assembly. CF, KK, CE, SZ and ACA drafted the manuscript. All authors contributed to the review and approval of the final manuscript.
Funding This study was funded by Deutsche Krebshilfe, grant number 109076 and 109030, ACA is a Cancer Research—UK Senior Cancer Research Fellow. Statistical analysis was supported by CR-UK grant: C12292/A11174.
Competing interests None.
Patient consent Obtained.
Ethics approval Local ethics committees.
Provenance and peer review Not commissioned; externally peer reviewed.