Objective: To test whether statistical models developed to calculate pre-test probability of being a BRCA1/2 carrier can differentiate better between the breast/ovarian families to be referred to the DNA test laboratory.
Study design: A retrospective analysis was performed in 109 Spanish breast/ovarian families previously screened for germline mutations in both the BRCA1 and BRCA2 genes. Four easy to use logistic regression models originally developed in Spanish (HCSC model), Dutch (LUMC model), Finnish (HUCH model), and North American (U Penn model) families and one model based on empirical data of Frank 2002 were tested. A risk counsellor was asked to assign a subjective pre-test probability for each family. Sensitivity, specificity, negative and positive predictive values, and areas under receiver operator characteristics (ROC) curves were calculated in each case. Correlation between predicted probability and mutation prevalence was tested. All statistical tests were two sided.
Results: Overall, the models performed well, improving the performances of a genetic counsellor. The median ROC curve area was 0.80 (range 0.77-0.82). At 100% sensitivity, the median specificity was 30% (range 25-33%). At 92% sensitivity, the median specificity was 42% (range 33.3-54.2%) and the median negative predictive value was 93% (range 89.7-98%). BRCA1 families tended to score higher risk than BRCA2 families in all models tested.
Conclusions: All models increased the discrimination power of an experienced risk counsellor, suggesting that their use is valuable in the context of clinical counselling and genetic testing to optimise selection of patients for screening and allowing for more focused management. Models developed in different ethnic populations performed similarly well in a Spanish series of families, suggesting that models targeted to specific populations may not be necessary in all cases. Carrier probability as predicted by the models is consistent with actual prevalence, although in general models tend to underestimate it. Our study suggests that these models may perform differently in populations with a high prevalence of BRCA2 mutations.
- pre-test probability assessment
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
The identification of the breast cancer susceptibility genes BRCA1 and BRCA2 in the past decade1,2 has permitted identification of presymptomatic subjects at risk of developing breast/ovarian cancer by means of a genetic test. Nowadays, many families with a moderate history of breast and/or ovarian cancer are self or physician referred to familial cancer clinics where genetic testing of these susceptibility genes is available. Unfortunately, the analysis is costly and time consuming and can cause considerable stress to many families. Moreover, a negative result does not imply a clear benefit, either psychological or clinical, given that genetic susceptibility cannot be ruled out in these families and other breast cancer genes unidentified to date may be involved.3 Accordingly, it would be advantageous to target the available resources to test families with the highest probability of being mutation carriers. Thus, the development of an accurate pre-test determination of carrier probability has become in recent years a major topic in familial cancer clinics throughout the world.
In a 1996 policy statement, the American Society of Clinical Oncology (ASCO) suggested that gene mutation testing should be limited to subjects whose probability of carrying a mutation exceeds 10%.4 There are a number of statistical approaches to calculating the pre-test probability of carrying a mutation.5–11 However, subjective assessment by professional risk counsellors remains essential. Indeed, many familial cancer clinics do not refer families to the DNA laboratory in accordance with a calculated pre-test probability but establish “minimal entry criteria” (in terms of cancer phenotype) which all families selected for genetic testing must meet. Although no consensus exists, most familial cancer clinics will agree to select families with at least three cases of breast/ovarian cancer for genetic testing.5–7
The present study is focused on this type of family, which should be considered as high risk. However, only 30% of these families harbour a pathogenic mutation.5–7 Therefore, the majority of the families currently referred to the DNA laboratory in cancer clinics throughout the world do not obtain any benefit from genetic testing. To reduce this proportion, a better understanding of the cancer phenotype associated with germline mutations in these genes is necessary.
Recently, some easy to use logistic regression models to calculate the pre-test probability that a family with a given cancer phenotype carries a BRCA mutation have been developed.9,12–14 In most cases, these models have been devised with high risk families commonly attending familial cancer clinics. They take into account both BRCA1 and BRCA2 mutation status (with the exception of the model of Couch et al,9 which is restricted to BRCA1), and make no assumption regarding prevalence or penetrance of these alleles in the target population. The performance of these models in an independent cohort of high risk families and their use in familial cancer clinics to reduce the number of BRCA negative families referred to the DNA laboratory have not been properly evaluated. Moreover, these models have been developed in very specific populations and the predictive variables they use are similar but not identical, so it is not clear whether they can be implemented in populations other than the one for which they were devised.
The aim of our study was, therefore, to test the performance of easy to use prior probability models to decrease the number of true negative families that are currently referred to the DNA laboratory.
FAMILIES AND METHODS
We conducted this study in a clinic based cohort of 109 families. The Oncology and Genetics Departments of the Hospital de la Santa Creu i Sant Pau (Barcelona, Spain) and the Laboratory of Molecular Oncology, Department of Clinical Oncology, Hospital Clínico San Carlos (Madrid, Spain) submitted, respectively, 80 and 29 pedigrees and corresponding BRCA1 and BRCA2 results. These pedigrees had already been selected for complete BRCA gene sequencing on the basis of cancer family history information suggestive of an inherited breast and ovarian cancer predisposition (all pedigrees included at least three or more first or second degree relatives affected with breast or ovarian cancer in the same lineage). Pedigrees were constructed on the basis of an index case considered to have the highest probability of being a deleterious mutation carrier (generally the youngest affected subject available in each family). To construct pedigrees, patients were interviewed about their family history of cancer for information on cancer profiles and dates of diagnoses of all subjects, including first and second degree relatives of the index case. Characteristics of the study sample are summarised in table 1. Mutation analysis was performed in all index cases by either a combination of SSCP and PTT (Hospital de la Santa Creu i Sant Pau) or DGGE (Hospital Clínico San Carlos). In both cases, mutation screening protocols included all coding sequences and intron/exon boundaries.12,15,16 The probability of carrying a BRCA mutation was calculated in each pedigree according to the four logistic regression models tested in this study. The model developed at the Hospital Clínico San Carlos (HCSC model)12 considers as predictor variables the number of ovarian cancer cases in the family, mean age at diagnosis of breast cancer, and the presence/absence of concomitant breast and ovarian cancer in a single woman, bilateral breast cancer, and/or male breast cancer. The model of Peelen et al14 was developed at the Leiden University Medical Centre (LUMC model). In this case, the predictor variables are the number of ovarian cancer cases in the family, the number of breast cancer cases in the family, mean age at diagnosis of breast cancer, and the presence/absence of bilateral breast cancer. The model of Vahteristo et al13 was developed at the Helsinki University Central Hospital (HUCH model). The number of ovarian cancer cases in the family and the age of the youngest breast cancer patient in each family are the only predictor variables considered in this case. The model of Couch et al9 was developed at the University of Pennsylvania (U Penn model). Predictor factors included average age at breast cancer diagnosis in the family under than 55 years, ovarian cancer in the family (particularly in a subject with breast cancer), and Ashkenazi Jewish ancestry. Data concerning the predictor variables of each model were available in all 109 pedigrees included in this analysis. We also calculated the probability of founding mutations according to the model of Frank 2002.10 This is an empirical model which correlates prevalence of mutations in BRCA1 and BRCA2 with personal and family history of cancer. It is based on data from 10 000 subjects tested through Myriad Genetics. Probabilities according to the HCSC, LUMC, and HUCH models were calculated in a convenient Microsoft Excel format. In the case of the U Penn model, the probability for each possible permutation of the predictor variables has been previously calculated and tabulated9 and we used these tabulated data (non-Ashkenazi heritage subset) to assign a probability to every family in our study sample. In the case of the model of Frank 2002,10 we assigned a probability to each family according to the correlation found in 4716 non-Ashkenazi subjects (table 2).
An experienced risk counsellor from a familial cancer clinic was asked to evaluate each pedigree and assign a subjective pre-test probability for each family. Fifty percent of the risk counsellor practice was devoted specifically to breast-ovarian cancer susceptibility counselling. For the last five years, he has been counselling 8-10 Spanish breast-ovarian families each month. The risk counsellor was provided with data corresponding to the predictor variables used in the models. His assessment was based solely on his previous experience with Spanish breast/ovarian families and was not assisted by any pre-test probability statistical model.
Statistical analyses were two sided. Categorical variables were compared by the chi-square test and numerical variables by the t test. Receiver operator characteristics (ROC) curve areas, sensitivity, specificity, negative predictive values (PV−), positive predictive values (PV+), and correlation coefficients were calculated with the MedCalc software package.
The prevalence of BRCA mutations in our study sample was 33.9% (95% CI 26 to 43). This prevalence is consistent with that previously reported in Spanish and other populations.5–6,9,12,15–17 Nineteen families carried a BRCA1 mutation and 18 families had a BRCA2 mutation. All mutations are predicted to produce a truncated protein and are considered pathogenic in the BIC database.18 The spectrum of mutations is available on request from the authors. Other characteristics of the study sample are listed in table 1.
Families with ovarian cancer (57% v 21%, p=0.0003), concomitant breast and ovarian cancer in a single woman (22% v 5%, p=0.02), bilateral breast cancer (32% v 18%, p=0.09), and male breast cancer (13% v 4%, p=0.1) were more frequent in the BRCA positive group. However, only in the case of ovarian cancer and concomitant breast and ovarian cancer in a single woman did these differences reach statistical significance. Male breast cancer was the only phenotype clearly associated with BRCA2 but not with BRCA1 families (27.7% v 0%, p=0.02). The mean age at breast cancer diagnosis among women from mutation carrier families was lower than that for women from non-carrier families (43.6 years v 49.6 years, p=0.005). The mean age at diagnosis of the youngest breast cancer patient (relevant for the HUCH model) was also lower in BRCA positive families (37.8 years v 39.6 years) although the difference was not statistically significant. Overall, our study sample appears to be representative of breast/ovarian families commonly seen in familial cancer clinics and, therefore, relevant to our analysis.
As might be expected, the average pre-test probability of carrying a mutation was higher in positive than in negative families. Differences were as follows: 0.413 v 0.174 in the HCSC model, 0.356 v 0.133 in the LUMC model, 0.272 v 0.102 in the U Penn model, 0.381 v 0.174 in the HUCH model, 0.42 v 0.21 according to Frank 2002, and 0.58 v 0.46 according to the risk counsellor (all differences statistically significant at the p<0.001 level).
To compare the performance of the four logistic regression models with that of the Frank 2002 empirical data and the risk counsellor assessment, we calculated the Receiver Operator Characteristic (ROC) curve area, sensitivity, specificity, PV−, PV+, and the best discriminating probability threshold in each case (fig 1, table 3). The area under the ROC curve (a measure of the overall discrimination between BRCA positive and negative families) was 0.82 in the HCSC model, 0.80 in the LUMC model, 0.77 in the U Penn model, 0.77 in the HUCH model, 0.82 with the Frank 2002 prevalence, and 0.69 for the risk counsellor. Among statistical models, the maximum difference between the ROC areas (HCSC v HUCH, fig 1B) did not attain statistical significance (0.049 95% CI 0.05 to 0.148), indicating that these models have a similar power of discrimination. However, the ROC area calculated with data from the risk counsellor assessment was clearly lower when compared with any model. The maximum difference was 0.127 (95% CI 0.023 to 0.230) (HCSC v risk counsellor, fig 1D). This difference was significant (p=0.016) indicating that discrimination can be improved by using statistical models.
Three relevant probability thresholds were selected for our analysis: the 100% sensitivity threshold, the 92% sensitivity threshold (which we consider acceptable in clinical practice), and the best discriminating probability threshold (best performance from a statistical point of view). It is interesting to note that the best discriminating probability threshold was not clinically relevant in any statistical model. This was because the sensitivities were well below 90% in all cases, ranging from 86.5% (LUMC model) to 59.5% (U Penn model). Similarly, the sensitivity reached by the risk counsellor (59.5%) did not have any clinical relevance. However, the specificity ranged from 25% (U Penn model) to 33% (Frank 2002) if families with pre-test probabilities above the 100% sensitivity threshold were selected for testing. To compare the performances of these models further, we chose an arbitrary but clinically acceptable 92% sensitivity threshold (table 3). By selecting families with probabilities above this threshold, specificity ranged from 33.3% (HUCH model) to 54.2% (LUMC model). The risk counsellor specificity was clearly lower (26.4%), although the difference only reached statistical significance when compared with the HCSC (p<0.05) and LUMC models (p<0.05). Overall, the data shown in table 3 indicate that the selection of a suitable pre-test probability threshold (which is different in each model) will better differentiate the families to be referred to the DNA laboratory. Ovarian cancer is an important variable in all predictor models. It might therefore be possible that the performance of these models varies with the proportion of breast/ovarian families present in the cohort. To test this hypothesis, we performed a subanalysis in the 72 breast only families (families with no single case of ovarian cancer reported) present in our cohort. This subset of families included four BRCA1 families, 12 BRCA2 families, and 56 negative families. As shown in table 3, with 100% sensitivity, neither ROC area nor specificity are severely impacted, suggesting that these models are not dependent on ovarian cancer to discriminate BRCA positive from BRCA negative families.
To analyse these models further, it is interesting to study the characteristics (if any) of true positive families which tend to be misclassified as negatives. In our study series (37 BRCA positive families) a significant concordance among models was observed. The HCSC, LUMC, and U Penn models misclassified the SP18 (BRCA2), SP122 (BRCA2), and SP46 (BRCA1) families as negatives by using the 92% sensitivity threshold. The same families are misclassified if Frank 2002 tables are used. These families shared a common phenotype: three unilateral breast cancer cases with a median age of 50 years at diagnosis (borderline minimal entry criteria). The HUCH model, which considers the youngest age but not the median age at diagnosis, ruled out the SP46 family but correctly selected the SP18 and SP122 families. Both families include a breast cancer case diagnosed at the age of 33. By contrast, the SP33 (with three unilateral breast cancer cases, one bilateral breast cancer case, and a median age of 46 at diagnosis) and the SC182 (with five breast cancer cases with a median age of 46.8 at diagnosis) BRCA2 families were selected for analysis by the HCSC, LUMC, and U Penn model and by Frank 2002, but ruled out by the HUCH model. The median age at diagnosis is low (46 years) in both families, whereas the youngest age at diagnosis is not especially low (44 years).
Interestingly, four out of five positive families misclassified as negative by at least one of these models are related to BRCA2, suggesting that the discrimination power of these models is lower in these families. Indeed, the average probabilities scored by the BRCA1 families with the LUMC, HUCH, and U Penn models were twice as high as those scored by the BRCA2 families. A similar difference was observed with Frank 2002. The differences were smaller and not statistically significant for the HCSC model. The probabilities were almost identical in the BRCA1 and BRCA2 families based on the risk counsellor assessment (table 4).
These data suggest that the presence of BRCA2 families in the study sample impairs the discrimination power of the probabilistic models. To test this hypothesis, we calculated ROC curves in alternative study samples from which the BRCA1 or BRCA2 related families were selectively removed (BRCA1 negative and BRCA2 negative study sample, respectively). The ROC curve areas calculated with the BRCA2 negative study sample were higher than those obtained with the BRCA1 negative study sample and than those calculated with the original study sample in the five models (table 4). However, in the case of the HCSC, these differences were modest. Interestingly, of the 10 positive families with the lowest pre-test probabilities, five families were BRCA2 related in the HCSC model, six in the LUMC model, seven in the U Penn model, eight in Frank 2002, and nine in the HUCH model. Taken together these data indicate that all models, but especially U Penn, Frank 2002, and HUCH, discriminate BRCA1 better than BRCA2 families. Interestingly, there is no similar bias when risk counsellor assessment is considered. The average probability was almost identical in BRCA1 and BRCA2 families (0.56 v 0.59). Moreover, the three families ruled out at 92% sensitivity threshold were BRCA1 related, only four out of the 10 families with the lowest pre-test probability were BRCA2 related, and the calculated ROC area was higher in the BRCA1 negative than in the BRCA2 negative sample (table 4).
The logistic regression models analysed in this study take into consideration cancer phenotype but not pedigree structure. Therefore, it may well be that although the pre-test probabilities calculated by the models are useful to discriminate positive families, they do not reflect true probabilities, and are therefore not good estimators of prevalence. To investigate the relationship between pre-test probability and the prevalence of mutations, we partitioned our data set into quartiles by pre-test probabilities, and the prevalence of mutations was calculated in each quartile (table 5). The sample size was 28 in the first three quartiles and 25 in the last one. The correlation coefficient between pre-test probabilities and prevalence after genetic testing was 0.994 for the HCSC model (p=0.006), 0.947 for the LUMC model (p=0.053), 0.944 for the HUCH model (p=0.056), 0.869 for the U Penn model (p=0.131), and 0.933 for Frank 2002 (p=0.06). These data indicate that a reasonable correlation between pre-test probabilities and mutation prevalence exists, which is the highest in the HCSC model. However, it is also clear that models tend to underestimate prevalence (LUMC and HUCH predictions are below the 95% interval in two quartiles, U Penn in three quartiles, and Frank 2002 in one quartile) and some corrections should be done in the models to fit pre-test probabilities and prevalence. It should be pointed out that the correlation obtained by the risk counsellor (table 5) was among the best, although in this case probabilities were not underestimated but clearly overestimated (approximately two-fold). To test if mutation prevalence was equally underestimated in breast only and breast/ovarian families, we performed a subanalysis of predicted probability/prevalence correlation in these two groups separately (tables 6 and 7). Taken together, the data indicate that all models tend to underestimate mutation prevalence both in breast only families and breast/ovarian families, although this trend is more evident in breast only families.
Since the cloning of the breast cancer susceptibility genes BRCA1 and BRCA2,1,2 a number of statistical models have been developed to predict best the pre-test probability of carrying a germline mutation in one of these genes. Most of these models have not been properly evaluated to date.
Several pre-test probability models do exist.19 Among them, the models by Couch, Shattuck-Eidens, Frank, and BRCAPRO are widely used, although recently other models focused on different ethnic populations have been developed. Each model has been developed with different methodology, sample size, and population characteristics, and consequently each model has unique attributes and limitations, making them difficult to compare directly in a given set of families.
We have performed a retrospective analysis of easy to use statistical models predicting pre-test probability of carrying a BRCA mutation in a series of Spanish breast/ovarian families attending familial cancer clinics (all of them with cancer family history information suggestive of an inherited breast or breast/ovarian cancer predisposition). We did not pretend to test the sensitivity of these models at the lower end of the scale (our cohort did not include low risk families) but to test the performances of these models in high risk families who had already been selected for genetic testing on the basis of cancer family history. There are many probability models which can be investigated.19 We have decided to test four easy to use models which have been originally developed in different ethnic populations (HCSC in Spanish, LUMC in Dutch, HUCH in Finnish, and U Penn in white North American populations) but share a number of characteristics: a logistic regression approach, almost identical entry criteria, and prediction of familial not individual risk. To the best of our knowledge, this is the first time that these models have been tested in an independent set of families. On the other hand, we have tested the performance of Frank 2002, which represents empirical data obtained in white North Americans and it is therefore an empirical and not model approach. However, as our main objective was to test easy to use models in high risk families, we decided not to include in our analysis two other widely used models, Shattuck-Eidens and BRCAPRO. The Shattuck-Eidens model is not applicable to women diagnosed with breast cancer under 30 and therefore it is not applicable to 11 families in our cohort. Moreover, this model is not appropriate for high risk families. On the other hand, the BRCAPRO is not an easy to use model and it has some practical limitations, specific computer software is required, and it is limited to first and second degree relatives. More importantly, in this study we address familial risk while BRCAPRO gives a personal rather than a familial a priori probability and sometimes it is not obvious which proband to select to capture familial risk best.
Overall, our study shows no major differences in discrimination power (as measured by ROC areas) among the models. It should be noted that all models increased the discrimination power of an experienced risk counsellor, suggesting that their use is valuable in the context of clinical counselling and genetic testing to optimise selection of patients for screening. However, given that the ROC areas compare the performances of the models over the complete range of sensitivity, they do not accurately reflect the true merits in familial cancer clinics (as only the upper limit range of sensitivity is relevant in this case). In all the models tested, the optimal probability threshold is not clinically relevant (sensitivities are well below 90%), although it may be useful in some applications, for instance, in identifying a certain number of positive families with minimal screening effort. Our study indicates that these models can improve mutation risk assessment in high risk families commonly seen in a familial cancer clinic. For example, by calculating pre-test probability with the LUMC model and selecting for genetic testing those families scoring a probability greater than 3.5%, all the positive families would be selected (100% sensitivity) and as many as 32% of the true negative families could be considered as low risk families. Using the same model, by selecting all families scoring a probability higher than 7.5%, the sensitivity remains higher than 90%, and 54% of the true negative families could be considered low risk. Therefore, with this model, 41 out of 100 breast/ovarian families currently considered as high risk (38 negative families and three positive families) could be reconsidered as low risk. The HCSC model achieved similar performances. Recently, the performance of BRCAPRO was validated in 148 high risk families.20 By using a >10 BRCA mutation probability, sensitivity was 92%, specificity 32%, and PV−84%. These data, taken together with ours, suggest that although with different intrinsic characteristic, easy to use logistic regression models, computer assisted BRCAPRO, and empirical data on mutation prevalence, as exemplified by Frank 2002, may have similar performances in high risk breast/ovarian families.
Overall, we consider that the models tested in this study perform well. However, some differences are also observed. For instance, our data indicate that within the range of sensitivity required in a genetic cancer clinic, the specificity of U Penn and HUCH models are well below the range observed in the LUMC and HCSC models. At the same time, the BRCA2 families tend to score lower pre-test probabilities than BRCA1 families in all models, but this trend was more evident in the U Penn and HUCH models (table 4). This can be attributed to a strong bias of these models towards BRCA1 families and explains why the overall performance in our sample test (BRCA2 accounting for 50% of the positive families) was much better with the LUMC and HCSC models.
A BRCA1 bias was expected in the U Penn model (as this model is only strictly applicable to BRCA1 mutations)9 but not in the other models. This could reflect a true milder BRCA2 phenotype. This is in agreement with ovarian and breast cancer penetrance estimates of BRCA2, which are lower than those of BRCA1.21 Therefore, a milder phenotype might be expected in these families, which is reflected by scoring lower pre-test probabilities (for instance, ovarian cancer which is very important in all models is less frequent in BRCA2 families). These considerations raise the question of whether these models may be useful in populations with a high prevalence of BRCA2 mutations. However, we have performed a subanalysis in breast only families (12 out of 16 positive families being BRCA2 related), which suggests that BRCA2 families do not impair the performance of models.
Screening protocols like DGGE, SSCP+PTT, or others are widely used for BRCA1 and BRCA2 testing, although they are not 100% sensitive.22 A more sensitive protocol analysis (full sequencing plus gene rearrangement analysis) can be expected to increase slightly the prevalence of mutations in our group of families, probably increasing the performances of both models and risk counsellor assessment.
In conclusion, the four logistic regression models tested may be of use to familial cancer clinics although a BRCA1 related bias was observed in all of them. Models developed in specific populations (such as Dutch or Finnish) can be used in other populations (Spanish in this case). Our data suggest that there is no need for population specific logistic models but rather a need for models based on larger sets of families (this may be more easily accomplished by pooling pedigrees from different populations). At present, pre-test probability models are not good enough to rule out families from genetic analysis solely on the basis of a pre-test probability threshold. However, these models can help a risk counsellor to estimate gene mutation probability in a more consistent way. This estimation is an important initial task for risk counsellors, allows for more focused management, and permits reduction of the number of families considered as high risk.
This work was supported by Fondo de Investigación Sanitaria (FIS) grant number 01/3040, 01/0024-02, 01/0024-03. Javier Godino is a fellow of FIS (99/1906).