Validation of the BOADICEA model in a prospective cohort of BRCA1/2 pathogenic variant carriers

Background No validation has been conducted for the BOADICEA multifactorial breast cancer risk prediction model specifically in BRCA1/2 pathogenic variant (PV) carriers to date. Here, we evaluated the performance of BOADICEA in predicting 5-year breast cancer risks in a prospective cohort of BRCA1/2 PV carriers ascertained through clinical genetic centres. Methods We evaluated the model calibration and discriminatory ability in the prospective TRANsIBCCS cohort study comprising 1614 BRCA1 and 1365 BRCA2 PV carriers (209 incident cases). Study participants had lifestyle, reproductive, hormonal, anthropometric risk factor information, a polygenic risk score based on 313 SNPs and family history information. Results The full multifactorial model considering family history together with all other risk factors was well calibrated overall (E/O=1.07, 95% CI: 0.92 to 1.24) and in quintiles of predicted risk. Discrimination was maximised when all risk factors were considered (Harrell’s C-index=0.70, 95% CI: 0.67 to 0.74; area under the curve=0.79, 95% CI: 0.76 to 0.82). The model performance was similar when evaluated separately in BRCA1 or BRCA2 PV carriers. The full model identified 5.8%, 12.9% and 24.0% of BRCA1/2 PV carriers with 5-year breast cancer risks of <1.65%, <3% and <5%, respectively, risk thresholds commonly used for different management and risk-reduction options. Conclusion BOADICEA may be used to aid personalised cancer risk management and decision-making for BRCA1 and BRCA2 PV carriers. It is implemented in the free-access CanRisk tool (https://www.canrisk.org/).


Original research
Validation of the BOADICEA model in a prospective cohort of BRCA1/2 pathogenic variant carriers

INTRODUCTION
3][4][5] Providing more personalised BC risks will enable informed decision-making for the clinical management of BC risk, for example, opting for bilateral risk-reducing mastectomy and its timing.
The BOADICEA model, implemented in the CanRisk tool (https://www.canrisk.org/),predicts the risk of developing BC by considering the combined effects of rare genetic variants in BRCA1, BRCA2, PALB2, CHEK2, ATM, RAD51C, RAD51D and BARD1, a polygenic risk score (PRS), FH, mammographic density (MD) and questionnaire-based risk factors (QRFs) including hormonal, lifestyle and reproductive factors. 6 7][10] However, the model performance has not been evaluated specifically in BRCA1/2 PV carriers.Here, we evaluate the performance of BOADICEA V.6 7 in predicting BC risks in an independent prospective cohort of BRCA1 and BRCA2 PV carriers.

METHODS Subjects
Data on 2879 BRCA1 and 2208 BRCA2 female PV carriers were available from the prospective TRAN-sIBCCS cohort study. 11Participants were recruited

Cancer genetics
via clinical genetics centres in Germany (GC-HBOC), the UK (EMBRACE), France (GENEPSO), the Netherlands (HEBON), Austria (MUV) and Poland (IHCC) and were counselled with regard to their mutation status.All participants were heterozygotes of variants considered to be pathogenic on the basis of widely accepted criteria (ENIGMA consortium; https://enigmaconsortium.org/).
All the participants were actively followed up for cancer incidence and mortality through follow-up questionnaires.In addition, follow-up through linkage with cancer, pathology and death registries has been provided in countries where these registries are available (cancer/death registries in the Netherlands and the UK; pathology registries to collect information on preventive surgeries in the Netherlands and through medical record validation of self-reported preventive surgeries). 11

Censoring process
All participants were followed from age at baseline to the date of BC diagnosis (invasive or ductal carcinoma in situ (DCIS)), bilateral risk-reducing mastectomy, last follow-up, death, baseline plus 6 years or age 80 years, whichever occurred first.Only those with a BC diagnosis were considered affected.A total of 344 women were censored at bilateral prophylactic mastectomy.

Risk prediction, model calibration and discrimination
To exclude patients with potentially prevalent but undiagnosed BC at study recruitment, we predicted the 5-year BC risks starting from the age at study entry plus 1 year.The study used the latest version of BOADICEA V.6 7 implemented in CanRisk V.2.4 (https://canrisk.org/releases/). 12We evaluated the model calibration and discriminatory ability.The overall calibration was assessed by the ratio of the expected (E) to the observed (O) number of patients with incident BC during the 5-year risk prediction period. 13We also assessed the agreement between predicted and observed risks for each individual using the calibration slope, which was calculated by fitting a logistic regression in which the dependent variable was the observed outcome (1: affected; 0: unaffected) and the independent variable was the log odds of the predicted risks.The calibration slope assesses whether the predicted risks are too extreme or conversely too moderate especially at the high and low-risk tails and is expected to be equal to 1 if the model is perfectly calibrated.The observed and expected risks were also compared in categories by grouping the samples in quintiles of predicted risks.Discrimination was assessed by the area under the receiver operating characteristic curve (AUC) and Harrell's C-index. 14To assess the riskstratifying ability of the model, we calculated the proportions of all women who had 5-year BC risks of <1.65%, <3% or <5% which are the commonly used thresholds for discussing risk-reducing options, 15 16 and also examined the proportion of women younger than 50 years old in the low-risk groups who may opt out of or delay the risk-reducing surgeries.
From the total of 5087 women in the entire TRANsIBCCS prospective cohort, women were selected for inclusion in the analysis if they were younger than 74 years old at study entry, if they had no history of cancer or bilateral risk-reducing mastectomy, had more than 1-year follow-up and had data on QRFs and the 313-SNP PRS 17 (online supplemental figure 1).The 313-SNP PRS was standardised using a mean of −0.424 and SD of 0.611 as described in Mavaddat et al. 17 Models were then evaluated in: (1) the cohort of 2979 women who had QRF and PRS data (cohort-1); (2) among those, a cohort of 1804 women with QRF, PRS and pedigree-based cancer FH information available (cohort-2).To allow for the possibility that inclusion in these two subcohorts is non-random with respect to the incident BC status compared with the entire TRANsIBCCS prospective cohort, sampling weights were applied to the final set of eligible women in each subcohort.The sample inclusion probabilities were computed by fitting a logistic regression model in which the outcome (inclusion or not) was dependent on the age at baseline, follow-up duration, incident BC status, the interaction between BC status, age at baseline and the interaction between BC status and the follow-up duration.These were calculated for each country separately, except for Austria, Germany and Poland which were combined due to the limited sample size.The weights were then the inverse of the fitted probabilities for each individual.
All the statistical analyses were performed in R V.3.6.3. 18

RESULTS
A total of 2979 European ancestry BRCA1/2 PV carriers with information on PRS and QRFs were eligible for inclusion in the analysis, of whom 209 (127 BRCA1 and 82 BRCA2 PV carriers) developed BC during the 5-year risk prediction period (cohort-1).Among these, 1804 women (191 with incident BC) also had pedigree-based FH (cohort-2).A detailed summary of the genetic and epidemiological characteristics of the study participants at baseline is shown in online supplemental table 1.We evaluated the model separately in cohort-1 without considering FH and in cohort-2 considering the pedigree-based FH information.Using cohort-1, when considering BRCA1 and BRCA2 PV status only, or BRCA1 and BRCA2 PV status and QRFs, the predicted risks were underestimated (table 1), in particular for women in the higher predicted risk quintiles (figure 1A).The addition of PRS to PV status improved the calibration of the predicted risks (E/O=0.88,95% CI: 0.76 to

Cancer genetics
slope=0.95, 95% CI: 0.90 to 1.00, figure 1A).Similarly, adding PRS to the model with PV and QRF information improved calibration, but discrimination was similar (table 1).Using cohort-2, we first assessed the model predictions by leaving FH out to contrast against the results in cohort-1.The model discriminatory ability and model calibration were similar to the estimates using all 2979 samples (table 1).These suggest that no bias was introduced when using the weighting cohort approach in analysing the data.After including full pedigree FH information in the model 5-year risk predictions, the model was well calibrated (overall E/O=1.07,95% CI: 0.92 to 1.24; calibration slope=1.06,95% CI: 1.00 to 1.12, table 1 and figure 1B).There was a small increase in the model discriminatory ability (Harrell's C=0.70, 95% CI: 0.67 to 0.74; AUC=0.79,95% CI: 0.76 to 0.82, table 1).The model performance was similar in BRCA1 and BRCA2 PV carriers (table 1 and figure 1B).
When considering all risk factors jointly, the predicted 5-year risks varied from 0.1% to 47.6%.A total of 5.8%, 12.9% and 24.0% of women had 5-year BC risks of <1.65%, <3% and <5% with a negative predictive value at the 5% risk threshold of 0.96 (95% CI: 0.96 to 0.97).98.0% of women with a 5-year BC risk of 3% or lower, and 95.7% women (including all BRCA1 PV carriers and 91.5% of BRCA2 PV carriers) with a 5-year BC risk of 5% or lower were younger than 50 years old.Among women younger than 50 years old, 98.5% with 5-year risk of <5% remained unaffected during the risk prediction period.Furthermore, 78.4% of women younger than 30 years old were predicted to have 5-year risk of <5%; among them, 99.2% remained unaffected during the risk prediction period.

DISCUSSION
][10] Since BRCA1/2 PVs are rare in the population, it has not been possible to assess the model performance specifically in PV carriers who are typically seen in clinical genetics. 10Although previous studies have indicated that multiple risk factors modify the BC risks for PV carriers, 2 19-22 their combined effects on risk prediction 1 have not been studied.Here, for the first time, we examined the model performance of the multifactorial BOADICEA model in predicting BC risks in BRCA1 and BRCA2 PV carriers seen at clinical genetics using information on PV, PRS, QRFs and FH jointly and showed that the BOADICEA is well calibrated and discriminated in this population.The results suggest that considering FH when predicting cancer risks for PV carriers seen in clinical genetics, in addition to QRFs and the PRS, can improve the calibration within individual risk categories.Given the majority of such women come from families with cancer FH, and the FH distribution in this cohort is not representative of the distribution in the general population, ignoring FH can result in some underprediction of risk among those who are at higher risk.Therefore, considering only average, published penetrance estimates for the counselling of BRCA1/2 PV carriers typically seen in genetic clinics may underestimate BC risks-a scenario equivalent to the predictions in cohort-1, when using only BRCA1 and BRCA2 PV status.The analyses considered the full pedigree-based FH collected, which included third-degree or more distant relatives.When the analysis was restricted to include only first or second-degree relatives, the model performance was comparable (online supplemental table 2 and online  genetics supplemental figure 2), indicating the collection of less extensive FH may be cost-effective in clinical risk assessment.
][10] Terry et al, using multigenerational pedigree data from Australia, Canada and the USA, 23 showed that a previous version of BOADICEA that considered FH and PV status only had a C-index of 0.59 and overpredicted the 10-year risk for combined BRCA1 and BRCA2 PV carriers in the highest quintile.However, the study used an older version of BOADICEA (V.3).Here, we used the latest model, 7 12 and the analysis included additional risk factors (eg, QRFs and PRS).These, together with the differences in the risk prediction period, the age distributions and other cohort characteristics, make a direct comparison difficult.The present study suggests that the latest model is well calibrated across different risk categories in BRCA1 and BRCA2 carriers.The AUC estimates here could potentially have been overestimated because the risks for healthy women were predicted to the censoring age if they were censored within the risk prediction period.To address this, we also estimated and presented the Harrell's C-index 14 which considers time to event.The Harrell's C-index yielded lower estimates than the AUC for all models.The full model that jointly considered all risk factors provided the highest discrimination as measured by Harrell's C-index (table 1).Another potential explanation of the higher discriminatory ability observed in the current study is most likely due to the larger effect of age on BC risks for BRCA1 and BRCA2 PV carriers compared with the general population and the age range of study participants in this study.When only age was considered in the model, the estimated AUC in the present cohort of BRCA1 and BRCA2 PV carriers was 0.70 (95% CI: 0.66 to 0.73), much higher than the effect of age alone in population-based studies 10 (figure 1A and table 1, cohort-1).
The changes in the C-index (or AUC) by the inclusion of additional risk factors on top of PV status are not significant, based on the associated CIs.This could be a consequence of the relatively small sample size.Nevertheless, the full model that includes PV status, FH, PRS and QRFs has the highest C-index.Given the high BC risks for BRCA1/2 PV, even modest increases in the C-index can lead to changes in risk stratification. 10For example, when considering the half of the PV carriers with the highest predicted risks, the full model identifies 91.2% of incident BCs occurring during the prediction period.This compares with identifying 82.2% of incident BCs when only age and PV are considered.Moreover, the observed variability in the BOADICEA-predicted risks suggests that it is possible to identify BRCA1 and BRCA2 PV carriers with relatively low risks, in particular among women under 50 years old (or women under 30 years), who remain disease-free during the 5-year period.The results suggest that during the genetic counselling process, considering the joint effects of risk factors could be informative for decisions on the timing of risk-reducing interventions.
Analysis was repeated by censoring women diagnosed with DCIS as unaffected at the age at diagnosis (online supplemental table 3 and online supplemental figure 3).The model discriminatory ability as measured by the AUC remained similar to the overall analyses, when DCIS was considered as affected; as expected, there was some increase in the ratio of E/O cases (1.18; 95% CI: 1.01 to 1.38) and the calibration slope (1.11, 95% CI: 1.05 to 1.17) for the full model using cohort-2, suggesting some overall overprediction of risks.However, the model was still well calibrated within quintiles of predicted risk, with no significant differences between the observed and predicted risks (online supplemental figure 3).
BOADICEA does not consider the potential effect of riskreducing salpingo-oophorectomy (RRSO) on BC risk.Censoring at RRSO resulted in some miscalibration in quintiles of predicted risk (online supplemental figure 4).Previous studies have shown that MD is also a risk factor for BC in BRCA1 and BRCA2 PV carriers. 24 25Although BOADICEA considers the effect of MD in predicting BC risks, the number of women with MD data at baseline was too small (N=794) to allow for a model assessment, which is a major limitation of the study.The number of BRCA1 and BRCA2 carriers was relatively small when divided by age.Nevertheless, when assessed separately by age 50 years, the full model was well calibrated in the <50 years age group.There was some overprediction in women aged 50 years or older with E/O ratio of 1.28 (95% CI: 0.96 to 1.71), but this was not significant (online supplemental figure 5 and online supplemental table 4).Larger number of carriers at older ages, with a larger number of incident cancers, will be required to assess the predicted risks with greater precision, in particular among different risk categories. 26BOADICEA assumes that the joint effects of BRCA1/2 PVs with the PRS and QRFs are multiplicative on the risk scale, 6 7 but studies suggested that deviations from the multiplicative model may exist. 2 5 The BOADICEA model assumes an age-dependent effect of the PRS, as previously described 2 and the present study suggests the BOADICEA assumptions provide valid risks for BRCA1/2 PV carriers.Much larger sample sizes will be required to detect small deviations between the observed and predicted risks.
In conclusion, in the overall TRANsIBCCS prospective cohort of BRCA1 and BRCA2 PV carriers who were ascertained through genetic clinics, primarily on the basis of cancer FH, the multifactorial BOADICEA provided good discriminatory ability and was calibrated in predicting 5-year risks within different risk categories.The results suggest that BOADICEA may be used to aid personalised cancer risk management and decision-making for BRCA1 and BRCA2 PV carriers.However, the number of PV carriers by country was too small to assess differences in the predictive ability of the model by country or to assess how potential differences in data collection practices for outcomes and elective surgeries by country/study may influence the results.Future studies with much larger sample sizes of BRCA1 and BRCA2 PV carriers by country and with long-term follow-up should be performed to assess BOADICEA.Furthermore, it will be important to assess whether the prediction performance can be improved by using BRCA1/2-specific parameter estimates for the effects of the PRS and QRFs in the model.

Figure 1
Figure 1 Observed and expected (E/O) 5-year breast cancer risks in quintiles of predicted risks: (A) using the cohort-1 samples (N=2979) under the models considering null (age only), PV, PV+PRS, PV+QRFs and PV+QRFs+PRS; and (B) using the cohort-2 samples with FH information (N=1804) under the models considering PV+QRFs+PRS and FH+QRFs+PRS+PV.The dashed line is the diagonal line with slope equal to 1 (corresponding to E/O ratio of 1 for each quintile).FH, family history; PRS, polygenic risk score; PV, pathogenic variant status in BRCA1 and BRCA2; QRFs, questionnaire-based risk factors.
No study has assessed the clinical validity of the multifactorial BOADICEA model for predicting future breast cancer risks specifically for BRCA1/2 pathogenic variant (PV) carriers.
1.01, calibration WHAT IS ALREADY KNOWN ON THIS TOPIC ⇒ ⇒ BOADICEA is freely available via the CanRisk tool (www.canrisk.org).Rather than relying solely on average published penetrance estimates commonly used in genetic clinics for counselling of BRCA1/2 PV carriers, BOADICEA offers more personalised breast cancer risks.This can facilitate informed decision-making regarding the clinical management of breast cancer risk, including considerations for surveillance and the timing of risk-reducing surgery.

Table 1
Calibration and discrimination of 5-year predicted breast cancer (BC) risks under the BOADICEA model using different risk factor combinations AUC, area under the receiver operating characteristic curve; FH, family history; PRS, polygenic risk score; PV, pathogenic variant status in BRCA1 and BRCA2; QRFs, questionnairebased risk factors.