Article Text
Abstract
Background/aims: The identification of Lynch syndrome is hampered by the absence of specific diagnostic features and underutilisation of genetic testing. Prediction models have therefore been developed, but they have not been validated for a clinical genetic setting. The aim of the present study was to evaluate the usefulness of currently available prediction models.
Methods: The authors collected data of 321 index probands who were referred to the department of clinical genetics of the Erasmus Medical Center because of a family history of colorectal cancer. These data were used as input for five previously published models. External validity was assessed by discriminative ability (AUC: area under the receiver operating characteristic curve) and calibration. For further insight, predicted probabilities were categorised with cut-offs of 5%, 10%, 20% and 40%. Furthermore, costs of different testing strategies were related to the number of extra detected mutation carriers.
Results: Of the 321 index probands, 66 harboured a germline mutation. All models discriminated well between high risk and low risk index probands (AUC 0.82–0.84). Calibration was well for the Premm1,2 and Edinburgh model, but poor for the other models. Cut-offs could be found for the prediction models where costs could be saved while missing only few mutations.
Conclusions: The Edinburgh and Premm1,2 model were the models with the best performance for an intermediate to high risk setting. These models may well be of use in clinical practice to select patients for further testing of mismatch repair gene mutations.
Statistics from Altmetric.com
Lynch syndrome, also known as hereditary non-polyposis colorectal cancer, is the most common form of hereditary colorectal cancer (CRC). This syndrome is characterised by early onset of CRC and endometrial cancer. In addition, tumours of the stomach, small bowel, urinary tract, ovaries, brain and skin occur at higher frequencies in Lynch syndrome families compared to the general population.1 2 Lynch syndrome is caused by mutations in the mismatch repair (MMR) genes (MLH1, MSH2, MSH6 and PMS2). As a consequence, Lynch syndrome associated tumours exhibit microsatellite instability (MSI) and absent protein expression of the affected gene. Early identification of mutation carriers is important because of the high risk for colorectal cancer and other malignancies.
Identification of mutation carriers can be based on a combination of clinical diagnostic criteria, the Amsterdam II criteria (AC II) and the revised Bethesda guidelines (BG).3 4 The AC II, however, have a limited sensitivity for the detection of mutation carriers and therefore it is suboptimal for identification of mutation carriers. The BG were therefore developed to identify individuals eligible for molecular screening by MSI and immunohistochemical (IHC) staining of MMR proteins. However, some components of the BG are rather complex. Molecular screening by MSI testing and IHC staining of proteins has a higher sensitivity compared to the AC II and BG,5 6 7 8 9 10 11 but MSI with absent staining of MLH1 and PMS2 is also present in 15% of sporadic CRC cases.12 13 14 Furthermore, MSI and IHC analysis can only be performed when tumour tissue is available. The optimal diagnostic strategy for Lynch syndrome is still under debate. Performing MSI and IHC in CRC cases selected by the pathologists has been proposed as a diagnostic strategy by some authors,9 15 16 while others propose fulfilment of BG followed by MSI and IHC analysis as a more effective strategy.10 These different views emphasise the need for other diagnostic tools in the identification of Lynch syndrome.
In recent years, several models have been developed to predict the likelihood of carrying a germline mutation.11 17 18 19 20 These models use information from personal and family history as input to predict the probability of mutation carriership. A major advantage of prediction models is that these models give a quantitative estimate instead of a binary (yes/no) assessment as provided by the clinical diagnostic criteria. Mutation prediction models are thus potentially useful in clinical practice to optimise the identification of Lynch syndrome carriers, but the performances of these models have not been evaluated other than in similar settings as where they were developed. External validation is necessary to study generalisability of these models. Moreover, we need to consider the cut-off that should be used to select patients for further evaluation for Lynch syndrome. For example, a cut-off of 5% may be preferable to 10% if the additional costs of testing patients with risks between 5% and 10% are reasonable compared to the detection of some extra mutation carriers in this group.
The aim of the present study was to evaluate the usefulness of five easily applicable mutation prediction models in a clinical population of familial CRC.
Methods
Study population
Since 1992, high risk families for Lynch syndrome have been referred by general practitioners and medical specialists to the department of clinical genetics of the Erasmus MC University Medical Center for oncogenetic counselling. We included families with one or more subjects who had undergone cancer risk assessment for Lynch syndrome. From every family one or more family members underwent tumour analysis by MSI and IHC analysis, and/or underwent germline mutation analysis for mutations in the MLH1, MSH2 and MSH6 genes. In total, 321 unrelated families with 17 552 members were included in this study.
In every family, an index proband was identified, defined as the youngest family member diagnosed with CRC and in which MSI, IHC analysis and/or germline mutation analysis had been performed. Demographic, clinical and tumour related characteristics of the index proband and a detailed family history were obtained from medical records and family pedigrees. Only findings that were confirmed by pathology or medical records were included. Family pedigrees were traced backwards and laterally as far as possible, at least to second-degree relatives. For all the affected individuals the age of cancer onset, type, location, tumour stage of the malignancy, the presence and number of colorectal adenomas and results of MSI, IHC and/or germline mutation analysis were recorded. Furthermore, the presence and number of colorectal adenomas and the results of germline mutation analysis in healthy individuals was recorded. We defined synchronous colorectal cancers as a second primary cancer diagnosed within 6 months and metachronous colorectal cancers as a second primary cancer, diagnosed at least 6 months after the primary cancer.
Molecular screening
MSI analysis was performed on paired tumour and normal tissue DNA using a panel of five validated microsatellite markers, as described previously.21 Tumours were regarded as MSI-high if at least two of the five markers showed instability, MSI-low if one of the five markers showed instability, and MSI-stable if none of the markers showed instability.
Immunostaining for MLH1, MSH2 and MSH6 was carried out as described previously.21 The slides were scored as either negative (that is, the absence of detectable nuclear staining of cancer cells), or positive for MLH1, MSH2 and MSH6 staining. Individuals found to have tumours with either an MSI-high phenotype or lack of MLH1, MSH2 or MSH6 protein expression were considered MMR deficient, while individuals with a tumour exhibiting an MSI-stable phenotype with normal MLH1, MSH2 or MSH6 protein expression were considered MMR proficient.
Germline mutation analysis
All individuals with an MMR deficient tumour underwent germline mutation analysis of the MLH1, MSH2 and MSH6 gene. Germline mutation analysis was performed by sequencing and multiplex ligation dependent probe amplification (MRC Holland).
Classification of index probands
Index probands were considered positive if germline mutation analysis revealed a mutation in one of the mismatch repair genes. Index probands in whom germline mutation analysis detected an unclassified variant or no mutation were classified as non-mutation carrier.
Index probands in which only MSI and IHC analysis had been performed and the analysis showed an MMR proficient tumour were also classified as non-mutation carrier.
Prediction models
Mutation probability estimates for MLH1, MSH2 and MSH6 were calculated using five, previously published prediction models: the Leiden,11 PREMM1,2,19 Edinburgh,18 UK-Amsterdam Plus (UK-Ams),17 and UK-Alternative (UK-Alt) model.17
The Leiden and PREMM1,2 model predict the likelihood of finding an MLH1 or MSH2 gene mutation, while the Edinburgh, UK-Ams and UK-Alt model predict the likelihood of finding a MLH1, MSH2 or MSH6 gene mutation in high risk individuals (table 1).
Cost effectiveness analysis
We estimated the costs and effects (number of detected mutations) of different diagnostic strategies. The costs of tumour MMR deficiency (MSI and IHC) testing were estimated as €590, the tariff recommended by the Dutch Pathology association (www.pathology.nl). The costs of germline mutation analysis were assumed to be €620 for each MMR gene.22 In case of direct germline mutation analysis, three MMR genes have to be analysed, thus the costs associated with this strategy was €1860. Tumour MMR deficiency analysis is mostly indicative for the mutated MMR gene and therefore the costs of subsequent germline mutation analysis was calculated for a single MMR gene (€620). Costs were related to the number of mutations detected in MLH1, MSH2, or MSH6 genes for all prediction models. The diagnostic strategies included: (1) direct germline mutation analysis in all index probands; (2) tumour MMR deficiency analysis in all index probands followed by germline mutation analysis in those with MMR deficient tumours; and (3) a strategy in which prediction models were used to select cases eligible for additional tumour MMR deficiency analysis and subsequent germline mutation analysis. Strategy 3 considered consecutive cut-off values for the predicted probability of a mutation of each prediction model, such that incremental cost effectiveness could be calculated, expressed as extra costs (k€ = thousands of Euro) per extra detected mutation.
Statistical analysis
Sensitivity and specificity were calculated for the five different models with respect to the presence of a MLH1, MSH2, or MSH6 germline mutation, considering cut-offs for the predicted probability of a mutation of 5%, 10%, 20%, and 40%. The predicted probabilities for each index proband included in the study were calculated using an extensive coding program in the SPSS software package (version 12.0.1, SPSS Inc, Chicago, Illinois, USA), with verification in an independently developed Microsoft Excel spreadsheet (Microsoft Corporation, Redmond, Washington, USA). The performance of the models was assessed with respect to discrimination and calibration. Discrimination is the model’s ability to separate patients with and without mutations. To quantify discrimination, the area under the receiver operating characteristic curve (AUC) was calculated. A model with an AUC of 0.50 has no discriminative power, while an AUC of 1.0 reflects perfect discrimination. Calibration reflects the ability of a model to produce unbiased estimates of the probability of an outcome. For example, if patients with certain characteristics are predicted to have a 10% chance of carrying a mutation, the observed prevalence of mutation carriership should also be 10%. Calibration was assessed graphically by plotting observed outcome against the predicted probability (0–100%). Calibration was further tested with the Hosmer–Lemeshow goodness-of-fit test.23 For the five models we compared observed outcome versus predicted risks for each decile. Discrimination and calibration were calculated using R software (Version 2.5.1, the R Foundation for Statistical Computing). All p values were two sided and a value of p<0.05 was considered statistically significant.
Results
Index proband characteristics
A total of 321 unrelated index probands (155 male/166 female) of 321 families were included in this study (table 2).
Twenty-seven (8%) and 155 (48%) of them met the Amsterdam II criteria and Bethesda guidelines, respectively. Among the MLH1 and MSH2 mutation carriers, the BG were positive in 84% and 91%, respectively. Among the MSH6 mutation carriers, the BG were positive in only 50%. The index probands were primarily analysed by MSI and IHC (n = 277; 86%) or by direct germline mutation analysis (n = 44; 14%). Of the 277 (86%) index probands who underwent molecular analysis by MSI and IHC, 54 (19%) index probands had a tumour demonstrating MMR deficiency. Germline mutation analysis was performed in 175 (55%) index probands, 54 (31%) with an MMR deficient tumour, 77 (44%) with an MMR proficient tumour, and 44 (25%) who directly underwent germline mutation analysis. In total, 66 mutations in either MLH1 (n = 25; 38%), MSH2 (n = 23; 35%) or MSH6 (n = 18; 27%) were identified. In 146 (45%) index probands only MSI and IHC analysis was performed which showed an MMR proficient tumour. These index probands were considered as non-mutation carriers.
Detection of mutations
The AC II had a sensitivity for identification of mutation carriers of 30% (20/66) and a specificity of 89% (228/255). The BG missed 15 of the 66 mutation carriers, corresponding with a sensitivity of 77% and a specificity of 59% (151/255). Molecular analysis with MSI and IHC in tumours of 277 index probands identified 54 (19%) index probands with an MMR deficient tumour, corresponding with a sensitivity of 100% and a specificity of 89% for detection of mutation carriers.
Performance of prediction models
All prediction models discriminated well between high risk and low risk probands, with an AUC of 0.82 for the UK-Ams and UK-Alt model and an AUC of 0.84 for the Leiden, Premm1,2 and Edinburgh model. Calibration curves showed considerable differences between observed mutation frequency and predictions from the Leiden, UK-Ams and UK-Alt models (fig 1, Hosmer–Lemeshow goodness of fit test p<0.0001). For example, the calibration curve in the Leiden model was above the dotted ideal line. This implies that most predicted probabilities for mutation carriership were systematically too low—for example, index probands with a predicted probability of 40% actually had an observed probability for a mutation carriership around 70%.
Using a probability cut-off value of ⩾5%, the UK-Alt model identified all mutation carriers, corresponding with a sensitivity of 100% (table 3). The 100% sensitivity of the UK-Alt model is explained by the fact that the predicted probabilities were systematically too high, as shown in fig 1. The two well-calibrated models, the Premm1,2 and Edinburgh, both had a sensitivity of 98% at a 5% cut-off. The specificity of both models at a 5% cut-off was 22% and 9%, respectively. The Leiden model had the lowest sensitivity at a 5% cut-off (73%) and the highest specificity (80%), which relates to systematic underestimation of actual frequencies of mutations (fig 1). Using cut-off values of 10% or above resulted in a progressive loss of sensitivity of all the models (table 3).
As expected, the combination of prediction models with subsequent MMR deficiency analysis before MMR mutation testing increased the specificity associated with each model. For example, at a 5% cut-off the specificity of the Premm1,2 model increased from 22% to 82%.
Cost effectiveness of different diagnostic strategies
The most expensive diagnostic strategy was direct germline mutation analysis in all index probands (k€597 or k€9.1 per mutation). Performing tumour MMR deficiency analysis in all index probands followed by germline mutation analysis in those with a MMR deficient tumour was less expensive (k€248, k€3.8 per mutation) (table 4).
Using prediction models to select cases for additional tumour MMR deficiency analysis and subsequent germline mutation analysis was associated with lower costs, at the expense of missing mutations. At a 5% cut-off, the Leiden model was associated with the lowest costs (k€84), but 23 mutations would be missed. To detect all mutations, the extra costs per extra detected mutation were k€7. The Edinburgh and Premm1,2 models detected nearly all mutations at a 5% cut-off (65/66 and 64/66, respectively). To detect the one or two missed mutations would cost an additional k€16 or k€20, respectively. With a cut-off of 10%, three (65–62) or 11 (64–53) more mutations would be missed. The extra costs were k€37 and k€8 per extra mutation for the Edinburgh and Premm1,2 model, respectively. A cut-off of 20% would lead to many more mutations missed (62–55 = 7 and 53–41 = 12, respectively), with costs of k€11 and k€4 per extra detected mutation. A cut-off of 5% for the Premm1,2 model hence had a similar cost effectiveness as a threshold of 10% for the Edinburgh model, which is related to the slight difference in calibration at low predicted probabilities (fig 1). Accepting a threshold for the costs per extra detected mutation of approximately k€10 would result in the use of different cut-off values. The Leiden and UK-Ams model should be used with a cut-off value <5%, the UK-Alt and Premm1,2 model with a ⩾5% cut-off, and the Edinburgh model with a cut-off value of ⩾10%.
Discussion
Identification of subjects at high risk for Lynch syndrome still remains difficult. In this study we found adequate discriminative ability of five different mutation prediction models for Lynch syndrome using clinical data of 321 index probands with suspicion of Lynch syndrome. Among the five models considered, the Premm1,2 and Edinburgh model had the best performance in predicting mutation carriership, because these two models had the highest discriminative ability and were well calibrated. Combining prediction models with tumour MMR deficiency analysis resulted in a substantial increase of the specificity.
The calibration of the Leiden, UK-Ams and UK-Alt models was relatively poor, reflecting that predicted probabilities were systematically too high or too low. Although the Edinburgh model was developed using population based data, this model was well calibrated in our intermediate to high risk population. The Premm1,2 model and the Leiden model were designed to predict only the presence of MLH1 and MSH2 mutations. Recently, the Premm1,2 model was evaluated in a population based cohort of 1222 colorectal cancer patients, and identified all MLH1 and MSH2 mutation carriers.24 However, the number of identified mutations (n = 8) was very low, limiting the strength of the conclusions from this study.
In subjects suspected of Lynch syndrome, high sensitivity is important to identify mutation carriers. Tumour MMR testing in all by definition has the highest sensitivity (100%), but the lowest specificity (0%). This strategy was associated with high costs. Therefore, selection of high risk individuals may be considered. The low sensitivity of the AC II and the complexity of the BG necessitate new diagnostic strategies to identify individuals at risk for Lynch syndrome. Prediction models have a higher sensitivity compared to the AC II and BG if relatively low cut-offs are used for the predicted probability of mutation. With cut-offs of 5–10% such models can well be useful in a clinical setting. External validation showed that especially the Premm1,2 and Edinburgh model were accurate in predicting mutation carriership, with a sensitivity of 98% at a 5% cut-off. Although the number of variables in the Edinburgh model is only half of those in the UK-Ams and UK-Alt, the Edinburgh model performed better in our study population. The difference in performance between models can largely be explained by the difference in included predictors. In clinical practice, the Premm1,2 and Edinburgh model can be used to identify high risk individuals who are eligible for tumour MMR deficiency analysis.
Using prediction models led to lower costs than tumour MMR deficiency analysis in all CRC cases at the expensive of missing no or only few mutation carriers. The cost effectiveness is determined by the cut-off value used, with lower cut-off values resulting in higher costs per extra detected mutation. The optimal threshold in terms of cost effectiveness is not known and may depend on the specific health care setting. One might accept high costs per extra detected mutations when proven mutation carriers will follow colonoscopic surveillance. Such surveillance is highly effective in Lynch syndrome. A 25-year-old person would gain 13.5 years of life expectance compared to no surveillance.25 26 The reported compliance in mutation carriers is approximately 90%, thus the expected gain in life expectancy may be only slightly less.27 Further establishment of the optimal threshold in terms of cost effectiveness and life years gained is needed. We note that the availability of user friendly formats, such as the web based Premm1,2 model (http://www.dana-farber.org/pat/cancer/gastrointestinal/crc-calculator/), enables easy calculation of predicted probabilities in clinical practice.
Our study population consisted of families that had been counselled at the department of clinical genetics, including the use of “family forms”, where all family members were noted by name, gender and birth date, and family members affected by cancer were noted. This enabled further verification of the family history by medical records and pathology reports. The family history was hence as complete as possible, which can be considered a strength of our study. But the results of this study may not apply to the performance of mutation prediction models in the general population, because we used an intermediate to high risk study population. Another limitation of our study was that not all probands underwent germline mutation analysis. We classified the probands with normal tumour MMR results and in whom no germline mutation analysis was performed as non-mutation carriers. Theoretically, these probands could harbour a mutation. However, the reported sensitivity of both MSI (80–100%) and IHC analysis (85–95%) are high.8 9 10 18 28 29 30 Finally, we only evaluated logistic regression based prediction models and therefore excluded the MMRpro model. MMRpro is a Mendelian model, which uses Bayesian calculations considering information from the full pedigree, including unaffected relatives. The performance of MMRpro may be similar to the Edinburgh and Premm1,2 models in the clinical setting, but further evaluation is needed.
In conclusion, we have evaluated five easily applicable mutation prediction models for Lynch syndrome in a selected clinical population. The models have a high accuracy and cost effectiveness for detecting germline mutations in the mismatch repair genes. The Edinburgh and Premm1,2 model had the best performance in an intermediate to high risk setting, and these models may well be of use in clinical practice. The poor calibration of the Leiden, UK-Ams and UK-Alt model hampers direct application of these mutation prediction models. Further evaluation of mutation prediction models across different settings is needed.
REFERENCES
Footnotes
Competing interests None.
Patient consent Not required.
Provenance and Peer review Not commissioned; externally peer reviewed.