External Validation of a Commercial Artificial Intelligence Algorithm on a Large Diverse Population for Detection of Interval Cancers

Background

Several breast artificial intelligence (AI) programs with regulatory approval are available commercially, promising to increase cancer detection rates. However, whether AI can consistently deliver these benefits when applied to diverse populations is largely underexplored. Previous studies have demonstrated the potential of AI in detecting interval cancers but were largely performed using enriched data sets. 

Objective

This study examines the characteristics of interval cancers that were detected by Transpara using data from a real-world population.

Materials and Methods 

A subset of digital 2D screening mammograms acquired between December 2010 and October 2015 at our institution was analyzed using Transpara (version 1.7). Screening examinations were assigned a  Transpara score 1-10, with scores of 1 to 7 considered negative (BI-RADS 1 or 2) and scores of 8 to 10 reflecting an increased likelihood of malignancy. Descriptive statistics of interval cancers (defined as malignancies diagnosed within 12 months of a negative screening examination) were performed.

Results

A total of 26,702 screening mammograms from 20,409 women (54% White, 10% Hispanic, 9% Asian, 9% mixed, 8% Black, 7% not specified, 2% other, and 0.1% American Indian/Alaskan Native). This population contained a total of 167 malignancies (125 invasive and 42 in situ), including 19 interval cancers and 148 screening-detected cancers. The abnormal interpretation rate (Transpara score 10) was 13.7%. A total of 157 malignancies (94%) and 16 interval cancers (84%) were detected by Transpara. Of these AI-detected interval cancers, 14 were invasive and two were in situ (density count of A=1, B=4, C=9, D=2). Invasive interval cancers detected by Transpara were majority luminal A/B subtype with 12 (75%) ER-positive, 11 (69%) PR-positive, zero HER-2 positive, and 13 (81%) with Ki-67 = 20%. Transpara identified an interval cancer median [SD] of 245 (137) days earlier than radiologist detection.

Conclusion

Increased interval cancer detection in a large, diverse cohort may be possible when AI is used as a radiologist-assist tool. Transpara identified 16 cancers that were initially missed by a radiologist (84% of all interval cancers in the cohort), with a propensity to detect invasive disease, spanning all breast densities in a distribution similar to the general screening population. Transpara can identify which patients may benefit from additional imaging and clinical evaluation earlier than by radiologist-review alone; however, at the cost of an increased number of cases flagged for review. Large prospective studies are needed to evaluate AI reliability and performance in combination with a radiologist in a real-world clinical setting.