Accurate Classification of Diminutive Colorectal Polyps Using Computer-Aided Analysis

Published:October 14, 2017DOI:

      Background & Aims

      Narrow-band imaging is an image-enhanced form of endoscopy used to observed microstructures and capillaries of the mucosal epithelium which allows for real-time prediction of histologic features of colorectal polyps. However, narrow-band imaging expertise is required to differentiate hyperplastic from neoplastic polyps with high levels of accuracy. We developed and tested a system of computer-aided diagnosis with a deep neural network (DNN-CAD) to analyze narrow-band images of diminutive colorectal polyps.


      We collected 1476 images of neoplastic polyps and 681 images of hyperplastic polyps, obtained from the picture archiving and communications system database in a tertiary hospital in Taiwan. Histologic findings from the polyps were also collected and used as the reference standard. The images and data were used to train the DNN. A test set of images (96 hyperplastic and 188 neoplastic polyps, smaller than 5 mm), obtained from patients who underwent colonoscopies from March 2017 through August 2017, was then used to test the diagnostic ability of the DNN-CAD vs endoscopists (2 expert and 4 novice), who were asked to classify the images of the test set as neoplastic or hyperplastic. Their classifications were compared with findings from histologic analysis. The primary outcome measures were diagnostic accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and diagnostic time. The accuracy, sensitivity, specificity, PPV, NPV, and diagnostic time were compared among DNN-CAD, the novice endoscopists, and the expert endoscopists. The study was designed to detect a difference of 10% in accuracy by a 2-sided McNemar test.


      In the test set, the DNN-CAD identified neoplastic or hyperplastic polyps with 96.3% sensitivity, 78.1% specificity, a PPV of 89.6%, and a NPV of 91.5%. Fewer than half of the novice endoscopists classified polyps with a NPV of 90% (their NPVs ranged from 73.9% to 84.0%). DNN-CAD classified polyps as neoplastic or hyperplastic in 0.45 ± 0.07 seconds—shorter than the time required by experts (1.54 ± 1.30 seconds) and nonexperts (1.77 ± 1.37 seconds) (both P < .001). DNN-CAD classified polyps with perfect intra-observer agreement (kappa score of 1). There was a low level of intra-observer and inter-observer agreement in classification among endoscopists.


      We developed a system called DNN-CAD to identify neoplastic or hyperplastic colorectal polyps less than 5 mm. The system classified polyps with a PPV of 89.6%, and a NPV of 91.5%, and in a shorter time than endoscopists. This deep-learning model has potential for not only endoscopic image recognition but for other forms of medical image analysis, including sonography, computed tomography, and magnetic resonance images.

      Graphical abstract


      Abbreviations used in this paper:

      CAD (computer-aided diagnosis), CRC (colorectal cancer), DNN-CAD (computer-aided diagnosis with a deep neural network), NBI (narrow-band imaging), NPV (negative predictive value), PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations), PPV (positive predictive value), ROI (regions of interest)
      To read this article in full you will need to make a payment
      AGA Member Login
      Login with your AGA username and password.
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Liou J.M.
        • Lin J.T.
        • Huang S.P.
        • et al.
        Screening for colorectal cancer in average-risk Chinese population using a mixed strategy with sigmoidoscopy and colonoscopy.
        Dis Colon Rectum. 2007; 50: 630-640
        • Walsh J.M.
        • Terdiman J.P.
        Colorectal cancer screening: scientific review.
        JAMA. 2003; 289: 1288-1296
        • Winawer S.J.
        • Zauber A.G.
        • Ho M.N.
        • et al.
        Prevention of colorectal cancer by colonoscopic polypectomy. The National Polyp Study Workgroup.
        N Engl J Med. 1993; 329: 1977-1981
        • Zauber A.G.
        • Winawer S.J.
        • O'Brien M.J.
        • et al.
        Colonoscopic polypectomy and long-term prevention of colorectal-cancer deaths.
        N Engl J Med. 2012; 366: 687-696
        • Patel S.G.
        • Schoenfeld P.
        • Kim H.M.
        • et al.
        Real-time characterization of diminutive colorectal polyp histology using narrow-band imaging: implications for the resect and discard strategy.
        Gastroenterology. 2016; 150: 406-418
        • Tanaka S.
        • Sano Y.
        Aim to unify the narrow band imaging (NBI) magnifying classification for colorectal tumors: current status in Japan from a summary of the consensus symposium in the 79th Annual Meeting of the Japan Gastroenterological Endoscopy Society.
        Dig Endosc. 2011; 23: 131-139
        • Hewett D.G.
        • Kaltenbach T.
        • Sano Y.
        • et al.
        Validation of a simple classification system for endoscopic diagnosis of small colorectal polyps using narrow-band imaging.
        Gastroenterology. 2012; 143: 599-607
        • Ladabaum U.
        • Fioritto A.
        • Mitani A.
        • et al.
        Real-time optical biopsy of colon polyps with narrow band imaging in community practice does not yet meet key thresholds for clinical decisions.
        Gastroenterology. 2013; 144: 81-91
        • Kuiper T.
        • Marsman W.A.
        • Jansen J.M.
        • et al.
        Accuracy for optical diagnosis of small colorectal polyps in nonacademic settings.
        Clin Gastroenterol Hepatol. 2012; 10: 1016-1020
        • Kominami Y.
        • Yoshida S.
        • Tanaka S.
        • et al.
        Computer-aided diagnosis of colorectal polyp histology by using a real-time image recognition system and narrow-band imaging magnifying colonoscopy.
        Gastrointest Endosc. 2016; 83: 643-649
        • Tischendorf J.J.
        • Gross S.
        • Winograd R.
        • et al.
        Computer-aided classification of colorectal polyps based on vascular patterns: a pilot study.
        Endoscopy. 2010; 42: 203-207
        • Misawa M.
        • Kudo S.E.
        • Mori Y.
        • et al.
        Characterization of colorectal lesions using a computer-aided diagnostic system for narrow-band imaging endocytoscopy.
        Gastroenterology. 2016; 150: 1531-1532
      1. Tensorflow. Available at: (Accessed April 25, 2016).

      2. Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016;2818–2826.

      3. TensorBoard. Available at: (Accessed April 25, 2016).

      4. The Paris endoscopic classification of superficial neoplastic lesions: esophagus, stomach, and colon: November 30 to December 1, 2002.
        Gastrointest Endosc. 2003; 58: S3-S43
        • Rex D.K.
        • Kahi C.
        • O'Brien M.
        • et al.
        The American Society for Gastrointestinal Endoscopy PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations) on real-time endoscopic assessment of the histology of diminutive colorectal polyps.
        Gastrointest Endosc. 2011; 73: 419-422
        • Vogelstein B.
        • Fearon E.R.
        • Hamilton S.R.
        • et al.
        Genetic alterations during colorectal-tumor development.
        N Engl J Med. 1988; 319: 525-532
        • Abu Dayyeh B.K.
        • Thosani N.
        • et al.
        • ASGE Technology Committee
        ASGE Technology Committee systematic review and meta-analysis assessing the ASGE PIVI thresholds for adopting real-time endoscopic assessment of the histology of diminutive colorectal polyps.
        Gastrointest Endosc. 2015; 81: 502.e1-502.e16
        • Su M.Y.
        • Hsu C.M.
        • Ho Y.P.
        • et al.
        Comparative study of conventional colonoscopy, chromoendoscopy, and narrow-band imaging systems in differential diagnosis of neoplastic and nonneoplastic colonic polyps.
        Am J Gastroenterol. 2006; 101: 2711-2716
        • Chiu H.M.
        • Chang C.Y.
        • Chen C.C.
        • et al.
        A prospective comparative study of narrow-band imaging, chromoendoscopy, and conventional colonoscopy in the diagnosis of colorectal neoplasia.
        Gut. 2007; 56: 373-379
        • Rogart J.N.
        • Jain D.
        • Siddiqui U.D.
        • et al.
        Narrow-band imaging without high magnification to differentiate polyps during real-time colonoscopy: improvement with experience.
        Gastrointest Endosc. 2008; 68: 1136-1145
        • Sikka S.
        • Ringold D.A.
        • Jonnalagadda S.
        • et al.
        Comparison of white light and narrow band high definition images in predicting colon polyp histology, using standard colonoscopes without optical magnification.
        Endoscopy. 2008; 40: 818-822
      5. Abadi M, Agarwal A, Barham P, et al. TensorFlow: large-scale machine learning on heterogeneous distributed systems. Available at: (Accessed April 14, 2016).

      6. Frome A, Corrado GS, Shlens J et al. DeVISE: a deep visual-semantic embedding model. Available at: (Accessed April 15, 2016).

      7. Rosenberg C. Improving photo search: a step across the semantic gap. Available at: (Accessed April 15, 2016).

      8. Szegedy C, Wei L, Yangqing J, et al. Going deeper with convolutions. Paper presented at the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7–12 June 2015; Boston, MA, USA.

        • Iwatate M.
        • Sano Y.
        • Hattori S.
        • et al.
        The addition of high magnifying endoscopy improves rates of high confidence optical diagnosis of colorectal polyps.
        Endosc Int Open. 2015; 3: E140-E145
        • Rex D.K.
        Narrow-band imaging without optical magnification for histologic analysis of colorectal polyps.
        Gastroenterology. 2009; 136: 1174-1181
        • Rees C.J.
        • Rajasekhar P.T.
        • Wilson A.
        • et al.
        Narrow band imaging optical diagnosis of small colorectal polyps in routine clinical practice: the Detect Inspect Characterise Resect and Discard 2 (DISCARD 2) study.
        Gut. 2017; 66: 887-895