Novel Content Aware Pixel Abstraction for Image Semantic Segmentation

Mehak Maqbool Memon, Manzoor Ahmed Hashmani, Syed Sajjad Hussain Rizvi


Image semantic segmentation is one of the recently researched topics due to the rise in visual deep learning-based applications. These applications work on the meaningful segments of the visual scene created by the application base. The literature identifies the current research status and highlights the existing problems of image semantic segmentation algorithms. These problems include the handling of complex images. Complex images can be of form high/low pixel intensities or dense structure regions of the image. Existing state-of-the-art deep learning algorithms fail to segment complex images semantically. For semantic segmentation of complex images, deep learning algorithms are proposed to be accompanied by pixel abstraction algorithm. The pixel abstraction algorithm creates atomic segments of the visual scene called super-pixels.  Super-pixels generate feature vectors supporting the same regions. These feature vectors reduce the computational complexity to create semantic segments of the visual scene. The pixel abstraction algorithms lack functionality due to different aspects, one of which is the initial hand-crafted seed from the user to create super-pixels that do not work for all types of visual scenarios to create accurate semantic segments. The second aspect that limits pixel abstraction algorithms' functionality is the distance measure used for super-pixel (cluster) creation. The distance measures employed in existing algorithms do not capture content-aware information of visual scene; instead, end-up creating super-pixels based on Euclidean distance, which is based on straight line distance. Hence, the created pixels are distorted and irregular. For proving the flawed functionality of the existing super-pixel creation algorithm, detailed visual analysis is presented, uncovering the indicators for future research towards the development of a novel algorithm creating continuous and regular super-pixels. For creating content-aware super-pixels, the article describes an automatic super-pixel creation algorithm based on the idea of capturing image information in relevance to the content present in it. For example, we illustrate the proposed framework in detail as two modular approaches to improve the resulting super-pixels' quality. Firstly, to automate the entire process, the probability density function is proposed to initialize the cluster centers such that hand-crafted seed is not required from the user. Secondly, to retrieve fine-grained object boundaries, a novel distance measure with induced content-aware nature and complex image handling is proposed. The novel algorithm has the potential to tackle the problem of discontinuity and irregularity of retrieved segment boundaries.


Keywords:  Super-Pixels, Simple Linear Iterative Clustering, Distance Measures.

Full Text:



MARMANIS D., SCHINDLER K., WEGNER, J.D., and GALLIANI, S. Classification with an edge: Improving semantic image segmentation with boundary detection. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 135: 158-172. doi: 10.1016/j.isprsjprs.2017.11.009

ZHAO Hengshuang., QI Xiaojuan, SHEN Xiaoyong, SHI Jianping and JIA Jiaya. ICNet for real-time semantic segmentation on high-resolution images // Proceedings of the 15th European Conference on Computer Vision, Munich, Germany, 8-14 Sept 2018, pp. 405-420.

SONG W., ZHENG N., ZHENG R., ZHAO X.-B. and WANG, A. Digital image semantic segmentation algorithms: A survey. Journal of Information Hiding and Multimedia Signal Processing, 2019, 10(1):196-211.

JAMEEL S. M., HASHMANI M., ALHUSSAIN, A. H. and BUDIMAN A. A fully adaptive image classification approach for industrial revolution 4.0. In: SAEED F., GAZEM N., MOHAMMED F., BUSALIM A. (eds) Recent Trends in Data Science and Soft Computing. IRICT 2018. Advances in Intelligent Systems and Computing, vol 843. Springer, Cham.

ACHANTA, R., SHAJI A., SMITH K., LUCCHI A., FUA P. and SÜSSTRUNK S. SLIC superpixels, EPFL Technical Report 149300, June 2010.

LI Yin, SUN Jian, CHI-KEUNG Tang and HEUNG-YEUNG Shum. Lazy snapping. ACM Transactions on Graphics, 2004, 23(3): 303-308. doi: 10.1145/1015706.1015719

FULKERSON B., VEDALDI A. and SOATTO, S. Class segmentation and object localization with superpixel neighborhoods // Proceedings of 2009 IEEE 12th international Conference on computer Vision, Kyoto, Japan, 29 Sept.-2 Oct. 2009, pp. 670-677: doi:10.1109/ICCV.2009.5459175

WANG M., LIU X., GAO Y., MA X. and SOOMRO I. C. Superpixel segmentation: A benchmark, Signal Processing Image Communication, 2017, 56:28-39

ACHANTA R., SHAJI A., SMITH K., LUCCI A., FUA P. and SUSSTRUNK S. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Software Engineering, 2012, 34(11): 2274-2282, doi: 0.1109/TPAMI.2012.120.

SHEN J., DU Y., WANG W., and LI X. Lazy random walks for superpixel segmentation. IEEE Transactions on Software Engineering, 2014, 23(4): 1451-1462. doi: 10.1109/TIP.2014.2302892

XU L., WANG Z., and ZENG L. Saliency-based superpixels. Signal Image and Video Processing, 2014, 8(1): 181-190.

LIU Yong-Jin, YU Cheng-Chi, YU Min-Jing and HE Ying. Manifold SLIC: A fast method to compute content-sensitive superpixels // Proceedings of the I 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Nevada, USA, 27-30 June, 2016.pp.651-659. doi: 10.1109/CVPR.2016.77

ZHU Hongyuan, MENG Fanman, CAI Jianfei and LU Shijian. Beyond pixels: A comprehensive survey from bottom-up to semantic image segmentation and cosegmentation. Journal of Visual Communication and Image Representation , 2015, 34:12-27 doi: 10.1016/j.jvcir.2015.10.012.

SHI J., and MALIK J. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(8): 888-905. doi: 10.1109/34.868688

FELZENSZWALB P. F. and HUTTENLOCHER D.P. Efficient graph-based image segmentation, International Journal of Computer Vision, 2004, 59(2):167-181.

MOORE A. P., PRINCE S. J. D., WARRELL J., MOHAMMED U. and JONES G. Superpixel lattices// Proceedings of 2008 IEEE conference on computer vision and pattern recognition, Anchorage, Alaska, USA, 23-28 June 2008, pp. 1-8. doi: 10.1109/CVPR.2008.4587471.

ZHANG Y., HARTLEY R., MASHFORD J., and BURN S. Superpixels via pseudo-boolean optimization. // Proceedings of 2011 IEEE International Conference on Computer Vision, Barcelona, Spain, 6-13 November, 2011, pp. 1387-1394. doi:10.1109/ICCV.2011.6126393

LIU M.-Y., TUZEL O., RAMALINGAM S., and CHELLAPPA R. Entropy rate superpixel segmentation// Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, USA, 21-23 June, 2011, pp. 2097-2104.

VINCENT L. and SOILLE, P. Watersheds in digital spaces: an efficient algorithm based on immersion simulations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1991, 13(6):583-598.

LEVINSHTEIN A., STERE A., KUTULAKOS K., FLEET D., DICKINSON S. and SIDDIQI, K. TurboPixels: Fast superpixels using geometric flows. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2009, 31(12): 2290-2297. doi: 10.1109/TPAMI.2009.96

COMANICIU D. and PETER M. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2002, 25(5):603-619.

VEDALDI A. and SOATTO S. Quick shift and kernel methods for mode seeking // Proceedings of 10th European Conference on Computer Vision, Marseille, France, 12-18 October 2008, pp. 705-718.

FANG Zhuoqun, YU, Xiaosheng WU, Chengdong , CHEN Dongyue and JIA Tong. Superpixel Segmentation Using Weighted Coplanar Feature Clustering on RGBD Images. Applied Sciences, 2018, 8(6): 902. doi: 10.3390/app8060902.

CONG Jinyu, WEI Benzheng, YIN Yilong, XI Xiaoming, and ZHENG Yuanjie. Performance evaluation of simple linear iterative clustering algorithm on medical image processing. Bio-medical materials and engineering. 2014, 24(6): 3231-3238. 10.3233/BME-141145.

QIN Fachao, GUO Jiming and LANG Fengkai. Superpixel segmentation for polarimetric SAR imagery using local iterative clustering. Geoscience and Remote Sensing Letters, IEEE, 2014, 12(1): 13-17. doi: 10.1109/LGRS.2014.2322960.

WANG Peng, ZENG Gang, GAN Rui, WANG Jingdong and ZHA Hongbin. Structure-Sensitive Superpixels via Geodesic Distance. International Journal of Computer Vision, 2013, 103(10: 1-21. doi: 10.1007/s11263-012-0588-6

RAMU B. and THIRUPATHI D. A new methodology for diagnosis of appendicitis using sonographic images in image mining //Proceedings of the 2nd International Conference on Computational Science, Engineering and Information Technology, Coimbatore, India, 26- 28 October,2012, pp. 717-721.

REN X. and MALIK J. Learning a classification model for segmentation //Proceedings of the 9th IEEE International Conference on Computer Vision, St. Petersburg, Russia, 16-18 July, 2013, 1:10-17 doi:10.1109/ICCV.2003.1238308.

ARBELAEZ P., MAIRE M., FOWLKES C., and MALIK J. Contour detection and hierarchical image segmentation IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(5):898-916. doi: 10.1109/TPAMI.2010.161.

XU C. and CORSO J. J. Evaluation of super-voxel methods for early video processing //Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, Rhode Island, USA, 16-21 June, 2012 pp. 1202-1209, doi: 10.1109/CVPR.2012.6247802.

STUTZ D. Superpixel segmentation: An evaluation. In: GALL P. GEHLER B. and LEIBE B. (Eds.) Lecture Notes in Computer Science, 2015, 9358:555-562. doi:10.1007/978-3-319-24947-6_46

RAND W. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 1971, 66(336): 846-850. doi: 10.2307/2284239. 66


  • There are currently no refbacks.