Integral Object Mining via Online Attention Accumulation

Peng-Tao Jiang, Qibin Hou, Yang Cao, Ming-Ming Cheng, Yunchao Wei, Hongkai Xiong

Abstract

Object attention maps generated by image classifiers are usually used as priors for weakly-supervised segmentation approaches. However, normal image classifiers produce attention only at the most discriminative object parts, which limits the performance of weakly-supervised segmentation task. Therefore, how to effectively identify entire object regions in a weakly-supervised manner has always been a challenging and meaningful problem. We observe that the attention maps produced by a classification network continuously focus on different object parts during training. In order to accumulate the discovered different object parts, we propose an online attention accumulation (OAA) strategy which maintains a cumulative attention map for each target category in each training image so that the integral object regions can be gradually promoted as the training goes. These cumulative attention maps, in turn, serve as the pixel-level supervision, which can further assist the network in discovering more integral object regions. Our method (OAA) can be plugged into any classification network and progressively accumulate the discriminative regions into integral objects as the training process goes. Despite its simplicity, when applying the resulting attention maps to the weakly-supervised semantic segmentation task, our approach improves the existing state-of-the-art methods on the PASCAL VOC 2012 segmentation benchmark, achieving a mIoU score of 66.4% on the test set.

Attention Shift Video (8MB Video)

Attention Accumulation Video (8MB Video)

Papers

• Integral Object Mining via Online Attention Accumulation. Peng-Tao Jiang, Qibin Hou, Yang Cao, Ming-Ming Cheng, Yunchao Wei, Hongkai Xiong. ICCV, 2019. [pdf][poster][code][中文翻译]
• Online Attention Accumulation for Weakly Supervised Semantic Segmentation, Peng-Tao Jiang#, Ling-Hao Han#, Qibin Hou, Ming-Ming Cheng*, Yunchao Wei, IEEE TPAMI, 2021. [pdf | bib | project]

If you think this work is helpful for you, please cite:

@inproceedings{jiang2019integral,   
      title={Integral Object Mining via Online Attention Accumulation},   
      author={Jiang, Peng-Tao and Hou, Qibin and Cao, Yang and Cheng, Ming-Ming and Wei, Yunchao and Xiong, Hong-Kai},   
      booktitle={Proceedings of the IEEE International Conference on Computer Vision},   
      pages={2070--2079},   
      year={2019} 
}

Downloads

1. The attention maps generated by OAA and OAA+ can be download.
2. The saliency maps used to generating proxy segmentation labels can be download here.
3. The code for generating proxy segmentation labels can be download here.
4. CRF parameters: bi_w = 3, bi_xy_std = 67, bi_rgb_std = 4, pos_w = 1, pos_xy_std = 3.

Comparisons with state of the art methods

Methods(VGG)	OAA(Ours)	SeeNet [1]	Dilated [2]	Affinity [3]	MCOF [4]	DSRG [5]	GAIN [6]	TPL [7]	DCSP [8]	WebS-i2 [9]	Hong et al. [10]	AE-PSL [11]	Oh et al. [12]	Roy et al. [13]	STC [14]	AugFeed [15]	SEC [16]	DCSM [17]	MIL [18]	EM-Adapt [19]	CCNN [20]
Supervision	10k	10k	10k	10k	10k	10k	10k	10k	10k	19k	970k	10k	10k	10k	50k	10k	10k	10k	700k	10k	10k
mIOU(val)	63.1	61.1	60.4	58.4	56.2	59.0	55.3	53.1	58.6	53.4	58.1	55.0	55.7	52.8	49.8	54.3	50.7	44.1	42.0	38.2	35.3
mIOU(test)	62.8	60.7	60.8	60.5	57.6	60.4	56.8	53.8	59.2	55.3	58.7	55.7	56.7	53.7	51.2	55.5	51.7	45.1	-	39.6	-

Methods(ResNet)	OAA(ours)	SeeNet[1]	AffinityNet[2]	MCOF[3]	DSRG[4]	DCSP[5]
Supervision	10k	10k	10k	10k	10k	10k
mIOU(val)	65.2	63.1	61.7	60.3	61.4	60.8
nIOU(test)	66.4	62.8	63.7	61.2	63.2	61.9

Most related projects on this website:

• Self-Erasing Network for Integral Object Attention. Qibin Hou, Peng-Tao Jiang, Yunchao Wei, Ming-Ming Cheng. NeurIPS, 2018. [pdf][bib][code]
• Adversarial Complementary Learning for Weakly Supervised Object Localization. Xiaolin Zhang, Yunchao Wei, Jiashi Feng, Yi Yang, Thomas Huang. CVPR, 2018. [pdf][code]
• WebSeg: Learning Semantic Segmentation from Web Searches, Qibin Hou, Ming-Ming Cheng, Jiangjiang Liu, Philip H.S. Torr, arXiv eprint, 2018.
• Bottom-Up Top-Down Cues for Weakly Supervised Semantic Segmentation, Qinbin Hou, Puneet Kumar Dokania, Daniela Massiceti, Yunchao Wei, Ming-Ming Cheng, Philip Torr, EMMCVPR, 2017. [pdf][bib][simple Dataset] [official version]
• Deeply supervised salient object detection with short connections, Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, Philip Torr, IEEE TPAMI, 41(4):815-828, 2019.  [pdf] [project page] [bib] [source code] [official version]