Salient Object Detection: A Benchmark

19/03/201409/06/2018 MM Cheng

[:en]

Ali Borji, Ming-Ming Cheng, Huaizu Jiang, Jia Li

Notice: Welcome to contact Ming-Ming Cheng for adding new comparisons. Adding new results should supply either source code or executable.

Abstract

We extensively compare, qualitatively and quan- titatively, 42 state-of-the-art models (30 salient object detection, 10 fixation prediction, 1 objectness, and 1 baseline) over 6 challenging datasets for the purpose of benchmarking salient object detection and segmentation methods. From the results obtained so far, our evaluation shows a consistent rapid progress over the last few years in terms of both accuracy and running time. The top contenders in this benchmark significantly outperform the models identified as the best in the previous benchmark conducted just two years ago. We find that the models designed specifically for salient object detection generally work better than models in closely related areas, which in turn provides a precise definition and suggests an appropriate treatment of this problem that distinguishes it from other problems. In particular, we analyze the influences of center bias and scene complexity in model performance, which, along with the hard cases for state-of-the-art models, provide useful hints towards constructing more challenging large scale datasets and better saliency models. Finally, we propose probable solutions for tackling several open problems such as evaluation scores and dataset bias, which also suggest future research directions in the rapidly-growing field of salient object detection.

Papers

Salient Object Detection: A Benchmark, Ali Borji, Ming-Ming Cheng, Huaizu Jiang, Jia Li, IEEE TIP, 2015. [pdf] [Project page] [Bib]
Salient Object Detection: A Survey, Ali Borji, Ming-Ming Cheng, Huaizu Jiang, Jia Li, arXiv eprint, 2014. [pdf] [Project page] [Bib]

Code

C++ & Matlab: Salient Object Detection: A Benchmark, IEEE TIP, 2015.
More public available source code on this website.

Downloads

We provide the evaluation data (images, ground truth, saliency maps, etc.) downloads here to facilitate future research. We suggest to use BT software to download these zip files using their url lists, which is available here. Evaluation resutls in form of matlab and csv file for plots and talbes could be downloaded here. If you use any parts of our results, please cite the corresponding paper above. 您也可以通过百度网盘下载。

Performance: FMeasure of saliency maps and salient object segmentations

Model	T-M	T-A	T-S	J-M	J-A	J-S	D-M	D-A	D-S	S-M	S-A	S-S	M-M	M-A	M-S	E-M	E-A	E-S
MBD	.622	.594	.642	.472	.422	.470	.624	.592	.636	.799	.803	.759	.849	.830	.890	.739	.703	.785
ST	.631	.580	.648	.455	.394	.459	.631	.577	.635	.818	.805	.768	.868	.825	.896	.752	.690	.777
QCUT	.651	.625	.620	.509	.454	.480	.683	.647	.647	.810	.801	.672	.874	.843	.843	.779	.738	.747
HDCT	.602	.571	.636	.412	.378	.422	.609	.572	.643	.822	.802	.758	.837	.807	.877	.705	.669	.74
RBD	.596	.566	.618	.457	.403	.461	.63	.58	.647	.837	.825	.75	.856	.821	.884	.718	.68	.757
GR	.551	.509	.546	.418	.338	.378	.599	.54	.58	.798	.753	.639	.816	.77	.83	.664	.583	.677
MNP	.495	.523	.603	.367	.337	.405	.467	.486	.576	.621	.778	.765	.668	.724	.822	.568	.555	.709
UFO	.579	.557	.61	.432	.385	.433	.545	.541	.593	.742	.781	.729	.842	.806	.862	.701	.654	.739
MC	.61	.603	.6	.46	.42	.434	.627	.603	.615	.779	.803	.63	.847	.824	.855	.742	.704	.745
DSR	.611	.604	.597	.454	.421	.41	.626	.614	.593	.794	.821	.632	.835	.824	.833	.737	.717	.703
CHM	.612	.591	.643	.417	.368	.424	.604	.586	.637	.75	.75	.658	.825	.804	.857	.722	.684	.735
GC	.533	.517	.497	.384	.321	.342	.535	.528	.506	.729	.73	.616	.794	.777	.78	.641	.612	.593
LBI	.519	.534	.618	.371	.353	.416	.482	.504	.609	.692	.776	.764	.696	.714	.857	.586	.563	.738
PCA	.544	.558	.601	.432	.404	.368	.554	.554	.624	.754	.796	.701	.782	.782	.845	.646	.627	.72
DRFI	.67	.607	.674	.475	.419	.447	.665	.605	.669	.831	.839	.702	.881	.838	.905	.787	.733	.801
GMR	.597	.594	.579	.454	.409	.432	.61	.591	.591	.773	.789	.643	.847	.825	.839	.74	.712	.736
HS	.585	.549	.602	.442	.358	.428	.616	.565	.616	.811	.776	.713	.845	.8	.87	.731	.659	.769
LMLC	.54	.519	.588	.375	.302	.397	.521	.493	.551	.653	.712	.674	.801	.772	.86	.659	.6	.735
SF	.5	.495	.342	.373	.319	.219	.519	.512	.377	.764	.794	.509	.779	.759	.573	.619	.576	.378
FES	.547	.575	.426	.424	.411	.333	.52	.555	.38	.617	.785	.174	.717	.753	.534	.645	.655	.467
CB	.581	.556	.615	.444	.375	.435	.542	.534	.593	.73	.704	.657	.815	.775	.857	.717	.656	.761
SVO	.554	.441	.609	.414	.279	.419	.557	.407	.609	.744	.667	.746	.789	.585	.863	.639	.357	.737
SWD	.528	.56	.649	.434	.386	.454	.478	.506	.613	.548	.714	.737	.689	.705	.871	.624	.549	.781
HC	.386	.401	.436	.286	.257	.28	.382	.38	.435	.736	.759	.646	.677	.663	.74	.46	.441	.499
RC	.61	.586	.639	.431	.37	.425	.599	.578	.621	.774	.807	.649	.844	.82	.875	.741	.701	.776
SEG	.5	.425	.58	.376	.268	.393	.516	.45	.562	.704	.64	.669	.697	.585	.812	.568	.408	.715
MSS	.478	.49	.2	.341	.324	.089	.476	.49	.193	.743	.783	.298	.696	.711	.362	.53	.536	.203
CA	.458	.494	.557	.353	.33	.394	.435	.458	.532	.591	.737	.565	.621	.679	.748	.515	.494	.625
FT	.386	.4	.238	.278	.25	.132	.381	.388	.259	.715	.734	.436	.635	.628	.472	.434	.431	.257
AC	.41	.431	.068	.227	.199	.049	.354	.383	.04	.684	.729	.14	.52	.566	.014	.411	.41	.038
LC	.386	.408	.289	.264	.246	.156	.327	.353	.243	.683	.752	.486	.569	.589	.432	.39	.396	.219
OBJ	.498	.482	.593	.368	.282	.413	.481	.445	.578	.685	.723	.731	.718	.681	.84	.574	.456	.698
BMS	.568	.578	.594	.434	.404	.416	.573	.576	.58	.713	.76	.627	.805	.798	.822	.683	.659	.69
COV	.51	.587	.398	.429	.427	.315	.486	.579	.373	.518	.724	.212	.667	.755	.394	.641	.677	.413
SS	.415	.482	.523	.344	.321	.397	.396	.443	.502	.533	.696	.641	.572	.642	.675	.467	.441	.574
SIM	.372	.429	.568	.295	.292	.384	.358	.402	.539	.498	.685	.725	.498	.585	.794	.433	.391	.672
SeR	.374	.419	.536	.316	.285	.388	.385	.411	.532	.521	.714	.702	.542	.607	.755	.419	.391	.596
SUN	.387	.432	.486	.303	.291	.285	.321	.36	.445	.504	.661	.613	.505	.596	.67	.388	.376	.478
SR	.374	.457	.002	.279	.27	.001	.298	.363	0	.504	.7	.002	.473	.569	.001	.381	.385	.001
GB	.526	.571	.65	.419	.396	.455	.507	.548	.638	.571	.746	.695	.688	.737	.837	.624	.613	.765
AIM	.427	.461	.559	.317	.26	.36	.361	.377	.495	.541	.718	.693	.555	.575	.75	.449	.357	.571
IT	.373	.437	.005	.297	.283	0	.378	.449	.005	.579	.697	.008	.471	.586	.158	.407	.414	.003
AVG	.458	.569	.62	.392	.367	.411	.406	.514	.534	.388	.524	.64	.58	.692	.779	.597	.627	.756

Max FMeasure of average precision recall curve, average FMeasure for adaptive thresholding results, average FMeasure for SalCut. The subtitle of each column is in the [Dataset]-[Evaluation Metric] format, where [Dataset] is represented by the initial letter for the 6 benchmarks {THUR15K, JuddDB, DUT-OMRON, SED2, MSRA10K, ECSSD}. Click the title of the column to rerank the table according to that metric.

Performance: AUC & MAE

Method	T-AUC	T-MAE	J-AUC	J-MAE	D-AUC	D-MAE	S-AUC	S-MAE	M-AUC	M-MAE	E-AUC	E-MAE
MBD	0.915	0.162	0.838	0.225	0.903	0.168	0.922	0.137	0.964	0.107	0.917	0.172
ST	0.911	0.179	0.806	0.240	0.895	0.182	0.922	0.145	0.961	0.122	0.914	0.193
QCUT	0.907	0.128	0.831	0.178	0.897	0.119	0.860	0.148	0.956	0.118	0.909	0.171
HDCT	0.878	0.177	0.771	0.209	0.869	0.164	0.898	0.162	0.941	0.143	0.866	0.199
RBD	0.887	0.15	0.826	0.212	0.894	0.144	0.899	0.13	0.955	0.108	0.894	0.173
GR	0.829	0.256	0.747	0.311	0.846	0.259	0.854	0.189	0.925	0.198	0.831	0.285
MNP	0.854	0.255	0.768	0.286	0.835	0.272	0.888	0.215	0.895	0.229	0.82	0.307
UFO	0.853	0.165	0.775	0.216	0.839	0.173	0.845	0.18	0.938	0.15	0.875	0.207
MC	0.895	0.184	0.823	0.231	0.887	0.186	0.877	0.182	0.951	0.145	0.91	0.204
DSR	0.902	0.142	0.826	0.196	0.899	0.139	0.915	0.14	0.959	0.121	0.914	0.173
CHM	0.91	0.153	0.797	0.226	0.89	0.152	0.831	0.168	0.952	0.142	0.903	0.195
GC	0.803	0.192	0.702	0.258	0.796	0.197	0.846	0.185	0.912	0.139	0.805	0.214
LBI	0.876	0.239	0.792	0.273	0.854	0.249	0.896	0.207	0.91	0.224	0.842	0.28
PCA	0.885	0.198	0.804	0.181	0.887	0.206	0.911	0.2	0.941	0.185	0.876	0.248
DRFI	0.938	0.15	0.851	0.213	0.933	0.155	0.944	0.13	0.978	0.118	0.944	0.166
GMR	0.856	0.181	0.781	0.243	0.853	0.189	0.862	0.163	0.944	0.126	0.889	0.189
HS	0.853	0.218	0.775	0.282	0.86	0.227	0.858	0.157	0.933	0.149	0.883	0.228
LMLC	0.853	0.246	0.724	0.303	0.817	0.277	0.826	0.269	0.936	0.163	0.849	0.26
SF	0.799	0.184	0.711	0.218	0.803	0.183	0.871	0.18	0.905	0.175	0.817	0.23
FES	0.867	0.155	0.805	0.184	0.848	0.156	0.838	0.196	0.898	0.185	0.86	0.215
CB	0.87	0.227	0.76	0.287	0.831	0.257	0.839	0.195	0.927	0.178	0.875	0.241
SVO	0.865	0.382	0.784	0.422	0.866	0.409	0.875	0.348	0.93	0.331	0.857	0.404
SWD	0.873	0.288	0.812	0.292	0.843	0.31	0.845	0.296	0.901	0.267	0.857	0.318
HC	0.735	0.291	0.626	0.348	0.733	0.31	0.88	0.193	0.867	0.215	0.704	0.331
RC	0.896	0.168	0.775	0.27	0.859	0.189	0.852	0.148	0.936	0.137	0.892	0.187
SEG	0.818	0.336	0.747	0.354	0.825	0.337	0.796	0.312	0.882	0.298	0.808	0.342
MSS	0.813	0.178	0.726	0.204	0.817	0.177	0.871	0.192	0.875	0.203	0.779	0.245
CA	0.83	0.248	0.774	0.282	0.815	0.254	0.853	0.229	0.872	0.237	0.784	0.31
FT	0.684	0.241	0.593	0.267	0.682	0.25	0.82	0.206	0.79	0.235	0.661	0.291
AC	0.74	0.186	0.548	0.239	0.721	0.19	0.831	0.206	0.756	0.227	0.668	0.265
LC	0.696	0.229	0.586	0.277	0.654	0.246	0.827	0.204	0.771	0.233	0.627	0.296
OBJ	0.839	0.306	0.75	0.359	0.822	0.323	0.87	0.269	0.907	0.262	0.818	0.337
BMS	0.879	0.181	0.788	0.233	0.856	0.175	0.852	0.184	0.929	0.151	0.865	0.216
COV	0.883	0.155	0.826	0.182	0.864	0.156	0.833	0.21	0.904	0.197	0.879	0.217
SS	0.792	0.267	0.754	0.301	0.784	0.277	0.826	0.266	0.823	0.266	0.725	0.344
SIM	0.797	0.414	0.727	0.412	0.783	0.429	0.833	0.384	0.808	0.388	0.734	0.433
SeR	0.778	0.345	0.746	0.379	0.786	0.352	0.835	0.29	0.813	0.31	0.695	0.404
SUN	0.746	0.31	0.674	0.319	0.708	0.349	0.789	0.307	0.778	0.306	0.623	0.396
SR	0.741	0.175	0.676	0.2	0.688	0.181	0.769	0.22	0.736	0.232	0.633	0.266
GB	0.882	0.229	0.815	0.261	0.857	0.24	0.839	0.242	0.902	0.222	0.865	0.263
AIM	0.814	0.298	0.719	0.331	0.768	0.322	0.846	0.262	0.833	0.286	0.73	0.339
IT	0.623	0.199	0.586	0.2	0.636	0.198	0.682	0.245	0.64	0.213	0.577	0.273
AVG	0.849	0.248	0.797	0.343	0.814	0.288	0.736	0.405	0.857	0.26	0.863	0.276

Comparison of AUC scores (larger better) and MAE scores (smaller better). Similar to the table above, the subtitle of each column is in the [Dataset]-[Evaluation Metric] format, where [Dataset] is represented by the initial letter for the 6 benchmarks {THUR15K, JuddDB, DUT-OMRON, SED2, MSRA10K, ECSSD}. Click the title of the column to rerank the table according to that metric.

Salient object detection datasets

Abbr.	Images	References
MSRA10K	10000	Learning to Detect A Salient Object , IEEE CVPR 2007, Liu et al. Frequency-tuned Salient Region Detection, IEEE CVPR 2009, Achanta et al. Global Contrast based Salient Region Detection, IEEE TPAMI 2015, Cheng et al.
ECSSD	1000	Hierarchical Saliency Detection, IEEE CVPR 2013, Yan et al.
THUR15K	15000	SalientShape: Group Saliency in Image Collections, The Visual Computer 2013, Cheng et al.
JuddDB	900	What is a salient object? A dataset and a baseline model for salient object detection, arXiv, ePrints
DUTOMRON	5000	Saliency Detection Via Graph-Based Manifold Ranking, IEEE CVPR 2013, Yang et al.
SED 2	100	Image segmentation by probabilistic bottom-up aggregation and cue integration, IEEE CVPR 2007, Alpert et al

Information about different methods

Abbr.	Date	Run time (s)	Code	Booktitle	Paper title
MB+	2015		C	IEEE ICCV	Minimum Barrier Salient Object Detection at 80 FPS
ST	2014		M	IEEE TIP	Saliency Tree: A Novel Saliency Detection Framework
QCUT	2014		M+C	ICPR	Automatic Object Segmentation by Quantum Cuts
HDCT	2014	4.12	M	IEEE CVPR	Salient Region Detection via High-dimensional Color Transform
RBD	2014	0.269	M	IEEE CVPR	Saliency Optimization from Robust Background Detection
GR	2013	1.35	M+C	Sig. Proc. Lett.	Graph-Regularized Saliency Detection With Convex-Hull-Based Center Prior
MNP	2013	21.0	M+C	The Vis. Comp.	Saliency for Image Manipulation
UFO	2013	20.3	M+C	IEEE ICCV	Salient Region Detection by UFO: Uniqueness, Focusness and Objectness
MC	2013	0.195	M+C	IEEE ICCV	Saliency Detection via Absorbing Markov Chain
DSR	2013	10.2	M+C	IEEE ICCV	Saliency Detection via Dense and Sparse Reconstruction
CHM	2013	15.4	M+C	IEEE ICCV	Contextual Hypergraph Modeling for Salient Object Detection
GC	2013	0.037	C	IEEE ICCV	Efficient Salient Region Detection with Soft Image Abstraction
LBI	2013	251	M+C	IEEE CVPR	Looking Beyond the Image: Unsupervised Learning for Object Saliency and Detection
PCA	2013	4.34	M+C	IEEE CVPR	What Makes a Patch Distinct?
DRFI	2013	0.697	M+C	IEEE CVPR	Salient Object Detection: A Discriminative Regional Feature Integration Approach
GMR	2013	0.149	C	IEEE CVPR	Saliency Detection via Graph-based Manifold Ranking
HS	2013	0.528	C	IEEE CVPR	Hierarchical Saliency Detection
LMLC	2013	140	M+C	IEEE TIP	Bayesian Saliency via Low and Mid Level Cues
SF	2012	0.202	C	IEEE CVPR	Saliency Filters: Contrast Based Filtering for Salient Region Detection
FES	2011	0.096	M+C	Image Ana.	Fast and Eﬃcient Saliency Detection Using Sparse Sampling and Kernel Density Estimation
CB	2011	2.24	M+C	BMVC	Automatic salient object segmentation based on context and shape prior
SVO	2011	56.5	M+C	IEEE ICCV	Fusing Generic Objectness and Visual Saliency for Salient Object Detection
SWD	2011	0.190	M+C	IEEE CVPR	Visual saliency detection by spatially weighted dissimilarity
HC	2011	0.017	C	IEEE CVPR	Global Contrast based Salient Region Detection
RC	2015	0.136	C	IEEE TPAMI	Global Contrast based Salient Region Detection
SEG	2010	10.9	M	ECCV	Segmenting salient objects from images and videos
MSS	2010	0.076	C	IEEE ICIP	Saliency detection using maximum symmetric surround
CA	2010	49.0	M+C	IEEE CVPR	Context-aware saliency detection
FT	2009	0.072	C	IEEE CVPR	Frequency-tuned salient region detection
AC	2008	0.129	M	ICVS	Salient region detection and segmentation
LC	2006	0.009	C	ACM Multi.	Visual attention detection in video sequences using spatiotemporal cues
OBJ	2010	3.01	M+C	IEEE CVPR	What is an object?
BMS	2013	0.575	M+C	IEEE ICCV	Saliency Detection: A Boolean Map Approach
COV	2013	25.4	M	J. of Vis.	Visual saliency estimation by nonlinearly integrating features using region covariances
SS	2012	0.053	M	IEEE PAMI	Image Signature: Highlighting sparse salient regions
SIM	2011	1.11	M	IEEE CVPR	Saliency estimation using a non-parametric low-level vision model
SeR	2009	1.31	M	J. of Vis.	Static and space-time visual saliency detection by self-resemblance
SUN	2008	3.56	M	J. of Vis.	SUN: A bayesian framework for saliency using natural statistics
SR	2007	0.040	M	IEEE CVPR	Saliency detection: A spectral residual approach
GB	2006	0.735	M+C	NIPS	Graph-based visual saliency
AIM	2009	8.66	M	J. of Vis.	Saliency, attention, and visual search: An information theoretic approach
IT	1998	0.302	M	IEEE PAMI	A model of saliency-based visual attention for rapid scene analysis

Detailed information of each method. Regarding source code type: `C' means 'C/C++', 'M' means 'Matlab', 'M+C' means a mixture of Matlab and C/C++.

News

2015/4/24: evaluation results of QCUT has been added.
2015/10/24: evaluation results of MBD has been added.

[:zh]