Research

DenseCut: Densely Connected CRFs for Realtime GrabCut

Ming-Ming Cheng    Victor Adrian Prisacariu   Shuai Zheng    Philip H. S. Torr   Carsten Rother

Abstract

Figure-ground segmentation from bounding box input, provided either automatically or manually, has been extremely popular in the last decade and influenced various applications. A lot of research has focused on high-quality segmentation, using complex formulations which often lead to slow techniques, and often hamper practical usage. In this paper, we demonstrate a very fast segmentation technique that still achieves very high-quality results. We propose to replace the time consuming iterative refinement of global color models in traditional GrabCut formulation by a densely connected CRF. To motivate this decision, we show that a dense CRF implicitly models unnormalized global color models for foreground and background. Such a relationship provides insightful analysis to bridge between dense CRF and GrabCut functional. We extensively evaluate our algorithm using two famous benchmarks. Our experimental results demonstrated that the proposed algorithm achieves an order of magnitude (10) speed-up with respect to the closest competitor, and at the same time achieves considerably higher accuracy.

Paper

1. DenseCut: Densely Connected CRFs for Realtime GrabCut. Ming-Ming Cheng, Victor Adrian Prisacariu, Shuai Zheng, Philip H. S. Torr, Carsten Rother. Computer Graphics Forum, 2015. [Project page] [pdf] [slides] [bib] [c++]

Datasets

  1. The GrabCut dataset used in this paper is no longer maintained by its original research team (GrabCut – Microsoft Research). We provide an alternative download link.
  2. Here is another binary segmentation dataset: MSRA10K.
(Visited 20,293 times, 2 visits today)
Subscribe
Notify of
guest

31 Comments
Inline Feedbacks
View all comments
LiuDian

老师 您好,我最近也是在研究传统图像分割的方法,我下载了这篇文章的源码。请问下面这个头文件您能提供吗?代码里面的下载地址无法使用,这是我的邮箱 1009250352@qq.com,谢谢!

#include <CmCode/CmLib/CmLib.h> // https://bitbucket.org/cmmthu/cmcode/wiki/Home

Jenny

老师您好,这个项目的input数据可否提供?或者能否告诉我imgDir, salDir, iluDir中分别是什么图片也行,感激不尽!

yao

您的GetMaskRange函数是做什么用的呢?

Baobei Xu

Hello, Mr. Cheng, DenseCut is based on bounding box. I read the your paper DenseCut, I consider it may be the same as GrabCut that allows users to assign some correct FG pixels and the FG is absolutely in bounding box. But I read your code roughly, I just found the bounding box input, but not found the directly choosing some correct FG pixels to be FG. So how do I edit the code that satisfy my demand?

Thank you very much!

Baobei Xu

程老师, 您好. 我阅读了您的DenseCut的程序代码, 我说下我的理解, 如果我的理解有错误, 能否帮忙指正一下. 程序的输入是: rgb图片和bounding box, 接着程序将bounding box区域作为ROI区域, 后面的操作不作用在ROI区域外, 然后将bounding box往内缩一定的像素做为rect(通过blur来实现), 接着对ROI区域进行边缘检测, 将边缘进行膨胀处理, 再用连通域填充处理, 将rect内边缘首尾相接的内部区域作为可能的前景区域, 将其他区域作为可能的背景区域, 进行GMM训练, 再用crf作为精调处理; 第二步, 将第一步的结果进行10次膨胀之后, 再将此作为之前的rect区域, 进行此区域内的连通域填充处理(有些bounding box较小的例子, 经过此处理之后, 就充满整个rect区域了, 这个情况不会效果较差吗?), 再进行GMM训练, 最后用crf精调. 我不知道我的理解是否有错, 我有几点想问的:
1. 为何bounding box区域外的内容就完全抛弃了, 倘若前景部分基本上占满整个bounding box, 那么无法对背景部分进行好的GMM训练, 效果是否就比较差? 将bounding box区域外的内容完全抛弃是否是提高速度用的?
2. 使用边缘检测进行, 对在bouding box内背景的连通区域是否很敏感?
3. 根据我的理解, 如果要改成grabCut类似的可以与用户交互的, 那么是否需要更改GMM部分和CRF部分?

谢谢!~

Baobei Xu

程老师, 您好。我看你的代码里,用bounding box外一定范围内作为固定背景,是否是这段代码:
blur(_gt1u, _gt1u, Size(3,3));
Mat _res1u = Mat::zeros(_imMat3u.size(), CV_8U);
Rect wkRect = CmCv::GetMaskRange(_gt1u, 30, 200);
如果对于图像比较大,前景在所占的像素较多,使用3×3的blur是否足够?假设我的背景部分有棋盘格,而棋盘格占背景很大部分,且我的bounding box不是特别准确,就是bounding box内部还有较多的背景区域,只使用3×3的blur区域作为背景区域是否能够将bounding box内的棋盘格给判断成背景?

Qing Han

I have the same problem,the opencv downloaded don’t contain ‘opencv_core330d.lib’,but it keeps saying ‘can’t open the file ‘whether you have resolved or not?please help me.

lai

你好请问您已经解决了这个问题吗?

yao

需要下OPENCV源码自己编译,否则你只有一个world库,而不是拆开来的各个分库

Allen Zhang

Dear Prof.Cheng

Would you write a .md file to tell us how to compile this cpp code? I’m using vs2010 and opencv 3.4.1 and it keeps saying ‘ cannot open file ‘opencv_core341d.lib’. It would be appreciated.

Allen Zhang

Did you use Visual Studio or cmake?

Qing Han

I have the same problem,the opencv downloaded don’t contain ‘opencv_core330d.lib’,but it keeps saying ‘can’t open the file ‘whether you have resolved or not?please help me.

Allen Zhang

Naah I wrote code myself instead

Jenny

I have the same problem. I installed opencv3.4.0, and configured it correctly, but the problem haven’t been solved…Could you share your code to me? It would be appreciated!!!

Jenny

Oh, I find the solution! Thank you, Yao!
When configuring:don’t selected the “Build opencv world”, the problem will be solved~

梁大双

greate work!
but I can not find the code refer to this paper, can you give me the place to download it?

Jackie Yung

Mr Cheng,I am a graduate student who is researching on graph cut.When I have read your paper-“DenseCut”,I found that your processing time is very fast.So,I have run your code,but I found some questions about the code.
First,if you don’t have the ground truth image for inputing,how to modify the code-“GrabCutMF::Demo”.I have modified the code,but it seems doesn’t work.The code is here:
void GrabCutMF::Demo(CStr &wkDir, float w1, float w2, float w3, float alpha, float beta, float gama, float mu)
{
CStr imgDir = wkDir + “Imgs/”, salDir = wkDir + “Sal4N/”, iluDir = wkDir + “Ilu4N/”;
vecS namesNE;
int imgNum = CmFile::GetNamesNE(imgDir + “*.jpg”, namesNE);
CmFile::MkDir(salDir);
CmFile::MkDir(iluDir);
printf(“w1 = %g, w2 = %g, w3 = %g, alpha = %g, beta = %g, gama = %g, mu = %g\n”, w1, w2, w3, alpha, beta, gama, mu);

CmTimer tm(“Time”), tmIni(“TimeIni”), tmRef(“TimeRef”);
double maxWeight = 2; // 2: 0.958119, 1: 0.953818,
tm.Start();
#pragma omp parallel for
for (int i = 0; i < imgNum; i++){
cout << "Image Number:" << i << endl;
Mat _imMat3u = imread(imgDir + namesNE[i] + ".jpg");//原图
Mat imMat3f, imMat3u, gt1u;
cout << "height:" << _imMat3u.rows << "\t" << "width:" << _imMat3u.cols << endl;
Mat _res1u = Mat::zeros(_imMat3u.size(), CV_8U);
Rect wkRect = Rect(20, 20, 460, 270);
_imMat3u(wkRect).copyTo(imMat3u);
imMat3u.convertTo(imMat3f, CV_32FC3, 1/255.0);

//* The Mean field based GrabCut
tmIni.Start();
GrabCutMF cutMF(imMat3f, imMat3u, salDir + namesNE[i], w1, w2, w3, alpha, beta, gama, mu);
imwrite(salDir + namesNE[i] + ".jpg", imMat3u);
tmIni.Stop();
cout << "cutMF:" << tmIni.TimeInSeconds() << endl;
tmRef.Start();
cutMF.refine();
tmRef.Stop();
cout << "refine:" << tmRef.TimeInSeconds() << endl;

Mat res1u = cutMF.drawResult(), invRes1u;
}
tm.Stop();
double avgTime = tm.TimeInSeconds()/imgNum;
printf("Speed: %gs, %gfps\t\t\n", avgTime, 1/avgTime);
char* pDes[] = { "GCMF1", "GCMF"}; //, "CudaG4", "Onecut", "GC", "CudaH",
vecS des = charPointers2StrVec (pDes);
CStr rootDir = CmFile::GetFatherFolder(wkDir), dbName = CmFile::GetNameNE(wkDir.substr(0, wkDir.size() – 1));
CmEvaluation::EvalueMask(imgDir + "*.png", salDir, des, wkDir.substr(0, wkDir.size() – 1) + "Res.m", 0.3, false, "", dbName);
}

Second,if you need a ground truth image for every image to process just like your code writed,how to process an image without the ground truth image.
I am looking forward to your reply.Thank you.

DrBalthar

Great work!

However, the code reference seems to refer some hard coded demo image set which looks like it includes some ground truth data at the same time? It is a bit hard to decipher as it seems to be used for the trimap or mask region at the same time? Is that data available anywhere by chance?

terry chan

Can you give me a password?