Sketch2Photo: Internet Image Montage
Tao Chen1 Ming-Ming Cheng 1 Ping Tan2 Ariel Shamir3 Shi-Min Hu1
1TNList, Department of Computer Science and Technology, Tsinghua University
2National University of Singapore 3The Interdisciplinary Center
Abstract
We present a system that composes a realistic picture from a simple freehand sketch annotated with text labels. The composed picture is generated by seamlessly stitching several photographs in agreement with the sketch and text labels; these are found by searching the Internet. Although online image search generates many inappropriate results, our system is able to automatically select suitable photographs to generate a high quality composition, using a filtering scheme to exclude undesirable images. We also provide a novel image blending algorithm to allow seamless image composition. Each blending result is given a numeric score, allowing us to find an optimal combination of discovered images. Experimental results show the method is very successful; we also evaluate our system using the results from two user studies.
Paper
Sketch2Photo: Internet Image Montage [Bib]
ACM Transactions on Graphics (SIGGRAPH Asia), 28(5), 124:1-10, 2009
Tao Chen, Ming-Ming Cheng, Ping Tan, Ariel Shamir, Shi-Min Hu
Note
This work was selected as “one of the 10 most innovative and promising worldwide initiatives of 2009” by the Netexplorateur jury, under the aegis of French government. The related introduction video in Vimeo has attracted more than 1,000,000 clicks during the half year after its sharing. It also received the best research achievement award in China National Computer Conference 2010. The online commercial software also draws great user attention. I’m in charge of the image retrieval algorithms (first half) of this work. This work have been reported by several famous international media, including UK Telegraph, Spiegel, etc.
System Pipeline
Retrieval Results
Composition Results
Video (download)
Supplementary Materials
1. Intermediate results
General supplementary materials, including intermediate results and comparisons. [pdf]
2. User studies
High resolution compositions and detailed statistics of the user studies. [pdf]
Sktech2Photo Team
Tao Chen, Kun Xu, Fang-Lve Zhang, Meng Ding and Ming-Ming Cheng
Update: A web-based Sketch2Photo application: click here (Chinese), collaborated with Tencent.
Acknowledgments
This work was supported by the National Basic Research Project of China (Project Number 2006CB303106), the National High Technology Research and Development Program of China (Project Number 2009AA01Z330) and FDCT, Macau (Project Number 008/2008/A1). Tan Ping is supported by Singapore FRC Grant R-263-000-477-112.
Closely related works
- Photo Clip Art, ACM SIGGRAPH 2007, Lalonde et a.
- Semantic Photo Synthesis, Computer Graphics Forum 2006, Johnson et al.
Note
Original Name: PhotoSketch. We change the name since it’s conflict with existing software: http://www.brainstormllc.com/.
What is it – V7BOMDEFEX
Please tell me-where is it? Or what is it V7BOMDEFEX ?
’tis a shame this software is not available anymore. and rolling my own implementation would take too long… but there could’ve been so much lulz with it! 🙂
Sorry for the commercial system is no available at the moment. Regarding implementation, you might get salient object shapes for internet images from: https://mmcheng.net/salobj/ Alpha matting from: http://www.alphamatting.com/code.php Shape matching from http://www.eecs.berkeley.edu/Research/Projects/CS/vision/shape/sc_digits.html
These components would enable the major function of Sketch2Photo. Proper speed up and optimization are needed for better results.