RepFinder: Finding Approximately Repeated Scene Elements for Image Editing
Ming-Ming Cheng1 Fang-Lue Zhang1 Niloy J. Mitra2 Xiaolei Huang3 Shi-Min Hu1
1TNList, Tsinghua University, Beijing 2KAUST/IIT Delhi 3Lehigh University
Abstract
Repeated elements are ubiquitous and abundant in both manmade and natural scenes. Editing such images while preserving the repetitions and their relations is nontrivial due to overlap, missing parts, deformation between instances, illumination variation, etc. Manually enforcing such relations is laborious and error prone. We propose a novel framework where simple user input in the form of scribbles are used to guide detection and extraction of such repeated elements. Our detection process is based on a novel boundary band method, and robustly extracts the repetitions along with their mutual depth relations. We then use topological sorting to establish a partial depth ordering of overlapping repeated instances. Missing parts on occluded instances are completed using information from other instances. The extracted repeated instances can then be seamlessly edited and manipulated for a variety of high level tasks that are otherwise difficult to perform. We demonstrate the versatility of our framework on a large set of inputs of varying complexity, showing applications to image rearrangement, edit transfer, deformation propagation, and instance replacement.
Paper
RepFinder: Finding Approximately Repeated Scene Elements for Image Editing [Bib] [pdf] [PPT Zip File]
ACM Transactions on Graphics, 29, 4, 83:1-8, 2010.
Ming-Ming Cheng, Fang-Lue Zhang, Niloy J. Mitra, Xiaolei Huang, Shi-Min Hu
System Pipeline
Results
Video (download)
Supplementary Materials
1. Source code
Our system is a huge project with many components. To help readers implementing our system, we supply some pointers to important source code components of our system. These components are:
- Grab-cut (Including color GMM and graph cut)
- Active contours: We use opencv version
- Shape context (with thin plate splines)
- Matting
- Hierarchical segmentation
2. FAQs:
- Edge-based vs. point based matching: Our first attempt to solve this problem was based on employing a state-of-the-art point-based matching method to establish correspondences. However, in images with repeated elements, SIFT matching leads to much ambiguity due to the presence of a large number of points having similar texture. Even spectral correspondence methods fail to resolve the ambiguities. Thus, we moved to an edge-based approach that uses shape information, to resolve ambiguity, and to enforce local coherence of correspondence. We have found it to be simple, effective and robust. [An earlier experiment]
Links
- Repeated Structure Extraction papers in Princeton course: “Advanced Topics in Computer Science: Geometric Modeling and Analysis “. Related paper list is here.