TU Berlin

Scene Completion using millions of images

 CG
seperator
  • Home -
  • Motivation -
  • Algorithm -
  • Results -
  • Contact
    •
    seperator
    Motivation
    The basic idea behind Scene Completion is that the user marks an area in an image that he wants to have replaced and sends this query to a software that automatically replaces the selected area with appropriate content. Think for example of this image where the user wants to replace the car in a holiday photograpgh:
    There exist several algorithms for the purpose of replacing or restoring image areas, but none of these do use external information but try to extend image information into the selected area instead. This leads to undesired results:
    Image Inpainting:
    Bertalmio, Sapiro, Caselles, and Ballester; SIGGRAPH 2000
    http://portal.acm.org/citation.cfm?id=344779.344972
    Texture synthesis by non-parametric sampling:
    Efros, Leung; ICCV 1999
    http://www2.computer.org/portal/web/csdl/doi/10.1109/ICCV.1999.790383
    Region filling and object removal by exemplar-based inpainting:
    Criminisi, Perez, Toyama; IEEE Transactions on Image Processing 2004
    http://ieeexplore.ieee.org/Xplore/login.jsp?url=/iel5/83/29293/01323101.pdf?arnumber=1323101
    Microsoft Digital Image Pro Smart Erase:
    What we want to do instead is to recognize the scene itself and replace the selected area with content that matches it. An approach to do this has been done by Jian Sun, Lu Yuan, Jiaya Jia and Heung-Yeung Shum.
    Image Completion with Structure Propagation:
    Sun, Yuan, Jia, Shum; SIGGRAPH 2005
    http://portal.acm.org/citation.cfm?id=1073274
    The result is much more closer to what we would call a desired result, but still it does not add any new information to the image. The Scene Completion algorithm described by Hays and Efros we implemented here, uses a huge image database to search for similiar images and replace the selected area with a matching area of a similiar scene. The result could look like something like this:
    Scene Completion:
    Efros, Hays; SIGGRAPH 2007
    http://graphics.cs.cmu.edu/projects/scene-completion
    The results of our implementation of the Scene Completion algorithm are presented on this page.

    Reference: Scene Completion presentation by Efros and Hays
    seperator