Inpaint Anything: Segment Anything Meets Image Inpainting
By Tao Yu, Runsen Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, Zhibo Chen. arXiv.org, 2023
320 citations recorded by the source corpus.
Modern image inpainting systems, despite the significant progress, often struggle with mask selection and holes filling. Based on Segment-Anything Model (SAM), we make the first attempt to the mask-free image inpainting and propose a new paradigm of ``clicking and filling'', which is named as Inpaint Anything (IA). The core idea behind IA is to combine the strengths of different models in order to build a very powerful and user-friendly pipeline for solving inpainting-related problems. IA supports three main features: (i) Remove Anything: users could click on an object and IA will remove it and smooth the ``hole'' with the context; (ii) Fill Anything: after certain objects removal, users could provide text-based prompts to IA, and then it will fill the hole with the corresponding generative content via driving AIGC models like Stable Diffusion; (iii) Replace Anything: with IA, users have another option to retain the click-selected object and replace the remaining background with the newly generated scenes. We are also very willing to help everyone share and promote new projects based on our Inpaint Anything (IA). Our codes are available at https://github.com/geekyutao/Inpaint-Anything.
Subfields: Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/72570016bbb28e8fb15ab4667eb84887f5dd35ad
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
Citations in the corpus
- EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (2023, background): “…shown remarkable zero-shot transfer performance and high versatility for many vision tasks including a variety of segmentation application[7, 8, 10, 17], in-painting[67], image restoration[29], image editing[21], image shadow removal[69], object tracking[14, 65], and 3D object reconstruction[49].”
- SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, methodology): “Based on the segmentation results from SAM, we refer to Inpaint Anything (Yu et al., 2023a) to integrate image inpainting models (LaMa (Suvorov et al., 2021)) and text-to-image generative models (Stable Diffusion (Rombach et al., 2021)) for high-quality image inpainting and editing.”
- LangSplat: 3D Language Gaussian Splatting (2023, background): “[38] introduced [48], super-resolution [25], image matting [45], object tracking [11, 42], medical image segmentation [29], image editing [12], and so on.”
- AnyDoor: Zero-shot Object-level Image Customization (2023, unknown): “Inpaint Anything [59] involves SAM [26] and Stable Diffusion [41] to replace any object in the source image with text described target.”