3D Scene Creation from Text Descriptions

✨ DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation | It's a two-step process (Text-to-3DGaussian > Gaussian-to-Mesh), but the results look very promising from the project's demonstration media. Credit: Peking University, Nanyang Technological University Singapore, Baidu, Inc. Abstract: "Recent advances in 3D content creation mostly leverage optimization-based 3D generation via score distillation sampling (SDS). Though promising results have been exhibited, these methods often suffer from slow per-sample optimization, limiting their practical usage. In this paper, we propose DreamGaussian, a novel 3D content generation framework that achieves both efficiency and quality simultaneously. Our key insight is to design a generative 3D Gaussian Splatting model with companioned mesh extraction and texture refinement in UV space. In contrast to the occupancy pruning used in Neural Radiance Fields, we demonstrate that the progressive densification of 3D Gaussians converges significantly faster for 3D generative tasks. To further enhance the texture quality and facilitate downstream applications, we introduce an efficient algorithm to convert 3D Gaussians into textured meshes and apply a fine-tuning stage to refine the details. Extensive experiments demonstrate the superior efficiency and competitive generation quality of our proposed approach. Notably, DreamGaussian produces high-quality textured meshes in just 2 minutes from a single-view image, achieving approximately 10 times acceleration compared to existing methods." Project Page: https://lnkd.in/eFs-HMTV arXiv: https://lnkd.in/eGmrCRwU GitHub: https://lnkd.in/eq7sAN8z License (MIT): https://lnkd.in/eB25AjsZ For more like this ⤵︎ 👉 Follow Orbis Tabula • GenAI × VP × Consulting #GenerativeAI #3DGaussians #text2img #text23D

Tencent presents TIP-Editor An Accurate 3D Editor Following Both Text-Prompts And Image-Prompts paper page: https://lnkd.in/ef3uC2d5 Text-driven 3D scene editing has gained significant attention owing to its convenience and user-friendliness. However, existing methods still lack accurate control of the specified appearance and location of the editing result due to the inherent limitations of the text description. To this end, we propose a 3D scene editing framework, TIPEditor, that accepts both text and image prompts and a 3D bounding box to specify the editing region. With the image prompt, users can conveniently specify the detailed appearance/style of the target content in complement to the text description, enabling accurate control of the appearance. Specifically, TIP-Editor employs a stepwise 2D personalization strategy to better learn the representation of the existing scene and the reference image, in which a localization loss is proposed to encourage correct object placement as specified by the bounding box. Additionally, TIPEditor utilizes explicit and flexible 3D Gaussian splatting as the 3D representation to facilitate local editing while keeping the background unchanged. Extensive experiments have demonstrated that TIP-Editor conducts accurate editing following the text and image prompts in the specified bounding box region, consistently outperforming the baselines in editing quality, and the alignment to the prompts, qualitatively and quantitatively.

Sharing another work from our team at Meta GenAI on 3D scene generation. ControlRoom3D is a work led by our research intern Jonas Schult. Despite the rapid progress of image and 3D object generation, generating 3D environment remain an unsolved problem. In this work, Jonas designed a system that let users first define the layout of a room by simply putting a bunch of bonding boxes together, with each bounding box representing a piece of furniture such as sofa, furnace, etc. Then, users provide a text prompt describing the style of the room, e.g., "Victoria style living room", and the model will generate a 3D room representation containing both mesh and texture. See the demo for an idea. Despite at its early stage, this work paves way for a new way of democratized 3D creation, enabling more people without years of training and expertise to create 3D environments for applications such as gaming, movie, AR/VR, and the real world. For more technical details, refer to the project page and paper for details: https://lnkd.in/gfzmXfpf