3d Object Generation Methods

Controllable Text-to-3D Generation via Surface-Aligned Gaussian Splatting While text-to-3D and image-to-3D generation tasks have received considerable attention, one important but under-explored field between them is controllable text-to-3D generation, which we mainly focus on in this work. To address this task, 1) we introduce Multi-view ControlNet (MVControl), a novel neural network architecture designed to enhance existing pre-trained multi-view diffusion models by integrating additional input conditions, such as edge, depth, normal, and scribble maps. Our innovation lies in the introduction of a conditioning module that controls the base diffusion model using both local and global embeddings, which are computed from the input condition images and camera poses. Once trained, MVControl is able to offer 3D diffusion guidance for optimization-based 3D generation. And, 2) we propose an efficient multi-stage 3D generation pipeline that leverages the benefits of recent large reconstruction models and score distillation algorithm. Building upon our MVControl architecture, we employ a unique hybrid diffusion guidance method to direct the optimization process. In pursuit of efficiency, we adopt 3D Gaussians as our representation instead of the commonly used implicit representations. We also pioneer the use of SuGaR, a hybrid representation that binds Gaussians to mesh triangle faces. This approach alleviates the issue of poor geometry in 3D Gaussians and enables the direct sculpting of fine-grained geometry on the mesh. Extensive experiments demonstrate that our method achieves robust generalization and enables the controllable generation of high-quality 3D content.

✨ DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation | It's a two-step process (Text-to-3DGaussian > Gaussian-to-Mesh), but the results look very promising from the project's demonstration media. Credit: Peking University, Nanyang Technological University Singapore, Baidu, Inc. Abstract: "Recent advances in 3D content creation mostly leverage optimization-based 3D generation via score distillation sampling (SDS). Though promising results have been exhibited, these methods often suffer from slow per-sample optimization, limiting their practical usage. In this paper, we propose DreamGaussian, a novel 3D content generation framework that achieves both efficiency and quality simultaneously. Our key insight is to design a generative 3D Gaussian Splatting model with companioned mesh extraction and texture refinement in UV space. In contrast to the occupancy pruning used in Neural Radiance Fields, we demonstrate that the progressive densification of 3D Gaussians converges significantly faster for 3D generative tasks. To further enhance the texture quality and facilitate downstream applications, we introduce an efficient algorithm to convert 3D Gaussians into textured meshes and apply a fine-tuning stage to refine the details. Extensive experiments demonstrate the superior efficiency and competitive generation quality of our proposed approach. Notably, DreamGaussian produces high-quality textured meshes in just 2 minutes from a single-view image, achieving approximately 10 times acceleration compared to existing methods." Project Page: https://lnkd.in/eFs-HMTV arXiv: https://lnkd.in/eGmrCRwU GitHub: https://lnkd.in/eq7sAN8z License (MIT): https://lnkd.in/eB25AjsZ For more like this ⤵︎ 👉 Follow Orbis Tabula • GenAI × VP × Consulting #GenerativeAI #3DGaussians #text2img #text23D

What do computer graphics and artificial intelligence have in common? GPUs! So naturally I had to look into the combination of this by reading up on the latest in 3D model generation with AI models. As it turns out, we can go from a text prompt to a full realistic 3D model now in less than a minute! AI researchers realized that the power of image generation (or diffusion) models already have a great understanding of 2D concepts and rushed to try and apply this to 3D representations. It started out by first using these diffusion models to generate novel views of objects. By training image generation models on many different viewpoints of objects, they learned to generalize this skill to any image input, which can already be generated from text. Companies like Stability AI also threw video models at the task to make even more robust multi view synthesis models. A step up from this are Neural Radiance Fields, or NeRFs that combine deep learning and computer graphics at an even deeper level for 3D representation AS a neural network! This concept has been improved with recent representations like gaussian splats and triplanes that do this differentiable 3D representation even better. These modes can be converted once more into meshes with algorithms like marching cubes, that can convert a density representation into meshes- a much better format for actual use in things like game design or 3D printing. You can catch up on how this works and the latest in state of the art text-to-3D GenAI modeling in my latest video: https://lnkd.in/eAQNQA8k