Innovations in Depth Estimation Techniques

🖥 Whereas yesterday I highlighted MiDaS from 2019, the 'next gen' version of this would be ZoeDepth which was released in February of this year. While MiDaS was a very effective 'relative depth' estimator, there are some limitations to how that can be used for certain projects. ZoeDepth aims to have consistency across both relative and metric depth. "Existing work either focuses on generalization performance disregarding metric scale, i.e. relative depth estimation, or state-of-the-art results on specific datasets, i.e. metric depth estimation. We propose the first approach that combines both worlds, leading to a model with excellent generalization performance while maintaining metric scale. Our flagship model, ZoeD-M12-NK, is pre-trained on 12 datasets using relative depth and fine-tuned on two datasets using metric depth. We use a lightweight head with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier. Our framework admits multiple configurations depending on the datasets used for relative depth pre-training and metric fine-tuning. Without pre-training, we can already significantly improve the state of the art (SOTA) on the NYU Depth v2 indoor dataset. Pre-training on twelve datasets and fine-tuning on the NYU Depth v2 indoor dataset, we can further improve SOTA for a total of 21% in terms of relative absolute error (REL). Finally, ZoeD-M12-NK is the first model that can jointly train on multiple datasets (NYU Depth v2 and KITTI) without a significant drop in performance and achieve unprecedented zero-shot generalization performance to eight unseen datasets from both indoor and outdoor domains." via Shariq Farooq, Reiner Birkl, Diana Wofk, Peter Wonka, Matthias Müller GitHub: https://lnkd.in/ePSQA_xh arXiv: https://lnkd.in/eXs6Czjw 🤗 Demo: https://lnkd.in/eDBe74ug For more like this ⤵ 👉 Follow Orbis Tabula // Digital Twins • Reality Capture • Generative AI #depthestimation #relativedepth #metricdepth

Apple's DepthPro is quite impressive, producing pixel-perfect, high-resolution metric depth maps with sharp boundaries through monocular depth estimation. It outperforms all of its contenders like Metric3D v2 and DepthAnything in "in-the-wild" dynamic scenes. In this article, we will get an in-depth understanding of the DepthPro model architecture and training strategy, which are the core reasons that make the model stand out. To make this read more interesting, we explore cross-functional applications of depth map in image editing software, where we tried a few applications like: - Simulating Depth of Field - Depth Blur - Parallax Effect to Static Images - 3D Point Cloud Projection of 2D Images Using Depth Maps https://buff.ly/3E1c5l7 Individuals working at the intersection of computer vision and creative digital tools will find the application section highly engaging. #DepthPro #MonocularDepth #AppleResearch #DepthAnythingV2 #MonocularDepthApplications

On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation Recent advances in monocular depth estimation have been made by incorporating natural language as additional guidance. Although yielding impressive results, the impact of the language prior, particularly in terms of generalization and robustness, remains unexplored. In this paper, we address this gap by quantifying the impact of this prior and introduce methods to benchmark its effectiveness across various settings. We generate "low-level" sentences that convey object-centric, three-dimensional spatial relationships, incorporate them as additional language priors and evaluate their downstream impact on depth estimation. Our key finding is that current language-guided depth estimators perform optimally only with scene-level descriptions and counter-intuitively fare worse with low level descriptions. Despite leveraging additional data, these methods are not robust to directed adversarial attacks and decline in performance with an increase in distribution shift. Finally, to provide a foundation for future research, we identify points of failures and offer insights to better understand these shortcomings. With an increasing number of methods using language for depth estimation, our findings highlight the opportunities and pitfalls that require careful consideration for effective deployment in real-world settings

Depth Anything V2 Monocular Depth Estimation (Explanation and Real Time Demo) I will discuss the monocular depth estimation model Depth Anything V2 in detail, talking about the challenges it solves, benefits and challenges with synthetic data, the student-teacher model architecture, the annotation pipeline, performance on reflective/transparent surfaces, and a real-time Demo. Code and Doc: https://lnkd.in/gJTzSchf 0:00 Introduction 1:13 Monocular Depth Estimation Applications 1:34 Key Performance Metrics 2:30 The Problems with Real Labeled Data 3:27 Advantages and Challenges with Synthetic Data 4:56 Depth Anything V2 Architecture via Student-Teacher Model 6:08 Depth Anything V2 Annotation Pipeline 6:41 Benchmark on Standard Datasets 7:13 DA-2K Dataset for Depth Anything V2 8:14 Reflective Surfaces using Depth Anything V2 8:57 Transparent Objects using Depth Anything V2 9:52 Fine Details using Depth Anything V2 10:35 Depth Anything V2 Real-Time Demo https://lnkd.in/g-bjzDbE