Add Prompt Depth Anything Model #35401
Conversation
|
@NielsRogge @qubvel @pcuenca Could you help review this PR when you have some time? Thanks so much in advance! Let me know if you have any questions or suggestions. 😊 |
tests/models/prompt_depth_anything/test_modeling_prompt_depth_anything.py
Outdated
Show resolved
Hide resolved
|
Hi @haotongl! Thanks for working on the model integration to transformers 🤗 I'm on holidays until Jan 3rd, and I'll do a review after that if it's still necessary. |
|
|
||
| *Prompts play a critical role in unleashing the power of language and vision foundation models for specific tasks. For the first time, we introduce prompting into depth foundation models, creating a new paradigm for metric depth estimation termed Prompt Depth Anything. Specifically, we use a low-cost LiDAR as the prompt to guide the Depth Anything model for accurate metric depth output, achieving up to 4K resolution. Our approach centers on a concise prompt fusion design that integrates the LiDAR at multiple scales within the depth decoder. To address training challenges posed by limited datasets containing both LiDAR depth and precise GT depth, we propose a scalable data pipeline that includes synthetic data LiDAR simulation and real data pseudo GT depth generation. Our approach sets new state-of-the-arts on the ARKitScenes and ScanNet++ datasets and benefits downstream applications, including 3D reconstruction and generalized robotic grasping.* | ||
|
|
||
| <img src="https://promptda.github.io/assets/teaser.jpg" |
There was a problem hiding this comment.
Feel free to open a PR on this repo, specifically this folder: https://huggingface.co/datasets/huggingface/documentation-images/tree/main/transformers/model_doc to add a prompt_depth_anything_architecture.jpg picture
There was a problem hiding this comment.
Hi, thanks for your kind help! I have uploaded the image and opened a PR.
https://huggingface.co/datasets/huggingface/documentation-images/discussions/408
src/transformers/models/prompt_depth_anything/modeling_prompt_depth_anything.py
Outdated
Show resolved
Hide resolved
src/transformers/models/prompt_depth_anything/modeling_prompt_depth_anything.py
Outdated
Show resolved
Hide resolved
| >>> prompt_depth_url = "https://github.com/DepthAnything/PromptDA/blob/main/assets/example_images/arkit_depth.png?raw=true" | ||
| >>> prompt_depth = Image.open(requests.get(prompt_depth_url, stream=True).raw) | ||
| >>> prompt_depth = torch.tensor((np.asarray(prompt_depth) / 1000.0).astype(np.float32)) | ||
| >>> prompt_depth = prompt_depth.unsqueeze(0).unsqueeze(0) |
There was a problem hiding this comment.
Usage-wise, it might be a good idea to create a PromptDepthAnythingProcessor to help handle processing the input image (via the image processor) and then optional prompt_depth input.
There was a problem hiding this comment.
I have added a PromptDepthAnythingImageProcessor! Thank you for your suggestions!
What does this PR do?
This PR adds the Prompt Depth Anything Model. Prompt Depth Anything builds upon Depth Anything V2 and incorporates metric prompt depth to enable accurate and high-resolution metric depth estimation.
The implementation leverages Modular Transformers. The main file can be found here.
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.