[Feature] Pet detection

[alternativesurfer]

The feature

I notice that dogs get picked up with the "people" facial recognition but cats don't.
Any thoughts on how we could add that?

Platform

• Server
• Web
• Mobile

[arhills]

One of my cats got detected as a person. The other 3 did not. None of my dogs got detected. This might not help at all :)

[aviv926]

There are few models trained for such an operation, do you know a model good enough for such a purpose?
The model we currently use for facial recognition does not support animal recognition.

As a temporary alternative: you can describe your pet in the search, I believe that the CLIP model on which the search is based will be able to find your pet with a fairly high accuracy.

[alternativesurfer]

True, CLIP does appear to search properly.

[rodrigopolo]

Here is one:
https://github.com/MarQuisCheshire/Pets-Face-Recognition

[Chuckame]

How to use it in immich? Wouldn't it replace the current face detection so we lose human faces?

[tekjoey]

I just realized Apple Photos does pet detection, so this is a feature I would like for sure. Immich picked up some of my dogs, but not all of them, and not all photos of dogs. I certainly would like to be able to link photos with a pet. My current implementation is manually adding to albums, which works but isn't great.

(Some musings for the devs) If this feature ever does get implemented, I think there should be a per-user setting to display the pet faces on the same page as people, on a separate (pet only) page, or not at all (currently Apple Photos displays all faces together). Perhaps the database could be updated with a new value of "face_type" with options of "person" or "pet" (maybe specify the type of pet? Maybe the type of pet can be configurable by the user?) Also, when a model does get chosen/trained, please remember that cats and dogs are not the only pets that people have! My wife has horses, so that is a must for us, but lots of others exist as well. (again, these are just some ideas. Take all, some, or none as you see fit)

[gilgwath]

Being able to recognise our cats would probably quadruple the spouse approval factor. I think about a third of our photos are of our cats :-)

[mrshmllow]

Unironically this is blocking me from switching from google photos

[VACIndustries]

It's worth noting that it may drive user adoption adding this properly. Google Photos does this at a very basic level. If your pet passes and you get another pet (similar breed), you cannot create a new pet and have it recognize it as a new pet. It continues to tag it as the old pet. No way to delete said old pet either.

This is what drove me to at least find Immich (at which point I realized there was zero pet recognition).

[TAT-Hins]

Hi everyone. This is my first time in the discussion, so sorry if I'm not expressing in the right way.

It's worth noting that it may drive user adoption adding this properly. Google Photos does this at a very basic level. If your pet passes and you get another pet (similar breed), you cannot create a new pet and have it recognize it as a new pet. It continues to tag it as the old pet. No way to delete said old pet either.

This is what drove me to at least find Immich (at which point I realized there was zero pet recognition).

I have a cat and two dogs. It's very easy to use smart search by ML clip models to locate the photos of them, so I suppose it should also work with facial recognition, so that I can tag and name them just like people to make it more easily, which seems to be a possible solution for the doubt.

I'm pretty sure that the feature is technically feasible, and I notice that Google Photos has this feature for a long time, but I have no idea how much work it should take to achieve this. It would be awesome if immich can make this happen, and I hope to be a part of the contribution, no matter as a code debugger or just as a user.

[WizenThorne]

I'd love to have this feature as well.

[Chuckame]

Just putting some random links about Dave detection for pets. It seems to be really hard to find on the internet...!

The most advanced source I've found: https://github.com/GuillaumeMougeot/DogFaceNet

ByteDance seems to provide api for cat and dog recognition (at which price?) : https://docs.byteplus.com/en/docs/effects/docs-pet-face-recognition

As generally people have only a few pets, maybe we could more have a clustering of similar dog than pure face recognition 🤔

[Chuckame]

For cats : https://levelup.gitconnected.com/detect-your-cats-face-with-opencv-21893210737e

[KuramaSyu]

Where in the code is the current face detection done? If it's not too hard, I would consider making a PR. I don't think that it would be so hard to use 2 AI models for scanning images.

[aviv926]

I'm pretty sure it's done here
https://github.com/immich-app/immich/tree/main/machine-learning/app/models/facial_recognition

You might want to consult with @mertalev before starting work on such a PR, to check if it is something that will be accepted for the project.

[mertalev]

Adding it to immich is of course fine, but getting a pet recognition (not detection) model in the first place is more involved. There's some discussion on training a model for this on Discord.

[pilot51]

I just discovered Immich today (thanks Google feed) and I'm interested in switching from Google Photos to a self-hosted solution. I have it running and I really like what I see, but it'll be hard for me to commit to a full switch if cat detection isn't working.

My cats are like 90% of all the faces recognized in Google Photos, which automatically adds the photos to an album for both cats shared with my family.

[aviv926]

I just discovered Immich today (thanks Google feed) and I'm interested in switching from Google Photos to a self-hosted solution. I have it running and I really like what I see, but it'll be hard for me to commit to a full switch if cat detection isn't working.

My cats are like 90% of all the faces recognized in Google Photos, which automatically adds the photos to an album for both cats shared with my family.

You can always use the smart search for now (hint, it really is smart) Check it out in its demo (this is the basic model) Read more here:
https://immich.app/docs/features/smart-search

Also:

https://demo.immich.app

[pilot51]

You can always use the smart search for now (hint, it really is smart) Check it out in its demo (this is the basic model) Read more here: https://immich.app/docs/features/smart-search

That works well for generic searches like "cat". However, what if I only want to find photos of a pet by name?

I would also like to ensure accurate results that show all of the desired photos and no false positives. It's practically impossible to be sure with on-demand results like that.
For example, I just searched for "cat" and it's very accurate at the top of the results but quickly becomes interspersed with other animals and eventually random photos with 0% cat-like objects as you scroll down (it's even worse in the demo with only one cat photo), which tells me there's no certainty cut-off.
That's why I like to use albums that I can curate myself, in this case assisted by ML to add detected faces to the album, which is essential for such high volumes of cat photos.

One thing Google Photos really struggles with is identifying which cat is which (they're both the same color but with distinct features), so I regularly have to correct it by removing the wrong detected cat and selecting the correct one. I don't expect any solution here to be much better in the short term, but if anything comes close to that and lets me easily correct detection errors, I'll be happy. Google hasn't improved at all in the 5 years I've had these cats, so I can see the open source community eventually doing it better.