RAW files (RAF, CR2, DNG) have wrong face preview due to wrong dimensions #13049

skylatian · 2024-09-30T09:50:50Z

The bug

This bug differs from #12440 in that the preview box correctly surrounds the person's face, but the preview image that shows in the People tab (or in file details) is not zoomed/cropped to the correct location in a photo.

This issue is fixed when manually selecting "Select Featured Photo." This cases the profile picture/preview for that particular person to be cropped correctly.

(note - the image shown vs. source for "featured thumbnail" are slightly different photos (taken a few seconds apart), but the effect happens with that photo as well and I didn't want to re-blur for anonymity)

The overall "People" tab. Every one of these is a validly recognized person, the thumbnail ("Featured Photo") is just misaligned.

The OS that Immich Server is running on

Windows 10 Server

Version of Immich Server

v1.116.2

Version of Immich Mobile App

n/a

Platform with the issue

Server
Web
Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - "D:/nas/Media:/home/user/photos1:ro"
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false
      
  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2d1463258f2764328496376f5d965f20c6a67f66ea2b06dc42af351f75248792
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION="D:/immich-library"
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=America/[]

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=[redacted]

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

The issue:

Open Photos tab
Witness faces not being centered

The manual fix:

Click on a person in the People tab (or from elsewhere)
Top right menu->Select Featured Photo
(Choose a photo)
See that Featured Photo now appears correctly system-wide

Relevant log output

No response

Additional information

No response

C-Otto · 2024-10-08T12:12:20Z

If I understand you correctly, each person's face photo (as shown on the "/people" URL, i.e. the "People" page) is showing a wrong part of the correct image (e.g. a plant behind the person instead of the actual head)?

skylatian · 2024-10-09T06:32:16Z

Correct, yes! That’s a much better way of putting it.

C-Otto · 2024-10-09T06:53:00Z

Thank you for the clarification! Do you know if there's anything special about the people/faces/photos? Did you modify them? Can you reproduce this (with new people/photos)?

skylatian · 2024-10-09T07:41:16Z

I didn't realize this until now, but it seems like it might only be happening with Fuji Raw (.RAF) files! The same behavior occurs if I select a .RAF photo as the preview manually. I just checked with Sony's Raw files (ARW) and they select the face correctly.

C-Otto · 2024-10-09T07:43:15Z

That's great (for debugging) :) Could you please provide a sample .RAF picture with a face on it?

skylatian · 2024-10-09T08:29:04Z

I will get back to you on that. For the life of me, I can't find one that uploads and causes the same issue. I appreciate your help!

C-Otto · 2024-10-09T08:30:18Z

It's possible that this issue was fixed in the meantime...

skylatian · 2024-10-09T08:43:48Z

I just updated from 116 to 117 earlier today, that might be it. All the scans had been done on 116. I’m re-running the face detection+recognition jobs now, I will report back once they’re done.

stjeanp · 2024-10-09T14:21:37Z

I just rebuilt all thumbnails in my library (~40k images), the majority of which are CR2 raw files. I'm seeing the exact same issue. The box around the face is correct, but the part of the photo used for the person's picture is wrong. I picked one person, opened an image with a (to me) good face, confirmed that the box was actually around his face, then went back to the person and set that as his featured photo. His featured photo is now a section of wall, not his face. I'm happy to share the file, if you need more data.

I'd rather not redo all of the face detection and grouping, because that was a lot of work, but if y'all think it'd be worthwhile, I'll kick it off tonight (US/Central) and it should be done by morning and will report back.

Edit: I just kicked off a full face detection/recognition job. Will report back when it's done.

I'm also running 1.117.0.

C-Otto · 2024-10-09T15:53:48Z

@stjeanp yes, please share the file and let us know of any pattern you might suspect (only CR2 raw files, for example?)

stjeanp · 2024-10-09T17:30:26Z

@C-Otto The redo of the face detection/grouping completed and I've found a good example, since this person only appears in five files, all of which are CR2 files. I can't attach the zip of them because it's too big, but if you message me, I can give you a URL to download it. Sorry. Here's some explanatory screenshots though. Please let me know what else you'd like me to do.

Before any changes were made:

Here's the person's face box:

And here's one where her feature photo is now a piece of the wall instead of her face, after selecting the topmost image to use as her feature photo:

stjeanp · 2024-10-09T17:35:14Z

Sorry, here's the face box on the image I used as the new featured photo:

C-Otto · 2024-10-09T19:34:14Z

Thanks, that looks helpful. I just sent a mail regarding the link to the raw file(s).

stjeanp · 2024-10-09T19:43:08Z

Got it, replied with the link. Let me know when you're good and I'll take it down.

C-Otto · 2024-10-09T19:46:21Z

Got it, thanks - feel free to delete the file.

skylatian · 2024-10-09T20:10:12Z

Looks like you got a sample file, thanks for taking care of that @stjeanp

I can confirm it's still happening after rebuilding all faces and people on 1.117. Thanks for your help @C-Otto

C-Otto · 2024-10-09T20:25:15Z

The image dimensions are off, for the images provided by @stjeanp I see 1728x1152 (2MP), while Gimp shows 3522x2348 (8MP). Using exiftool I get:

Image Width                     : 1728
Image Height                    : 1152
Canon Image Width               : 3504
Canon Image Height              : 2336
AF Image Width                  : 3504
AF Image Height                 : 2336
Exif Image Width                : 3504
Exif Image Height               : 2336
Image Size                      : 3504x2336

Any idea what might be the cause of having three different sizes?

This wrong image size does not only mess up the "generate face preview" bit, but also shows in the details pane on the screenshots above.

C-Otto · 2024-10-09T20:37:05Z

@skylatian do you notice any weirdness related to reported and actual image dimensions? You should be able to see what Immich thinks in the detail view (which you included in your screenshots). Maybe the actual images don't match these dimensions?

skylatian · 2024-10-09T20:48:13Z

@C-Otto I do, good catch.

The immich-reported dimensions for the .RAF file are 2944 x 4416.
All of my .RAF files show either 2944x4416 or 4416x2944 (depending on orientation) so it’s not random.
The corresponding jpegs for each raw are 4160 x 6240 (as expected)
If I download and check the .RAF in file explorer or finder, the dimensions correctly show as 4160 x 6240.

It seems immich is listing the resolution as scaled down by a factor of ~0.707 for mine compared to ~0.49 for @stjeanp’s. Sony RAWs do not have this problem. I have no idea why exiftool would report three sizes, maybe it’s something related to the baked-jpeg preview?

mertalev · 2024-10-09T20:49:46Z

There is an "Exif Image Width/Height" field in the CR2 sample above that's accurate. Can you check with exiftool if the RAF file also has this? We could prefer this field when it exists during metadata extraction.

C-Otto · 2024-10-09T20:55:48Z

Are you able to see/extract the JPG preview image that's part of the raw image? If so, is this preview smaller than the actual image? In https://exiftool.org/forum/index.php?topic=4709.0 this is mentioned as a possible cause.

With exiftool -b -previewImage -w .jpg IMG_8119.CR2 I get a JPG file that has size 1728x1152, confirming this idea. If this also holds true for RAF files, we might need to tweak exiftool or exiftool-vendored, or work around this in Immich itself.

stjeanp · 2024-10-09T20:57:08Z

I'm not sure where those Image Width values are coming from. Looking at the original in Lightroom, it agrees with the others that the dimensions are 3504x2336. I thought it might be one of the thumbnails, but the standard view one is 2160x1440. I'm hanging on to that hunch, but can't see the other thumbnails. I'm looking on the filesystem now.

stjeanp · 2024-10-09T21:02:37Z

Interestingly enough, the CR2 format allows for embedded, smaller JPEG versions of the image. I'm looking for a tool to do a deep dive into that CR2 file to see if it has any of that in it. Sort of a structure dump.

skylatian · 2024-10-09T21:10:12Z

from exiftool -'*height*' -'*width* -'*size*' filename.RAF I get (organized):

Exif Image Height               : 2944
Exif Image Width                : 4416

Image Height                    : 2944
Image Width                     : 4416

Raw Image Height                : 1008
Raw Image Width                 : 6276

Raw Image Full Height           : 4182
Raw Image Full Width            : 6384

File Size                       : 59 MB
AF Area Point Size              : 4
AF Area Zone Size               : n/a
Grain Effect Size               : Small
Raw Image Full Size             : 6384x4182
Raw Image Cropped Size          : 6240x4160
Raw Zoom Size                   : 6240x4160
Image Size                      : 6240x4160

I have no idea where that RawImageHeight term comes from.

Output of exiftool -b -previewImage -w .jpg confirms that my built-in jpeg preview is 2944×4416, seems like it's from Exif Image Height and Exif Image Width.

mertalev · 2024-10-09T21:13:29Z

Of course this one doesn't use Exif Image Width/Height for the original... that would be too simple. Image Size looks a bit more promising as it's correct for both images.

stjeanp · 2024-10-11T02:49:49Z

If that looks like it'd work, that's great! If you need to add a one of my images to the test suite, I've got some better ones that I'd be happy to contribute. Please let me know if there's anything I can do to help.

mertalev · 2024-10-11T03:30:14Z

It'd be great if you could make a PR to add one in the test-assets repo!

C-Otto · 2024-10-11T06:11:24Z

If we go along the route of considering multiple tags, we could just take the maximum we find.

skylatian · 2024-10-11T06:13:28Z

Raw Image Full Size : 6384x4182
Raw Image Cropped Size : 6240x4160

You’d think, but apparently not! As far as I know, the “cropped” resolution here is the correct one. This is from a Fuji x100v

stjeanp · 2024-10-11T11:33:28Z

@mertalev - I'm looking at the repo now and will open a PR later today. I've upgraded bodies a few times and have CR2s from an EOS 30D, EOS 70D, and a PowerShot G12. The raws from the 30D and G12 are showing the smaller image size issue. I can't find a 70D RAW that's exhibiting this issue though.

Would you like me to include all three files in the PR?

C-Otto · 2024-10-11T11:35:06Z

Ideally you'd open separate PRs (and include the exiftool output in each). That'd be my preference, if that counts :)

stjeanp · 2024-10-11T11:40:56Z

Will do! Working on that now.

stjeanp · 2024-10-11T11:52:25Z

@C-Otto - Quick question: do you want people in them or just landscapes/buildings/etc. I've got both, and have people ones that are me and my spouse, both of which give permission to use the files.

C-Otto · 2024-10-11T12:03:34Z

I think the actual contents don't matter, we only care about the metadata. If you provide pictures of your wall or just plain black images, that still helps (and it might keep the filesize down).

stjeanp · 2024-10-11T12:38:03Z

Sounds good. There are three PRs waiting for review. Glad to have been able to help out, and please let me know if there's anything else I can do to help move this fix forward.

C-Otto · 2024-10-11T12:42:42Z

immich-app/test-assets#11 can be used to reproduce this, I think. Image Width/Height are 1600x1200, but 3648x2736 seems to be closer to the truth.

C-Otto · 2024-10-11T15:11:44Z

@skylatian can you provide a RAF file where neither "ImageWidth" nor "ExifImageWidth" show the correct width?

For some RAW formats/cameras, the ImageWidth/ImageHeight information represents the dimensions of the preview image, i.e. much smaller than the actual image. This also causes issues with (face/people) thumbnail generation. The information in ImageSize seems to be correct, so we use this instead. fixes #13049

C-Otto · 2024-10-11T15:50:43Z

How many faces/people are affected? Even with the fix, you'd need to somehow fix the preview images (manually is one way, but there are others).

stjeanp · 2024-10-11T15:53:31Z

For me, I've got 900+ detected faces over ~40k images. I don't mind doing a rebuild once the fix is in. That might be onerous for some users though.

skylatian · 2024-10-11T16:38:51Z

@skylatian can you provide a RAF file where neither "ImageWidth" nor "ExifImageWidth" show the correct width?

Happy to, I'll open a PR later today.

Unfortunately, I looked at a raw (.DNG) from a family member's Ricoh GR3, and neither ImageWidth nor Image Size lists the correct resolution.

Image Height                    : 4064
Image Width                     : 6112
File Size                       : 31 MB
Default Crop Size               : 6000 4000
Preview Image Size              : 720x480
Image Size                      : 6112x4064

Image Size and Image Width are the same, but the correct resolution is Default Crop Size as confirmed by Lightroom and Finder. If I'm not mistaken, #13377 wouldn't address this. Thoughts?

mertalev · 2024-10-11T18:13:39Z

@skylatian can you provide a RAF file where neither "ImageWidth" nor "ExifImageWidth" show the correct width?

Happy to, I'll open a PR later today.

Unfortunately, I looked at a raw (.DNG) from a family member's Ricoh GR3, and neither ImageWidth nor Image Size lists the correct resolution.
Image Height                    : 4064

Image Width                     : 6112

File Size                       : 31 MB

Default Crop Size               : 6000 4000

Preview Image Size              : 720x480

Image Size                      : 6112x4064
Image Size and Image Width are the same, but the correct resolution is Default Crop Size as confirmed by Lightroom and Finder. If I'm not mistaken, #13377 wouldn't address this. Thoughts?

This is truly incredible.

At least using Image Size wouldn't be a regression in this case. Maybe we can consider Default Crop Size as well if it wouldn't cause issues elsewhere.

stjeanp · 2024-10-11T18:22:17Z

It's not going to bother the three Canon bodies I've got RAWs from, none of them use that tag.

stjeanp@gepetto:~$ ls Canon*
CanonEOS30D.CR2  CanonEOS70D.CR2  CanonPowerShotG12.CR2
stjeanp@gepetto:~$ fgrep -i 'default crop size' Canon*
stjeanp@gepetto:~$

skylatian · 2024-10-11T20:47:02Z

This is truly incredible.

Isn't it? Update on the Ricoh, though. I hadn't run exiftool with the -a flag previously, and running it with that reveals:

exiftool -'*height*' -'*width*' -'*size*' -'*jpg*' -a -g1 file.DNG

---- File ----
File Size                       : 31 MB
---- EXIF ----
Image Width                     : 160
Image Height                    : 120
Image Width                     : 6112
Image Height                    : 4064
Default Crop Size               : 6000 4000
Image Width                     : 6000
Image Height                    : 4000
---- MakerNotes ----
Preview Image Size              : 720x480
---- Composite ----
Image Size                      : 6112x4064

So... there are THREE Image Width tags? This is unique to the Ricoh, as far as I can tell. I agree Default Crop Size is probably the right move for this camera. That tag isn't present in the Fufi RAFs, though.

C-Otto · 2024-10-12T08:57:58Z

Please provide interesting example images so that I can tune the PR and retain that knowledge in automated tests. It doesn't make a lot of sense to break metadata extraction for other cameras, nor does it feel right to ignore information already noted in this issue. It's a mess, but if we're able to hide that mess in somewhat clean code, I'd say that's an improvement.

For some RAW formats/cameras, the ImageWidth/ImageHeight information represents the dimensions of the preview image, i.e. much smaller than the actual image. This also causes issues with (face/people) thumbnail generation. The information in ImageSize seems to be correct, so we use this instead. fixes #13049

For some RAW formats/cameras, the ImageWidth/ImageHeight information represents the dimensions of the preview image, i.e. much smaller than the actual image. This also causes issues with (face/people) thumbnail generation. The information in ImageSize seems to be correct, so we use this instead. fixes #13049 Add Fujifilm X100v RAW File (#12) Added Fujifilm X100v RAF RAW File

C-Otto · 2024-10-15T06:41:24Z

@skylatian could you provide a problematic DNG file, please?

For some RAW formats/cameras, the ImageWidth/ImageHeight information represents the dimensions of the preview image, i.e. much smaller than the actual image. This also causes issues with (face/people) thumbnail generation. The information in ImageSize seems to be correct, so we use this instead. fixes #13049 Add Fujifilm X100v RAW File (#12) Added Fujifilm X100v RAF RAW File

C-Otto changed the title ~~People thumbnail images cropped to incorrect region~~ RAF files: People thumbnail images cropped to incorrect region Oct 9, 2024

C-Otto changed the title ~~RAF files: People thumbnail images cropped to incorrect region~~ RAF files: Face preview show wrong part of image Oct 9, 2024

C-Otto changed the title ~~RAF files: Face preview show wrong part of image~~ RAF files: Face preview shows wrong part of image Oct 9, 2024

C-Otto mentioned this issue Oct 9, 2024

Unable to run job handler (thumbnailGeneration/generate-person-thumbnail): Error: extract_area: bad extract area #12530

Open

3 tasks

C-Otto changed the title ~~RAF files: Face preview shows wrong part of image~~ RAF and CR2 files: Face preview wrong due to wrong dimensions Oct 9, 2024

C-Otto linked a pull request Oct 11, 2024 that will close this issue

fix(server): wrong image dimensions for RAW files (RAF, CR2, DNG) (also fixes face preview) #13377

Draft

C-Otto changed the title ~~RAF and CR2 files: Face preview wrong due to wrong dimensions~~ RAW files (RAF, CR2, DNG) have wrong face preview due to wrong dimensions Oct 15, 2024

RAW files (RAF, CR2, DNG) have wrong face preview due to wrong dimensions #13049

RAW files (RAF, CR2, DNG) have wrong face preview due to wrong dimensions #13049

Comments

skylatian commented Sep 30, 2024

The bug

The OS that Immich Server is running on

Version of Immich Server

Version of Immich Mobile App

Platform with the issue

Your docker-compose.yml content

Your .env content

Reproduction steps

Relevant log output

Additional information

C-Otto commented Oct 8, 2024 • edited Loading

skylatian commented Oct 9, 2024

C-Otto commented Oct 9, 2024

skylatian commented Oct 9, 2024 • edited Loading

C-Otto commented Oct 9, 2024

skylatian commented Oct 9, 2024

C-Otto commented Oct 9, 2024

skylatian commented Oct 9, 2024

stjeanp commented Oct 9, 2024 • edited Loading

C-Otto commented Oct 9, 2024

stjeanp commented Oct 9, 2024

stjeanp commented Oct 9, 2024

C-Otto commented Oct 9, 2024

stjeanp commented Oct 9, 2024

C-Otto commented Oct 9, 2024 • edited Loading

skylatian commented Oct 9, 2024

C-Otto commented Oct 9, 2024 • edited Loading

C-Otto commented Oct 9, 2024 • edited Loading

skylatian commented Oct 9, 2024 • edited Loading

mertalev commented Oct 9, 2024

C-Otto commented Oct 9, 2024 • edited Loading

stjeanp commented Oct 9, 2024

stjeanp commented Oct 9, 2024

skylatian commented Oct 9, 2024

mertalev commented Oct 9, 2024

stjeanp commented Oct 11, 2024

mertalev commented Oct 11, 2024

C-Otto commented Oct 11, 2024

skylatian commented Oct 11, 2024

stjeanp commented Oct 11, 2024

C-Otto commented Oct 11, 2024

stjeanp commented Oct 11, 2024

stjeanp commented Oct 11, 2024

C-Otto commented Oct 11, 2024

stjeanp commented Oct 11, 2024

C-Otto commented Oct 11, 2024

C-Otto commented Oct 11, 2024

C-Otto commented Oct 11, 2024

stjeanp commented Oct 11, 2024

skylatian commented Oct 11, 2024 • edited Loading

mertalev commented Oct 11, 2024

stjeanp commented Oct 11, 2024

skylatian commented Oct 11, 2024

C-Otto commented Oct 12, 2024 • edited Loading

C-Otto commented Oct 15, 2024

C-Otto commented Oct 8, 2024 •

edited

Loading

skylatian commented Oct 9, 2024 •

edited

Loading

stjeanp commented Oct 9, 2024 •

edited

Loading

C-Otto commented Oct 9, 2024 •

edited

Loading

C-Otto commented Oct 9, 2024 •

edited

Loading

C-Otto commented Oct 9, 2024 •

edited

Loading

skylatian commented Oct 9, 2024 •

edited

Loading

C-Otto commented Oct 9, 2024 •

edited

Loading

skylatian commented Oct 11, 2024 •

edited

Loading

C-Otto commented Oct 12, 2024 •

edited

Loading