Postgres disconnect issue after updating to 1.125.3

[swanepoelriaan]

The bug

After updating to the latest release, the main PHOTOS page when opening triggers a failed to load Assets and a "CONNECTION_CLOSED" on the database. The EXPLORE page is still functional and displays the correct images but the main landing page displays two errors and fails to load any images.

The OS that Immich Server is running on

Ubuntu 24.04.1 LTS

Version of Immich Server

v1.125.3

Version of Immich Mobile App

0.0.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:eaba718fecd1196d88533de7ba49bf903ad33664a92debb24660a922ecd9cac8
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: >-
        pg_isready --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" || exit 1;
        Chksum="$$(psql --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" --tuples-only --no-align
        --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')";
        echo "checksum failure count is $$Chksum";
        [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: >-
      postgres
      -c shared_preload_libraries=vectors.so
      -c 'search_path="$$user", public, vectors'
      -c logging_collector=on
      -c max_wal_size=2GB
      -c shared_buffers=512MB
      -c wal_compression=on
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=******************

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
root@stone-immich:~/immich-app#

Reproduction steps

1. Log in to Immich web page
2. Bug is triggered on the main photos page.
3. The Explore page is still functional and displays all content.
   ...

Relevant log output

[Nest] 17  - 01/26/2025, 7:44:22 PM     LOG [Api:Bootstrap] Immich Server is listening on http://[::1]:2283 [v1.125.3] [production]
[Nest] 17  - 01/26/2025, 7:44:24 PM     LOG [Api:EventRepository] Websocket Connect:    B9Z4_55VDlYV4FGZAAAB
Query failed : {
  durationMs: 322.66821100000016,
  error: Error: write CONNECTION_CLOSED database:5432
      at Socket.closed (/usr/src/app/node_modules/postgres/cjs/src/connection.js:438:57)
      at Socket.emit (node:events:524:28)
      at TCP.<anonymous> (node:net:351:12)
      at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
    code: 'CONNECTION_CLOSED',
    errno: 'CONNECTION_CLOSED',
    address: [ 'database' ],
    port: [ 5432 ]
  },
  sql: `with "res" as (with "today" as (select make_date(year::int, $1::int, $2::int) as "date" from generate_series((select date_part('year', min(("localDateTime" at time zone 'UTC')::date))::int from assets), date_part('year', current_date)::int - 1) as "year") select "a".*, to_json("exif") as "exifInfo" from "today" inner join lateral (select "assets".* from "assets" inner join "asset_job_status" on "assets"."id" = "asset_job_status"."assetId" where "asset_job_status"."previewAt" is not null and (assets."localDateTime" at time zone 'UTC')::date = today.date and "assets"."ownerId" = any($3::uuid[]) and "assets"."isVisible" = $4 and "assets"."isArchived" = $5 and exists (select from "asset_files" where "assetId" = "assets"."id" and "asset_files"."type" = $6) and "assets"."deletedAt" is null limit $7) as "a" on true inner join "exif" on "a"."id" = "exif"."assetId") select ((now() at time zone 'UTC')::date - ("localDateTime" at time zone 'UTC')::date) / 365 as "yearsAgo", json_agg("res") as "assets" from "res" group by ("localDateTime" at time zone 'UTC')::date order by ("localDateTime" at time zone 'UTC')::date desc limit $8`,
  params: [
    1,
    26,
    '{5a4c0e6c-bef9-4bb3-9662-8196b6de14e6}',
    true,
    false,
    'preview',
    20,
    10
  ]
}
[Nest] 17  - 01/26/2025, 7:44:39 PM   ERROR [Api:ErrorInterceptor~1dlwgjdj] Unknown error: Error: write CONNECTION_CLOSED database:5432
Error: write CONNECTION_CLOSED database:5432
    at Socket.closed (/usr/src/app/node_modules/postgres/cjs/src/connection.js:438:57)
    at Socket.emit (node:events:524:28)
    at TCP.<anonymous> (node:net:351:12)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
Postgres notice: {
  severity_local: 'WARNING',
  severity: 'WARNING',
  code: '57P02',
  message: 'terminating connection because of crash of another server process',
  detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
  hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
  file: 'postgres.c',
  line: '2888',
  routine: 'quickdie'
}
Postgres notice: {
  severity_local: 'WARNING',
  severity: 'WARNING',
  code: '57P02',
  message: 'terminating connection because of crash of another server process',
  detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
  hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
  file: 'postgres.c',
  line: '2888',
  routine: 'quickdie'
}
Postgres notice: {
  severity_local: 'WARNING',
  severity: 'WARNING',
  code: '57P02',
  message: 'terminating connection because of crash of another server process',
  detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
  hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
  file: 'postgres.c',
  line: '2888',
  routine: 'quickdie'
}
Postgres notice: {
  severity_local: 'WARNING',
  severity: 'WARNING',
  code: '57P02',
  message: 'terminating connection because of crash of another server process',
  detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
  hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
  file: 'postgres.c',
  line: '2888',
  routine: 'quickdie'
}
Postgres notice: {
  severity_local: 'WARNING',
  severity: 'WARNING',
  code: '57P02',
  message: 'terminating connection because of crash of another server process',
  detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
  hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
  file: 'postgres.c',
  line: '2888',
  routine: 'quickdie'
}
Postgres notice: {
  severity_local: 'WARNING',
  severity: 'WARNING',
  code: '57P02',
  message: 'terminating connection because of crash of another server process',
  detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
  hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
  file: 'postgres.c',
  line: '2888',
  routine: 'quickdie'
}
Postgres notice: {
  severity_local: 'WARNING',
  severity: 'WARNING',
  code: '57P02',
  message: 'terminating connection because of crash of another server process',
  detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
  hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
  file: 'postgres.c',
  line: '2888',
  routine: 'quickdie'
}
Postgres notice: {
  severity_local: 'WARNING',
  severity: 'WARNING',
  code: '57P02',
  message: 'terminating connection because of crash of another server process',
  detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
  hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
  file: 'postgres.c',
  line: '2888',
  routine: 'quickdie'
}
Postgres notice: {
  severity_local: 'WARNING',
  severity: 'WARNING',
  code: '57P02',
  message: 'terminating connection because of crash of another server process',
  detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
  hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
  file: 'postgres.c',
  line: '2888',
  routine: 'quickdie'
}
/usr/src/app/node_modules/postgres/cjs/src/connection.js:250
    const x = socket.write(chunk, fn)
                     ^

TypeError: Cannot read properties of null (reading 'write')
    at Immediate.nextWrite (/usr/src/app/node_modules/postgres/cjs/src/connection.js:250:22)
    at process.processImmediate (node:internal/timers:491:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)

Node.js v22.13.1
api worker exited with code 1
Initializing Immich v1.125.3
Detected CPU Cores: 4
Starting api worker
Starting microservices worker
[Nest] 7  - 01/26/2025, 7:44:56 PM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:MetadataService] Bootstrapping metadata service
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:MetadataService] Initializing metadata service
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:MapRepository] Initializing metadata repository
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:MetadataService] Initialized local reverse geocoder
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:ServerService] Feature Flags: {
  "smartSearch": true,
  "facialRecognition": true,
  "duplicateDetection": true,
  "map": true,
  "reverseGeocoding": true,
  "importFaces": false,
  "sidecar": true,
  "search": true,
  "trash": true,
  "oauth": false,
  "oauthAutoLaunch": false,
  "passwordLogin": true,
  "configFile": false,
  "email": false
}
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:StorageService] Verifying system mount folder checks, current state: {"mountChecks":{"thumbs":true,"upload":true,"backups":true,"library":true,"profile":true,"encoded-video":true}}
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:StorageService] Successfully verified system mount folder checks
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:SystemConfigService] LogLevel=log (set via system config)
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:NestFactory] Starting Nest application...
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:InstanceLoader] BullModule dependencies initialized
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:InstanceLoader] OpenTelemetryModule dependencies initialized
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:InstanceLoader] TypeOrmModule dependencies initialized
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:InstanceLoader] KyselyModule dependencies initialized
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:InstanceLoader] KyselyCoreModule dependencies initialized
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:InstanceLoader] OpenTelemetryCoreModule dependencies initialized
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:InstanceLoader] DiscoveryModule dependencies initialized
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:InstanceLoader] ClsModule dependencies initialized
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:InstanceLoader] BullModule dependencies initialized
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:InstanceLoader] BullModule dependencies initialized
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:InstanceLoader] TypeOrmCoreModule dependencies initialized
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:InstanceLoader] TypeOrmModule dependencies initialized
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:InstanceLoader] MicroservicesModule dependencies initialized
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:NestApplication] Nest application successfully started
[Nest] 7  - 01/26/2025, 7:44:57 PM     LOG [Microservices:Bootstrap] Immich Microservices is running [v1.125.3] [production]
[Nest] 17  - 01/26/2025, 7:44:57 PM     LOG [Api:EventRepository] Initialized websocket server
[Nest] 17  - 01/26/2025, 7:44:57 PM     LOG [Api:ServerService] Feature Flags: {
  "smartSearch": true,
  "facialRecognition": true,
  "duplicateDetection": true,
  "map": true,
  "reverseGeocoding": true,
  "importFaces": false,
  "sidecar": true,
  "search": true,
  "trash": true,
  "oauth": false,
  "oauthAutoLaunch": false,
  "passwordLogin": true,
  "configFile": false,
  "email": false

Additional information

[alextran1502]

What did you update from? If you revert back to 1.125.2, do you see the error?

[bo0tzz]

Check the logs of the postgres container, which will be inside ./postgres/log. I suspect this is your postgres database crashing and not an Immich bug.

[swanepoelriaan]

I've updated from 1.125.2 without any issues, what are the step to revert back to v1.125.2? I don't have any issues opening images on other pages, just the main Photos page. So I don't suspect any postgres issues, here is the log error in the postgres folder:

2025-01-26 19:23:28.229 UTC [1] LOG: database system is ready to accept connections
[2025-01-26T19:23:28Z INFO service::utils::clean] Find directory "pg_vectors/indexes/17332".
[2025-01-26T19:23:28Z INFO service::utils::clean] Find directory "pg_vectors/indexes/17576".
[2025-01-26T19:23:28Z INFO service::utils::clean] Find directory "pg_vectors/indexes/17576/segments/ac8eb826-04ff-4046-99eb-f313d252539e".
[2025-01-26T19:23:28Z INFO service::utils::clean] Find directory "pg_vectors/indexes/17332/segments/6d9925e3-6c43-4c03-8b66-1627bcf1eb8b".
2025-01-26 19:24:31.725 UTC [1] LOG: server process (PID 109) was terminated by signal 7: Bus error
2025-01-26 19:24:31.725 UTC [1] DETAIL: Failed process was running: with "res" as (with "today" as (select make_date(year::int, $1::int, $2::int) as "date" from generate_series((select date_part('year', min(("localDateTime" at time zone 'UTC')::date))::int from assets), date_part('year', current_date)::int - 1) as "year") select "a"., to_json("exif") as "exifInfo" from "today" inner join lateral (select "assets". from "assets" inner join "asset_job_status" on "assets"."id" = "asset_job_status"."assetId" where "asset_job_status"."previewAt" is not null and (assets."localDateTime" at time zone 'UTC')::date = today.date and "assets"."ownerId" = any($3::uuid[]) and "assets"."isVisible" = $4 and "assets"."isArchived" = $5 and exists (select from "asset_files" where "assetId" = "assets"."id" and "asset_files"."type" = $6) and "assets"."deletedAt" is null limit $7) as "a" on true inner join "exif" on "a"."id" = "exif"."assetId") select ((now() at time zone 'UTC')::date - ("localDateTime" at time zone 'UTC')::date) / 365 as "yearsAgo", json_agg("res") as "assets" from "res" group
2025-01-26 19:24:31.726 UTC [1] LOG: terminating any other active server processes
2025-01-26 19:24:31.745 UTC [1] LOG: all server processes terminated; reinitializing
2025-01-26 19:24:32.017 UTC [110] LOG: database system was interrupted; last known up at 2025-01-26 19:23:28 UTC
2025-01-26 19:24:32.699 UTC [110] LOG: database system was not properly shut down; automatic recovery in progress
2025-01-26 19:24:32.787 UTC [110] LOG: redo starts at 2/7706D6E8
2025-01-26 19:24:32.787 UTC [110] LOG: invalid record length at 2/7706D720: wanted 24, got 0
2025-01-26 19:24:32.787 UTC [110] LOG: redo done at 2/7706D6E8 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
2025-01-26 19:24:33.440 UTC [1] LOG: database system is ready to accept connections

[gthelding]

To revert, change "IMMICH_VERSION=release" to "IMMICH_VERSION=v1.125.2" in your .env file and then execute docker compose pull and docker compose up -d

[swanepoelriaan]

I've reverted back to v1.125.2, but the errors are still present when opening the main "Photos" page. The other pages are still functioning as before. Log errors :

[Nest] 17 - 01/26/2025, 8:15:40 PM LOG [Api:Bootstrap] Immich Server is listening on http://[::1]:2283 [v1.125.2] [production]
[Nest] 17 - 01/26/2025, 8:15:41 PM LOG [Api:EventRepository] Websocket Connect: w67xmHNmgxadgL44AAAB
[Nest] 17 - 01/26/2025, 8:16:08 PM LOG [Api:EventRepository] Websocket Disconnect: w67xmHNmgxadgL44AAAB
[Nest] 17 - 01/26/2025, 8:16:09 PM LOG [Api:EventRepository] Websocket Connect: dIZn5Yk0HT41uYJUAAAD
Postgres notice: {
severity_local: 'WARNING',
severity: 'WARNING',
code: '57P02',
message: 'terminating connection because of crash of another server process',
detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
file: 'postgres.c',
line: '2888',
routine: 'quickdie'
}
Query failed : {
durationMs: 365.69261999999435,
error: Error: write CONNECTION_CLOSED database:5432
at Socket.closed (/usr/src/app/node_modules/postgres/cjs/src/connection.js:438:57)
at Socket.emit (node:events:524:28)
at TCP. (node:net:351:12)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
code: 'CONNECTION_CLOSED',
errno: 'CONNECTION_CLOSED',
address: [ 'database' ],
port: [ 5432 ]
},
sql: with "res" as (with "today" as (select make_date(year::int, $1::int, $2::int) as "date" from generate_series((select date_part('year', min(("localDateTime" at time zone 'UTC')::date))::int from assets), date_part('year', current_date)::int - 1) as "year") select "a".*, to_json("exif") as "exifInfo" from "today" inner join lateral (select "assets".* from "assets" inner join "asset_job_status" on "assets"."id" = "asset_job_status"."assetId" where "asset_job_status"."previewAt" is not null and (assets."localDateTime" at time zone 'UTC')::date = today.date and "assets"."ownerId" = any($3::uuid[]) and "assets"."isVisible" = $4 and "assets"."isArchived" = $5 and exists (select from "asset_files" where "assetId" = "assets"."id" and "asset_files"."type" = $6) and "assets"."deletedAt" is null limit $7) as "a" on true inner join "exif" on "a"."id" = "exif"."assetId") select ((now() at time zone 'UTC')::date - ("localDateTime" at time zone 'UTC')::date) / 365 as "yearsAgo", json_agg("res") as "assets" from "res" group by ("localDateTime" at time zone 'UTC')::date order by ("localDateTime" at time zone 'UTC')::date desc limit $8,
params: [
1,
26,
'{5a4c0e6c-bef9-4bb3-9662-8196b6de14e6}',
true,
false,
'preview',
20,
10
]
}
[Nest] 17 - 01/26/2025, 8:16:09 PM ERROR [Api:ErrorInterceptorhwpc348j] Unknown error: Error: write CONNECTION_CLOSED database:5432
Error: write CONNECTION_CLOSED database:5432
at Socket.closed (/usr/src/app/node_modules/postgres/cjs/src/connection.js:438:57)
at Socket.emit (node:events:524:28)
at TCP. (node:net:351:12)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
Postgres notice: {
severity_local: 'WARNING',
severity: 'WARNING',
code: '57P02',
message: 'terminating connection because of crash of another server process',
detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
file: 'postgres.c',
line: '2888',
routine: 'quickdie'
}
Postgres notice: {
severity_local: 'WARNING',
severity: 'WARNING',
code: '57P02',
message: 'terminating connection because of crash of another server process',
detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
file: 'postgres.c',
line: '2888',
routine: 'quickdie'
}
Postgres notice: {
severity_local: 'WARNING',
severity: 'WARNING',
code: '57P02',
message: 'terminating connection because of crash of another server process',
detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
file: 'postgres.c',
line: '2888',
routine: 'quickdie'
}
Postgres notice: {
severity_local: 'WARNING',
severity: 'WARNING',
code: '57P02',
message: 'terminating connection because of crash of another server process',
detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
file: 'postgres.c',
line: '2888',
routine: 'quickdie'
}
Query failed : {
durationMs: 303.1897339999996,
error: Error: write CONNECTION_CLOSED database:5432
at Socket.closed (/usr/src/app/node_modules/postgres/cjs/src/connection.js:438:57)
at Socket.emit (node:events:524:28)
at TCP. (node:net:351:12)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
code: 'CONNECTION_CLOSED',
errno: 'CONNECTION_CLOSED',
address: [ 'database' ],
port: [ 5432 ]
},
sql: select "assets".*, to_json("exif") as "exifInfo", to_json("stacked_assets") as "stack" from "assets" left join "exif" on "assets"."id" = "exif"."assetId" left join "asset_stack" on "asset_stack"."id" = "assets"."stackId" left join lateral (select "asset_stack".*, count("stacked") as "assetCount" from "assets" as "stacked" where "stacked"."stackId" = "asset_stack"."id" and "stacked"."deletedAt" is null and "stacked"."isArchived" = $1 group by "asset_stack"."id") as "stacked_assets" on "asset_stack"."id" is not null where "assets"."ownerId" = any($2::uuid[]) and "assets"."isArchived" = $3 and ("asset_stack"."primaryAssetId" = "assets"."id" or "assets"."stackId" is null) and "assets"."deletedAt" is null and "assets"."isVisible" = $4 and date_trunc($5, "localDateTime" at time zone 'UTC') at time zone 'UTC' = $6 order by "assets"."localDateTime" desc,
params: [
false,
'{5a4c0e6c-bef9-4bb3-9662-8196b6de14e6}',
false,
true,
'MONTH',
'2025-01-01T00:00:00.000Z'
]
}
[Nest] 17 - 01/26/2025, 8:16:09 PM ERROR [Api:ErrorInterceptorexd2tpyd] Unknown error: Error: write CONNECTION_CLOSED database:5432
Error: write CONNECTION_CLOSED database:5432
at Socket.closed (/usr/src/app/node_modules/postgres/cjs/src/connection.js:438:57)
at Socket.emit (node:events:524:28)
at TCP. (node:net:351:12)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
/usr/src/app/node_modules/postgres/cjs/src/connection.js:250
const x = socket.write(chunk, fn)
^

TypeError: Cannot read properties of null (reading 'write')
at Immediate.nextWrite (/usr/src/app/node_modules/postgres/cjs/src/connection.js:250:22)
at process.processImmediate (node:internal/timers:491:21)
at process.callbackTrampoline (node:internal/async_hooks:130:17)

Node.js v22.13.1
api worker exited with code 1

[bo0tzz]

There is some critical problem with your system or postgres installation.

[swanepoelriaan]

Was working perfectly before the update, didn't have any issues or problems. Only after updating to v1.125.3 did it start.

[swanepoelriaan]

Also, it is a fresh Ubuntu install done this weekend. System doesn't run anything other than the Immich-app and only has Ubuntu updates installed.

[alextran1502]

Hello, if you do a full restart docker compose down then docker compose up. Do you still see the issue?

[swanepoelriaan]

The Problem remains even after a full restart and server reboot:

immich_server            | [Nest] 17  - 01/26/2025, 8:36:17 PM     LOG [Api:Bootstrap] Immich Server is listening on http://[::1]:2283 [v1.125.2] [production]
immich_server            | [Nest] 17  - 01/26/2025, 8:38:08 PM     LOG [Api:EventRepository] Websocket Connect:    zCU_v8o_DJ10v0G9AAAB
immich_server            | Query failed : {
immich_server            |   durationMs: 317.0257779999956,
immich_server            |   error: Error: write CONNECTION_CLOSED database:5432
immich_server            |       at Socket.closed (/usr/src/app/node_modules/postgres/cjs/src/connection.js:438:57)
immich_server            |       at Socket.emit (node:events:524:28)
immich_server            |       at TCP.<anonymous> (node:net:351:12)
immich_server            |       at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
immich_server            |     code: 'CONNECTION_CLOSED',
immich_server            |     errno: 'CONNECTION_CLOSED',
immich_server            |     address: [ 'database' ],
immich_server            |     port: [ 5432 ]
immich_server            |   },
immich_server            |   sql: `with "res" as (with "today" as (select make_date(year::int, $1::int, $2::int) as "date" from generate_series((select date_part('year', min(("localDateTime" at time zone 'UTC')::date))::int from assets), date_part('year', current_date)::int - 1) as "year") select "a".*, to_json("exif") as "exifInfo" from "today" inner join lateral (select "assets".* from "assets" inner join "asset_job_status" on "assets"."id" = "asset_job_status"."assetId" where "asset_job_status"."previewAt" is not null and (assets."localDateTime" at time zone 'UTC')::date = today.date and "assets"."ownerId" = any($3::uuid[]) and "assets"."isVisible" = $4 and "assets"."isArchived" = $5 and exists (select from "asset_files" where "assetId" = "assets"."id" and "asset_files"."type" = $6) and "assets"."deletedAt" is null limit $7) as "a" on true inner join "exif" on "a"."id" = "exif"."assetId") select ((now() at time zone 'UTC')::date - ("localDateTime" at time zone 'UTC')::date) / 365 as "yearsAgo", json_agg("res") as "assets" from "res" group by ("localDateTime" at time zone 'UTC')::date order by ("localDateTime" at time zone 'UTC')::date desc limit $8`,
immich_server            |   params: [
immich_server            |     1,
immich_server            |     26,
immich_server            |     '{13a2b15f-b65f-4994-9e42-39e4a1d79460}',
immich_server            |     true,
immich_server            |     false,
immich_server            |     'preview',
immich_server            |     20,
immich_server            |     10
immich_server            |   ]
immich_server            | }
immich_server            | Postgres notice: {
immich_server            |   severity_local: 'WARNING',
immich_server            |   severity: 'WARNING',
immich_server            |   code: '57P02',
immich_server            |   message: 'terminating connection because of crash of another server process',
immich_server            |   detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
immich_server            |   hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
immich_server            |   file: 'postgres.c',
immich_server            |   line: '2888',
immich_server            |   routine: 'quickdie'
immich_server            | }
immich_server            | [Nest] 17  - 01/26/2025, 8:38:09 PM   ERROR [Api:ErrorInterceptor~u2eh7huv] Unknown error: Error: write CONNECTION_CLOSED database:5432
immich_server            | Error: write CONNECTION_CLOSED database:5432
immich_server            |     at Socket.closed (/usr/src/app/node_modules/postgres/cjs/src/connection.js:438:57)
immich_server            |     at Socket.emit (node:events:524:28)
immich_server            |     at TCP.<anonymous> (node:net:351:12)
immich_server            |     at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
immich_server            | Postgres notice: {
immich_server            |   severity_local: 'WARNING',
immich_server            |   severity: 'WARNING',
immich_server            |   code: '57P02',
immich_server            |   message: 'terminating connection because of crash of another server process',
immich_server            |   detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
immich_server            |   hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
immich_server            |   file: 'postgres.c',
immich_server            |   line: '2888',
immich_server            |   routine: 'quickdie'
immich_server            | }
immich_server            | Postgres notice: {
immich_server            |   severity_local: 'WARNING',
immich_server            |   severity: 'WARNING',
immich_server            |   code: '57P02',
immich_server            |   message: 'terminating connection because of crash of another server process',
immich_server            |   detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
immich_server            |   hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
immich_server            |   file: 'postgres.c',
immich_server            |   line: '2888',
immich_server            |   routine: 'quickdie'
immich_server            | }
immich_server            | Postgres notice: {
immich_server            |   severity_local: 'WARNING',
immich_server            |   severity: 'WARNING',
immich_server            |   code: '57P02',
immich_server            |   message: 'terminating connection because of crash of another server process',
immich_server            |   detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
immich_server            |   hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
immich_server            |   file: 'postgres.c',
immich_server            |   line: '2888',
immich_server            |   routine: 'quickdie'
immich_server            | }
immich_server            | Postgres notice: {
immich_server            |   severity_local: 'WARNING',
immich_server            |   severity: 'WARNING',
immich_server            |   code: '57P02',
immich_server            |   message: 'terminating connection because of crash of another server process',
immich_server            |   detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
immich_server            |   hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
immich_server            |   file: 'postgres.c',
immich_server            |   line: '2888',
immich_server            |   routine: 'quickdie'
immich_server            | }
immich_server            | Postgres notice: {
immich_server            |   severity_local: 'WARNING',
immich_server            |   severity: 'WARNING',
immich_server            |   code: '57P02',
immich_server            |   message: 'terminating connection because of crash of another server process',
immich_server            |   detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
immich_server            |   hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
immich_server            |   file: 'postgres.c',
immich_server            |   line: '2888',
immich_server            |   routine: 'quickdie'
immich_server            | }
immich_server            | Postgres notice: {
immich_server            |   severity_local: 'WARNING',
immich_server            |   severity: 'WARNING',
immich_server            |   code: '57P02',
immich_server            |   message: 'terminating connection because of crash of another server process',
immich_server            |   detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
immich_server            |   hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
immich_server            |   file: 'postgres.c',
immich_server            |   line: '2888',
immich_server            |   routine: 'quickdie'
immich_server            | }
immich_server            | Postgres notice: {
immich_server            |   severity_local: 'WARNING',
immich_server            |   severity: 'WARNING',
immich_server            |   code: '57P02',
immich_server            |   message: 'terminating connection because of crash of another server process',
immich_server            |   detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
immich_server            |   hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
immich_server            |   file: 'postgres.c',
immich_server            |   line: '2888',
immich_server            |   routine: 'quickdie'
immich_server            | }
immich_server            | Postgres notice: {
immich_server            |   severity_local: 'WARNING',
immich_server            |   severity: 'WARNING',
immich_server            |   code: '57P02',
immich_server            |   message: 'terminating connection because of crash of another server process',
immich_server            |   detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
immich_server            |   hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
immich_server            |   file: 'postgres.c',
immich_server            |   line: '2888',
immich_server            |   routine: 'quickdie'
immich_server            | }
immich_server            | Postgres notice: {
immich_server            |   severity_local: 'WARNING',
immich_server            |   severity: 'WARNING',
immich_server            |   code: '57P02',
immich_server            |   message: 'terminating connection because of crash of another server process',
immich_server            |   detail: 'The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.',
immich_server            |   hint: 'In a moment you should be able to reconnect to the database and repeat your command.',
immich_server            |   file: 'postgres.c',
immich_server            |   line: '2888',
immich_server            |   routine: 'quickdie'
immich_server            | }
immich_server            | Query failed : {
immich_server            |   durationMs: 322.1348080000025,
immich_server            |   error: Error: write CONNECTION_CLOSED database:5432
immich_server            |       at Socket.closed (/usr/src/app/node_modules/postgres/cjs/src/connection.js:438:57)
immich_server            |       at Socket.emit (node:events:524:28)
immich_server            |       at TCP.<anonymous> (node:net:351:12)
immich_server            |       at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
immich_server            |     code: 'CONNECTION_CLOSED',
immich_server            |     errno: 'CONNECTION_CLOSED',
immich_server            |     address: [ 'database' ],
immich_server            |     port: [ 5432 ]
immich_server            |   },
immich_server            |   sql: `with "assets" as (select date_trunc($1, "localDateTime" at time zone 'UTC') at time zone 'UTC' as "timeBucket" from "assets" left join "asset_stack" on "asset_stack"."id" = "assets"."stackId" and "asset_stack"."primaryAssetId" = "assets"."id" where "assets"."deletedAt" is null and "assets"."isVisible" = $2 and ("assets"."stackId" is null or "asset_stack" is not null) and "assets"."ownerId" = any($3::uuid[]) and "assets"."isArchived" = $4) select "timeBucket", count(*) as "count" from "assets" group by "timeBucket" order by "timeBucket" desc`,
immich_server            |   params: [ 'MONTH', true, '{13a2b15f-b65f-4994-9e42-39e4a1d79460}', false ]
immich_server            | }
immich_server            | [Nest] 17  - 01/26/2025, 8:38:09 PM   ERROR [Api:ErrorInterceptor~kea9jpxc] Unknown error: Error: write CONNECTION_CLOSED database:5432
immich_server            | Error: write CONNECTION_CLOSED database:5432
immich_server            |     at Socket.closed (/usr/src/app/node_modules/postgres/cjs/src/connection.js:438:57)
immich_server            |     at Socket.emit (node:events:524:28)
immich_server            |     at TCP.<anonymous> (node:net:351:12)
immich_server            |     at TCP.callbackTrampoline (node:internal/async_hooks:130:17)

[alextran1502]

Thanks, we will investigate.

@mertalev This error message stood out

immich_server            | Query failed : {
immich_server            |   durationMs: 322.1348080000025,
immich_server            |   error: Error: write CONNECTION_CLOSED database:5432
immich_server            |       at Socket.closed (/usr/src/app/node_modules/postgres/cjs/src/connection.js:438:57)
immich_server            |       at Socket.emit (node:events:524:28)
immich_server            |       at TCP.<anonymous> (node:net:351:12)
immich_server            |       at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
immich_server            |     code: 'CONNECTION_CLOSED',
immich_server            |     errno: 'CONNECTION_CLOSED',
immich_server            |     address: [ 'database' ],
immich_server            |     port: [ 5432 ]
immich_server            |   },
immich_server            |   sql: `with "assets" as (select date_trunc($1, "localDateTime" at time zone 'UTC') at time zone 'UTC' as "timeBucket" from "assets" left join "asset_stack" on "asset_stack"."id" = "assets"."stackId" and "asset_stack"."primaryAssetId" = "assets"."id" where "assets"."deletedAt" is null and "assets"."isVisible" = $2 and ("assets"."stackId" is null or "asset_stack" is not null) and "assets"."ownerId" = any($3::uuid[]) and "assets"."isArchived" = $4) select "timeBucket", count(*) as "count" from "assets" group by "timeBucket" order by "timeBucket" desc`,
immich_server            |   params: [ 'MONTH', true, '{13a2b15f-b65f-4994-9e42-39e4a1d79460}', false ]
immich_server            | }

[swanepoelriaan]

Thank you, as mentioned, the other pages seem to be working fine. I can still see people/places I've added and open photos through there. Only the main photos page gives the error.

[swanepoelriaan]

[alextran1502]

Can you access the database and try sorting the fileCreatedAt column in the assets table to see if you can spot any weirdness about the time, like very odd year?

[swanepoelriaan]

I can try, unfortunately my postgres command knowledge is very limited.

[swanepoelriaan]

[swanepoelriaan]

Can't seem to get into the postgres tables to see if there is any anomalies, the year range on the page seems ok, Caught the first error when opening the Photos page. But still the same issue. All the other pages seems fine.

[alextran1502]

If you don't mind, you can hope on discord and ping me. We can go through this together