0.23.0-RC9 eventually hanging

[tocklime]

I have a folder on my NAS with about 50000 photo, video, and related files (I think maybe half of them are 'something that should be uploaded to immich'. There is at least one unzipped old google takeout in there, as well as dead picasa libraries and possibly even some iPhoto stuff.

I'm running 0.23.0-rc9 built with this nix flake (modified from the one in nixpkgs):

flake.nix

{
  inputs = {
    utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, utils }: utils.lib.eachDefaultSystem (system:
    let
      pkgs = nixpkgs.legacyPackages.${system};
      my_ig = pkgs.immich-go.overrideAttrs (prev: rec {
          inherit (prev) pname;
          version = "0.23.0-RC9";
          src = pkgs.fetchFromGitHub {
            owner = "simulot";
            repo = "immich-go";
            rev = "v${version}";
            hash = "sha256-MLd3paooyq7A5haO/vM66y/dqjJvbqlmXiEPhUTEmWM=";
            leaveDotGit = true;
            postFetch = ''
              cd "$out"
              git log -1 --pretty=%H > "COMMIT"
              git log -1 --pretty=%cd --date=format:'%Y-%m-%dT%H:%M:%SZ' > "SOURCE_DATE"
              rm -rf ".git"
            '';
          };
          vendorHash = "sha256-P55dVLf8z+GmoWVGUBEjX+rSeRYr8OrAQYiecsmIcnI=";
          doCheck = false;
      });

    in
    {
      devShell = pkgs.mkShell {
        buildInputs = with pkgs; [
          my_ig
        ];
      };
    }
  );
}

When I run immich-go, it goes for quite a while, uploading about 10000 assets, but then it gets to a point where it does nothing. There have been no new log lines for over an hour (the most recent are about 'scanned image file'). There are no in progress jobs on the server. immich-go is still running (I assume it exits when it finishes).

When I press Ctrl-C, I get a lot (depending on the run, 100-700 ish) of ERR context canceled in the logs, and it exits.

I can rerun, and it spins through a lot of "Server has same quality", but eventually grinds to a halt after about 15000 files.

I'm getting occasional server errors (1 or 2 each run), but not sure if that's relevant.

Happy to provide logs and api traces, but would rather that wasn't public.

my immich-go command is

immich-go upload from-folder --manage-burst Stack --manage-heic-jpeg StackCoverHeic --manage-raw-jpeg StackCoverJPG --tag "{immich-go}/2025-01-22 20:55:02" /tank/Pictures --server $SERVER_URL -k $API_KEY --api-trace

(i started with -session-tag for the first run, but now using --tag to avoid API churn).

[tocklime]

...now running with --ban-file ThatOneUnzippedGoogleTakeout/ and it seems to be getting further without stalling.

[simulot]

Happy to provide logs and api traces, but would rather that wasn't public.

You can share them on my discord @simulot.

When I press Ctrl-C, I get a lot (depending on the run, 100-700 ish) of ERR context canceled in the logs, and it exits.

This is a clue... --from-folder tries to leverage goroutines (the go's multi-thread mechanism). Ctrl-C make all running routines to die with the context cancelled message. But, there is a limitation in the number of threads that should prevent this...

...now running with --ban-file ThatOneUnzippedGoogleTakeout/ and it seems to be getting further without stalling.

Is this folder symlinked or hadlinked to the /tank/Pictures? Maybe a ZFS special thing? A cycle?
If you have preserved the takeout content with all JSON, you may use import from-google-photos. It will import the photos with google features like albums and descriptions.

(i started with -session-tag for the first run, but now using --tag to avoid API churn).

What do you mean by to avoid API churn?
I'm working on a better tag assignment

[tocklime]

Happy to provide logs and api traces, but would rather that wasn't public.

You can share them on my discord @simulot.

I'll get something zipped up in a minute.

When I press Ctrl-C, I get a lot (depending on the run, 100-700 ish) of ERR context canceled in the logs, and it exits.

This is a clue... --from-folder tries to leverage goroutines (the go's multi-thread mechanism). Ctrl-C make all running routines to die with the context cancelled message. But, there is a limitation in the number of threads that should prevent this...

...now running with --ban-file ThatOneUnzippedGoogleTakeout/ and it seems to be getting further without stalling.

Is this folder symlinked or hadlinked to the /tank/Pictures? Maybe a ZFS special thing? A cycle? If you have preserved the takeout content with all JSON, you may use import from-google-photos. It will import the photos with google features like albums and descriptions.

It is ZFS, but all in one dataset. No symlinks. The run with --ban-file completed, as did a separate run with immich-go upload from-google-photos for just the google takeout directory, so I think I have everything imported I need to import now.

(i started with -session-tag for the first run, but now using --tag to avoid API churn).

What do you mean by to avoid API churn? I'm working on a better tag assignment

When I did my 2nd run of immich-go upload from-folder, it had a lot of photos that the server already had, but immich-go wanted to tag them again. It did so, which seemed to trigger a lot of jobs on the server. In any case, I want one tag for this whole upload, not one per upload attempt. I'm unsure if immich-go should try to tag an asset that it's not uploaded because the server already has it. I can make the argument both ways...

[simulot]

the session tag is useful to check what has been touched by immich-go.
It's easy to remove the tag, or delete photos just imported