Loop in 'Connection reset by peer.' and fails

[SingleCellW]

What's your browser and operating system? (Copy/paste the output of https://www.whatismybrowser.com/)

Chrome 87 on Windows 10

What is the problem?

The following code is the output log in bash which repeat hundreds of times with 'celery.beat.SchedulingError: Couldn't apply scheduled task process-pending-tasks: Error 104 while writing to socket. Connection reset by peer.' as ending. At first I thought the limit of memory trigger the error but after reading the documents I thought 128GB RAM would work fine.

broker | 1:M 11 Jan 2021 05:42:24.014 * 100 changes in 300 seconds. Saving...
broker | 1:M 11 Jan 2021 05:42:24.019 * Background saving started by pid 1979
broker | 1979:C 11 Jan 2021 05:42:24.430 * DB saved on disk
broker | 1979:C 11 Jan 2021 05:42:24.433 * RDB: 14 MB of memory used by copy-on-write
broker | 1:M 11 Jan 2021 05:42:24.525 * Background saving terminated with success
broker | 1:M 11 Jan 2021 05:43:28.777 # Client id=4368 addr=172.18.0.6:39000 fd=23 name= age=67 idle=0 flags=P db=0 sub=145004 psub=0 multi=-1 qbuf=79 qbuf-free=32689 argv-mem=62 obl=0 oll=596 omem=12220384 tot-mem=12281902 events=rw cmd=subscribe user=default scheduled to be closed ASAP for overcoming of output buffer limits.
webapp | [2021-01-11 05:43:29,089: ERROR/MainProcess] Message Error: Couldn't apply scheduled task process-pending-tasks: Error 104 while writing to socket. Connection reset by peer.
webapp | [' File "/usr/local/bin/celery", line 8, in \n sys.exit(main())\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/main.py", line 16, in main\n _main()\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/bin/celery.py", line 322, in main\n cmd.execute_from_commandline(argv)\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/bin/celery.py", line 495, in execute_from_commandline\n super(CeleryCommand, self).execute_from_commandline(argv)))\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/bin/base.py", line 305, in execute_from_commandline\n return self.handle_argv(self.prog_name, argv[1:])\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/bin/celery.py", line 487, in handle_argv\n return self.execute(command, argv)\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/bin/celery.py", line 419, in execute\n ).run_from_argv(self.prog_name, argv[1:], command=argv[0])\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/bin/base.py", line 309, in run_from_argv\n sys.argv if argv is None else argv, command)\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/bin/base.py", line 393, in handle_argv\n return self(*args, **options)\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/bin/base.py", line 253, in call\n ret = self.run(*args, **kwargs)\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/bin/beat.py", line 109, in run\n return beat().run()\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/apps/beat.py", line 81, in run\n self.start_scheduler()\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/apps/beat.py", line 109, in start_scheduler\n service.start()\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/beat.py", line 630, in start\n interval = self.scheduler.tick()\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/beat.py", line 343, in tick\n self.apply_entry(entry, producer=self.producer)\n', ' File "/usr/local/lib/python3.6/dist-packages/celery/beat.py", line 276, in apply_entry\n exc, traceback.format_stack(), exc_info=True)\n']
webapp | Traceback (most recent call last):
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 600, in send_packed_command
webapp | self._sock.sendall(item)
webapp | ConnectionResetError: [Errno 104] Connection reset by peer
webapp |
webapp | During handling of the above exception, another exception occurred:
webapp |
webapp | Traceback (most recent call last):
webapp | File "/usr/local/lib/python3.6/dist-packages/kombu/connection.py", line 439, in _reraise_as_library_errors
webapp | yield
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/app/base.py", line 770, in send_task
webapp | self.backend.on_task_call(P, task_id)
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/backends/redis.py", line 301, in on_task_call
webapp | self.result_consumer.consume_from(task_id)
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/backends/redis.py", line 136, in consume_from
webapp | self._consume_from(task_id)
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/backends/redis.py", line 142, in _consume_from
webapp | self._pubsub.subscribe(key)
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 3096, in subscribe
webapp | ret_val = self.execute_command('SUBSCRIBE', *iterkeys(new_channels))
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 3009, in execute_command
webapp | self._execute(connection, connection.send_command, *args)
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 3013, in _execute
webapp | return command(*args)
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 620, in send_command
webapp | self.send_packed_command(self.pack_command(*args))
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 613, in send_packed_command
webapp | (errno, errmsg))
webapp | redis.exceptions.ConnectionError: Error 104 while writing to socket. Connection reset by peer.
webapp |
webapp | During handling of the above exception, another exception occurred:
webapp |
webapp | Traceback (most recent call last):
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/beat.py", line 392, in apply_async
webapp | **entry.options)
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/app/task.py", line 568, in apply_async
webapp | **options
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/app/base.py", line 771, in send_task
webapp | amqp.send_task_message(P, name, message, **options)
webapp | File "/usr/lib/python3.6/contextlib.py", line 99, in exit
webapp | self.gen.throw(type, value, traceback)
webapp | File "/usr/local/lib/python3.6/dist-packages/kombu/connection.py", line 444, in _reraise_as_library_errors
webapp | sys.exc_info()[2])
webapp | File "/usr/local/lib/python3.6/dist-packages/vine/five.py", line 194, in reraise
webapp | raise value.with_traceback(tb)
webapp | File "/usr/local/lib/python3.6/dist-packages/kombu/connection.py", line 439, in _reraise_as_library_errors
webapp | yield
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/app/base.py", line 770, in send_task
webapp | self.backend.on_task_call(P, task_id)
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/backends/redis.py", line 301, in on_task_call
webapp | self.result_consumer.consume_from(task_id)
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/backends/redis.py", line 136, in consume_from
webapp | self._consume_from(task_id)
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/backends/redis.py", line 142, in _consume_from
webapp | self._pubsub.subscribe(key)
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 3096, in subscribe
webapp | ret_val = self.execute_command('SUBSCRIBE', *iterkeys(new_channels))
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 3009, in execute_command
webapp | self._execute(connection, connection.send_command, *args)
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 3013, in _execute
webapp | return command(*args)
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 620, in send_command
webapp | self.send_packed_command(self.pack_command(*args))
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 613, in send_packed_command
webapp | (errno, errmsg))
webapp | kombu.exceptions.OperationalError: Error 104 while writing to socket. Connection reset by peer.
webapp |
webapp | During handling of the above exception, another exception occurred:
webapp |
webapp | Traceback (most recent call last):
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/beat.py", line 273, in apply_entry
webapp | result = self.apply_async(entry, producer=producer, advance=False)
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/beat.py", line 400, in apply_async
webapp | entry, exc=exc)), sys.exc_info()[2])
webapp | File "/usr/local/lib/python3.6/dist-packages/vine/five.py", line 194, in reraise
webapp | raise value.with_traceback(tb)
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/beat.py", line 392, in apply_async
webapp | **entry.options)
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/app/task.py", line 568, in apply_async
webapp | **options
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/app/base.py", line 771, in send_task
webapp | amqp.send_task_message(P, name, message, **options)
webapp | File "/usr/lib/python3.6/contextlib.py", line 99, in exit
webapp | self.gen.throw(type, value, traceback)
webapp | File "/usr/local/lib/python3.6/dist-packages/kombu/connection.py", line 444, in _reraise_as_library_errors
webapp | sys.exc_info()[2])
webapp | File "/usr/local/lib/python3.6/dist-packages/vine/five.py", line 194, in reraise
webapp | raise value.with_traceback(tb)
webapp | File "/usr/local/lib/python3.6/dist-packages/kombu/connection.py", line 439, in _reraise_as_library_errors
webapp | yield
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/app/base.py", line 770, in send_task
webapp | self.backend.on_task_call(P, task_id)
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/backends/redis.py", line 301, in on_task_call
webapp | self.result_consumer.consume_from(task_id)
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/backends/redis.py", line 136, in consume_from
webapp | self._consume_from(task_id)
webapp | File "/usr/local/lib/python3.6/dist-packages/celery/backends/redis.py", line 142, in _consume_from
webapp | self._pubsub.subscribe(key)
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 3096, in subscribe
webapp | ret_val = self.execute_command('SUBSCRIBE', *iterkeys(new_channels))
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 3009, in execute_command
webapp | self._execute(connection, connection.send_command, *args)
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 3013, in _execute
webapp | return command(*args)
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 620, in send_command
webapp | self.send_packed_command(self.pack_command(*args))
webapp | File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 613, in send_packed_command
webapp | (errno, errmsg))
webapp | celery.beat.SchedulingError: Couldn't apply scheduled task process-pending-tasks: Error 104 while writing to socket. Connection reset by peer.

What should be the expected behavior?

Maybe the result should be generated in time? I'm so confused that I cant find similar mistakes in WebODM issues, only in some other celery-once projects.

How can we reproduce this? (What steps did you do to trigger the problem? What parameters are you using for processing? Please include multiple screenshots of the problem! If possible please include a copy of your dataset uploaded on Google Drive or Dropbox. Be detailed)

I use 894 images with 4096*2160 resolution (trials resizing to 1000 pixels fails) and finish finding and matching features in 1 day and stuck in 'Connection reset by peer.' loops in 3 more days. The parameters should be 'Options: mesh-octree-depth: 10, mesh-size: 300000, depthmap-resolution: 1000' and the total storage of the images is 2.38GB.