Failed to upgrade to 1.92.3~ynh1

[tio-trom]

ended_at: 2023-09-28 20:36:39.425090
env:
  BASH_XTRACEFD: '7'
  NO_BACKUP_UPGRADE: '1'
  YNH_APP_ACTION: upgrade
  YNH_APP_BASEDIR: /var/cache/yunohost/app_tmp_work_dirs/app_fzykotli
  YNH_APP_CURRENT_VERSION: 1.91.0~ynh1
  YNH_APP_ID: synapse
  YNH_APP_INSTANCE_NAME: synapse
  YNH_APP_INSTANCE_NUMBER: '1'
  YNH_APP_MANIFEST_VERSION: 1.92.3~ynh1
  YNH_APP_PACKAGING_FORMAT: '1.0'
  YNH_APP_UPGRADE_TYPE: UPGRADE_APP
  YNH_ARCH: amd64
  YNH_CWD: /var/cache/yunohost/app_tmp_work_dirs/app_fzykotli/scripts
  YNH_DEBIAN_VERSION: bullseye
  YNH_INTERFACE: cli
  YNH_STDRETURN: /tmp/tmp6gn89a9g/stdreturn
error: An error occurred inside the app upgrade script
interface: cli
operation: app_upgrade
parent: null
related_to:
- - app
  - synapse
started_at: 2023-09-28 20:36:10.319537
success: false
yunohost_version: 11.2.4

Full log here https://paste.yunohost.org/raw/satajenofu

[hieronymousch]

same here and AGAIN it removes everything after a failed upgrade and is unable to restore. I really don't understand why it deletes everything.

[hieronymousch]

Temporary fix, based on https://forum.yunohost.org/t/solved-synapse-update-failed-restore-not-possible/22888/5

cd /home/yunohost.backup/archives/
ls -l synapse-pre-upgrade* # This is to double check the name of the backup, adjust the file name accordingly later, and take the latest
cp synapse-pre-upgrade2.tar synapse-pre-upgrade2.tar.bak # Make a backup of the existing backup before modifying, just in case
tar --delete --file synapse-pre-upgrade2.tar apps/synapse/backup/opt/yunohost/matrix-synapse/lib/python3.9/site-packages/setuptools_rust apps/synapse/backup/opt/yunohost/matrix-synapse/lib/python3.9/site-packages/setuptools_rust-1.7.0.dist-info
yunohost backup restore synapse-pre-upgrade2

Not sure why this rust python became mandatory, as I understood things, this is not really necessary for Synapse to run.

I was able to successfully restore the backup after this.

[hieronymousch]

Upgrade to testing (1.93) worked fine
sudo yunohost app upgrade synapse -u https://github.com/YunoHost-Apps/synapse_ynh/tree/testing --debug

[tio-trom]

Ok thank you very much for the info. So the testing works then? I cannot try it these days but as soon as I will I can close this issue.

[hieronymousch]

To be clear, did a restore as shown above and then upgraded to testing

[MayeulC]

Thanks a lot @hieronymousch, I had the same issue, your fix worked. I just added --app synapse to the last command: yunohost backup restore synapse-pre-upgrade2 --app synapse.

[tio-trom]

Why do you have to also add --app synapse at the end?

[MayeulC]

Oh, it's not needed, it's just because yunohost backups can contain multiple apps, and the restore command otherwise shows a scary warning.

Restoring the database took about 4 hours and 15 minutes for my 40 GiB backup, I think a lot of that time was spent on index creation. As I don't want to wait such a long time again in case of failure, I'll also upgrade to testing.

Edit: BTW, you can keep an eye on the restore process by running sudo -u postgres psql -c "SELECT pg_database_size('matrix_synapse') ;" or maybe SELECT pg_size_pretty( pg_database_size('matrix_synapse') ); if you want a prettier output. My 40 GiB backup expanded to about 70 GiB.

[hieronymousch]

Oh, it's not needed, it's just because yunohost backups can contain multiple apps, and the restore command otherwise shows a scary warning.

Restoring the database took about 4 hours and 15 minutes for my 40 GiB backup, I think a lot of that time was spent on index creation. As I don't want to wait such a long time again in case of failure, I'll also upgrade to testing.

Thats a huge DB. I hope it's not for a familiy server because some cleaning might then be in order ;) (I'm at 3.8 GB for small familiy server but 5+ years of use)

[tio-trom]

My database was 260GB after 2 years of use and 500 users. I managed to optimize it and bring it down to 100GB. Read more here as to why this is so big and how I managed to make it much smaller #354

So yeah for me the restore process even backup, is not feasible. I usually do a server snapshot before updating Synapse without backing it up, then if it fails I restore a snapshot for the entire server....else it is not manageable.

[hieronymousch]

I got pretty paranoid after the upgrade to bullseye and now I have a remote backup running with borg (iterative backup) and at the same time pgbackrest for incremental DB commits as well... but luckily not the volume you talk about... interesting stuff ;)

[tio-trom]

Ok so now I finally tested it. My database is actually 60GB and files around 15GB.

Restoring the backup worked thanks to the #414 (comment) comment. Backing up now takes around 25-30 minutes. Restoring takes around 40-50 minutes.

Upgrading to testing also works with sudo yunohost app upgrade synapse -u https://github.com/YunoHost-Apps/synapse_ynh/tree/testing --debug and that took around 30 minutes with backup.

It is not terrible but it is something you have to do via the terminal. In case Matrix fails, then you will have to spend around 1h restoring it, if your instance is of a similar size.

I hope they will push the testing into the master soon to fix this terrible issue.