Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New S3 Configuration breaks prior S3 configuration and is missing uploadPartSize #612

Open
acidicX opened this issue Jul 29, 2024 · 9 comments · May be fixed by #614
Open

New S3 Configuration breaks prior S3 configuration and is missing uploadPartSize #612

acidicX opened this issue Jul 29, 2024 · 9 comments · May be fixed by #614
Assignees
Labels
bug Something isn't working S3 Anything to do with S3 object storage

Comments

@acidicX
Copy link

acidicX commented Jul 29, 2024

Describe your Issue

I've used the S3 integration before "manually" via configs.s3.config.php, now the helm chart upgrade (v5.4.0) fails because it's already defined

Logs and Errors

Helm upgrade failed: error while running post render on files: map[string]interface {}(nil): yaml: unmarshal errors:
line 168: mapping key "s3.config.php" already defined at line 23

Describe your Environment

  • Kubernetes distribution: k8s

  • Helm Version (or App that manages helm):

  • Helm Chart Version: 5.4.0

  • values.yaml:

configs:
        s3.config.php: |-
          <?php
          $CONFIG = array (
            'objectstore' => array(
              'class' => '\\OC\\Files\\ObjectStore\\S3',
              'arguments' => array(
                'hostname'         => 's3.eu-central-2.wasabisys.com',
                'bucket'           => '...',
                'autocreate'       => false,
                'key'              => '...',
                'secret'           => '...',
                'region'           => 'eu-central-2',
                'use_ssl'          => true,
                'use_path_style'   => false,
                'uploadPartSize'   => '2097152000'
              )
            )
          );

Additional context, if any

  1. It would be nice if such a change would be marked as a breaking change next time
  2. The chart should respect the manually set configuration
  3. The new options are missing uploadPartSize, which is required to be set there for some reason on S3 storage, so I can't even switch to the new configuration style
@provokateurin
Copy link
Member

Sorry for this breaking change :/ I think your config should continue to work fine by just renaming the config file.
Could you open a PR for the uploadPartSize so that it is present and allows you to switch to the new configs?

@provokateurin provokateurin added bug Something isn't working S3 Anything to do with S3 object storage labels Jul 29, 2024
@acidicX
Copy link
Author

acidicX commented Jul 29, 2024

@provokateurin thanks for the quick response, I can try to open a PR but I'm no expert in helm charts and I can't test the change locally at the moment.

@jessebot
Copy link
Collaborator

jessebot commented Jul 29, 2024

🙇 sorry about that, @acidicX!

I actually submitted nextcloud/docker#2271 upstream for these changes, but I'll submit one here as well now!

@jessebot
Copy link
Collaborator

jessebot commented Jul 29, 2024

In the meantime, you should be able to disable the default config being deployed by setting nextcloud.defaultConfigs.s3.config.php=false, or via values.yaml:

nextcloud:
  defaultConfigs:
    s3.config.php: false

Can you confirm if this works? Because if that doesn't work either, we should also try to fix that. Sorry again for the inconvenience regardless 🙏

@jessebot jessebot linked a pull request Jul 29, 2024 that will close this issue
4 tasks
@jessebot
Copy link
Collaborator

jessebot commented Jul 29, 2024

Sorry for the rapid updates, but if #614 gets merged, it will close this Issue automatically. That doesn't mean we can't still help you if other things are still broken here though, so please just let us know if that's the case and we can reopen this issue for you if it gets closed 🙏

Update: We're going to wait till nextcloud/docker reviews/merges our upstream PR, nextcloud/docker#2271 before merging #614

@acidicX
Copy link
Author

acidicX commented Jul 29, 2024

@jessebot thanks for the fast reaction, I can confirm that the workaround (nextcloud.defaultConfigs.s3.config.php=false) works for me :) I've been able to upgrade and everything seems to work fine.

@MaksonHerson
Copy link

MaksonHerson commented Aug 9, 2024

Hello @jessebot , @acidicX !
I got the same trouble with already defined mapping key for s3.config.php file.
I used:

nextcloud:
  defaultConfigs:
    s3.config.php: false

and it does not work for me. I try for 5.4.0 and 5.5.2 chart versions.

I was able to fix it with renaming my s3 config file to:

configs:
        s33.config.php:

Unfortunately it still needs to be fix. :(

Essential part of my helm release file:

    defaultConfigs:
      s3.config.php: false
      swift.config.php: false
    objectStore:
      s3:
        enabled: true
        ssl: true
        port: ***
        region: ***
        usePathStyle: true
        autoCreate: true
        existingSecret: nextcloud-values
        secretKeys:
          host: hostname
          accessKey: key
          secretKey: secret
          bucket: bucket

@jessebot
Copy link
Collaborator

@MaksonHerson I'm sorry about that. Could you check and see if s3.config.php is actually already in your config directory? Perhaps you just need to delete it? 🤔 Either way, let me know and I'm happy to keep troubleshooting it with you.

@jessebot jessebot self-assigned this Sep 25, 2024
@jstewart612
Copy link

@jessebot this now does not work after #464 with sse_c_key specified or not. I tried both:

level":3,"time":"2024-10-06T23:05:35+00:00","remoteAddr":"10.42.7.135","user":"jstewart","app":"objectstore","method":"GET","url":"/avatar/jstewart/64/dark?v=17282
55935126","message":"{\\"Exception\\":\\"Exception\\",\\"Message\\":\\"Creation of bucket \\\\\\"nextcloud\\\\\\" failed. Error executing \\\\\\"CreateBucket\\\\\\
" on \\\\\\"https://hl15-1.home.thecrimsontint.com:19000/nextcloud/\\\\\\"; AWS HTTP error: Client error: `PUT https://hl15-1.home.thecrimsontint.com:19000/nextclo
ud/` resulted in a `403 Forbidden` response:\\\\n<?xml version=\\\\\\"1.0\\\\\\" encoding=\\\\\\"UTF-8\\\\\\"?>\\\\n<Error><Code>SignatureDoesNotMatch</Code><Messa
ge>The request signature we calcul (truncated...)\\\\n SignatureDoesNotMatch (client): The request signature we calculated does not match the signature you provide
d. Check your key and signing method. - <?xml version=\\\\\\"1.0\\\\\\" encoding=\\\\\\"UTF-8\\\\\\"?>\\\\n<Error><Code>SignatureDoesNotMatch</Code><Message>The re
quest signature we calculated does not match the signature you provided. Check your key and signing method.</Message><BucketName>nextcloud</BucketName><Resource>/n
extcloud/</Resource><RequestId>17FBFF9DE0874B12</RequestId><HostId>470a0afc0f6fdbebb6a8e14ece29122784c0656e19563387135a7c8e02feb163</HostId></Error>\\",\\"Code\\":
0,\\"Trace\\":[{\\"file\\":\\"/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php\\",\\"line\\":90,\\"function\\":\\"getConnection\\",\\"class\\":\\"OC\\
\\\\\\Files\\\\\\\\ObjectStore\\\\\\\\S3\\",\\"type\\":\\"->\\",\\"args\\":[]},{\\"file\\":\\"/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php\\",\\"l
ine\\":152,\\"function\\":\\"writeSingle\\",\\"class\\":\\"OC\\\\\\\\Files\\\\\\\\ObjectStore\\\\\\\\S3\\",\\"type\\":\\"->\\",\\"args\\":[\\"urn:oid:20\\",[\\"Guz
zleHttp\\\\\\\\Psr7\\\\\\\\Stream\\"],\\"image/png\\"]},{\\"file\\":\\"/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php\\",\\"line\\":520,\\"func
tion\\":\\"writeObject\\",\\"class\\":\\"OC\\\\\\\\Files\\\\\\\\ObjectStore\\\\\\\\S3\\",\\"type\\":\\"->\\",\\"args\\":[\\"urn:oid:20\\",null,\\"image/png\\"]},{\
\"file\\":\\"/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php\\",\\"line\\":446,\\"function\\":\\"writeStream\\",\\"class\\":\\"OC\\\\\\\\Files\\
\\\\\\ObjectStore\\\\\\\\ObjectStoreStorage\\",\\"type\\":\\"->\\",\\"args\\":[\\"appdata_ocy4keitmxg4/avatar/jstewart/avatar-dark.png\\",null,9351]},{\\"file\\":\
\"/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php\\",\\"line\\":354,\\"function\\":\\"writeBack\\",\\"class\\":\\"OC\\\\\\\\Files\\\\\\\\ObjectS
tore\\\\\\\\ObjectStoreStorage\\",\\"type\\":\\"->\\",\\"args\\":[\\"/tmp/oc_tmp_5ZfU10-.png\\",\\"appdata_ocy4keitmxg4/avatar/jstewart/avatar-dark.png\\"]},{\\"fu
nction\\":\\"OC\\\\\\\\Files\\\\\\\\ObjectStore\\\\\\\\{closure}\\",\\"class\\":\\"OC\\\\\\\\Files\\\\\\\\ObjectStore\\\\\\\\ObjectStoreStorage\\",\\"type\\":\\"->
\\",\\"args\\":[\\"*** sensitive parameters replaced ***\\"]},{\\"file\\":\\"/var/www/html/3rdparty/icewind/streams/src/CallbackWrapper.php\\",\\"line\\":119,\\"fu
nction\\":\\"call_user_func\\",\\"args\\":[[\\"Closure\\"]]},{\\"function\\":\\"stream_close\\",\\"class\\":\\"Icewind\\\\\\\\Streams\\\\\\\\CallbackWrapper\\",\\"
type\\":\\"->\\",\\"args\\":[\\"*** sensitive parameters replaced ***\\"]},{\\"file\\":\\"/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php\\",\\"
line\\":470,\\"function\\":\\"fclose\\",\\"args\\":[\\"*** sensitive parameters replaced ***\\"]},{\\"file\\":\\"/var/www/html/lib/private/Files/Storage/Wrapper/Wr
apper.php\\",\\"line\\":240,\\"function\\":\\"file_put_contents\\",\\"class\\":\\"OC\\\\\\\\Files\\\\\\\\ObjectStore\\\\\\\\ObjectStoreStorage\\",\\"type\\":\\"->\
\",\\"args\\":[\\"appdata_ocy4keitmxg4/avatar/jstewart/avatar-dark.png\\",null]},{\\"file\\":\\"/var/www/html/lib/private/Files/View.php\\",\\"line\\":1134,\\"func
tion\\":\\"file_put_contents\\",\\"class\\":\\"OC\\\\\\\\Files\\\\\\\\Storage\\\\\\\\Wrapper\\\\\\\\Wrapper\\",\\"type\\":\\"->\\",\\"args\\":[\\"appdata_ocy4keitm
xg4/avatar/jstewart/avatar-dark.png\\",null]},{\\"file\\":\\"/var/www/html/lib/private/Files/View.php\\",\\"line\\":644,\\"function\\":\\"basicOperation\\",\\"clas
s\\":\\"OC\\\\\\\\Files\\\\\\\\View\\",\\"type\\":\\"->\\",\\"args\\":[\\"file_put_contents\\",\\"/appdata_ocy4keitmxg4/avatar/jstewart/avatar-dark.png\\",[\\"crea
te\\",\\"write\\"],null]},{\\"file\\":\\"/var/www/html/lib/private/Files/Node/Folder.php\\",\\"line\\":167,\\"function\\":\\"file_put_contents\\",\\"class\\":\\"OC
\\\\\\\\Files\\\\\\\\View\\",\\"type\\":\\"->\\",\\"args\\":[\\"/appdata_ocy4keitmxg4/avatar/jstewart/avatar-dark.png\\",null]},{\\"file\\":\\"/var/www/html/lib/pr
ivate/Files/SimpleFS/NewSimpleFile.php\\",\\"line\\":103,\\"function\\":\\"newFile\\",\\"class\\":\\"OC\\\\\\\\Files\\\\\\\\Node\\\\\\\\Folder\\",\\"type\\":\\"->\\",\\"args\\":[\\"avatar-dark.png\\",null]},{\\"file\\":\\"/var/www/html/lib/private/Avatar/UserAvatar.php\\",\\"line\\":208,\\"function\\":\\"putContent\\",\\"cla
ss\\":\\"OC\\\\\\\\Files\\\\\\\\SimpleFS\\\\\\\\NewSimpleFile\\",\\"type\\":\\"->\\",\\"args\\":[null]},{\\"file\\":\\"/var/www/html/core/Controller/AvatarControll
er.php\\",\\"line\\":84,\\"function\\":\\"getFile\\",\\"class\\":\\"OC\\\\\\\\Avatar\\\\\\\\UserAvatar\\",\\"type\\":\\"->\\",\\"args\\":[64,true]},{\\"file\\":\\"
/var/www/html/lib/private/AppFramework/Http/Dispatcher.php\\",\\"line\\":208,\\"function\\":\\"getAvatarDark\\",\\"class\\":\\"OC\\\\\\\\Core\\\\\\\\Controller\\\\
\\\\AvatarController\\",\\"type\\":\\"->\\",\\"args\\":[\\"jstewart\\",64,false]},{\\"file\\":\\"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php\\",\\"l
ine\\":114,\\"function\\":\\"executeController\\",\\"class\\":\\"OC\\\\\\\\AppFramework\\\\\\\\Http\\\\\\\\Dispatcher\\",\\"type\\":\\"->\\",\\"args\\":[[\\"OC\\\\
\\\\Core\\\\\\\\Controller\\\\\\\\AvatarController\\"],\\"getAvatarDark\\"]},{\\"file\\":\\"/var/www/html/lib/private/AppFramework/App.php\\",\\"line\\":161,\\"fun
ction\\":\\"dispatch\\",\\"class\\":\\"OC\\\\\\\\AppFramework\\\\\\\\Http\\\\\\\\Dispatcher\\",\\"type\\":\\"->\\",\\"args\\":[[\\"OC\\\\\\\\Core\\\\\\\\Controller\\\\\\\\AvatarController\\"],\\"getAvatarDark\\"]},{\\"file\\":\\"/var/www/html/lib/private/Route/Router.php\\",\\"line\\":302,\\"function\\":\\"main\\",\\"class\\
":\\"OC\\\\\\\\AppFramework\\\\\\\\App\\",\\"type\\":\\"::\\",\\"args\\":[\\"OC\\\\\\\\Core\\\\\\\\Controller\\\\\\\\AvatarController\\",\\"getAvatarDark\\",[\\"OC\\\\\\\\AppFramework\\\\\\\\DependencyInjection\\\\\\\\DIContainer\\"],[\\"jstewart\\",\\"64\\",\\"core.avatar.getavatardark\\"]]},{\\"file\\":\\"/var/www/html/lib
/base.php\\",\\"line\\":1001,\\"function\\":\\"match\\",\\"class\\":\\"OC\\\\\\\\Route\\\\\\\\Router\\",\\"type\\":\\"->\\",\\"args\\":[\\"/avatar/jstewart/64/dark\\"]},{\\"file\\":\\"/var/www/html/index.php\\",\\"line\\":24,\\"function\\":\\"handleRequest\\",\\"class\\":\\"OC\\",\\"type\\":\\"::\\",\\"args\\":[]}],\\"File\\
":\\"/var/www/html/lib/private/Files/ObjectStore/S3ConnectionTrait.php\\",\\"Line\\":145,\\"message\\":\\"Could not create object urn:oid:20 for appdata_ocy4keitmx
g4/avatar/jstewart/avatar-dark.png\\",\\"exception\\":{},\\"CustomMessage\\":\\"Could not create object urn:oid:20 for appdata_ocy4keitmxg4/avatar/jstewart/avatar-
dark.png\\"}","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36","version":"30.0.0.14"}
nextcloud 10.42.7.135 - - [06/Oct/2024:23:05:35 +0000] "GET /avatar/jstewart/64/dark?v=1728255935126 HTTP/1.1" 404 753 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x6
4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"```

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working S3 Anything to do with S3 object storage
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants