Issues with BlueCoat proxy

[trupf]

Describe the bug

A clear and concise description of what the bug is.

To Reproduce

Steps to reproduce the behavior:

1. Use Firefox Browser mit proxy setting (proxy requires authentication)
2. setup sync with nextcloud account, (using login with nextcloud)
3. Login and setup is working
4. sync doesn't work with error E018

Expected behavior

After a successfull connection setup the sync should also work

Screenshots

Desktop

(please complete the following information)

• OS: Windows 10
• Browser Firefox
• Browser Version 91.0.2
• Floccus version: 4.7.0
• Floccus sync method: nextcloud

Server

(please complete the following information)

• OS: Debian Buster
• Nextcloud version: 21.0.4
• Bookmarks app version: 4.4.1

Note: Please write down the actual version numbers instead of writing 'latest'.

Debug log

2021-08-26T06:40:01.836Z Starting sync process for account ******
2021-08-26T06:40:01.854Z Using "merge default" strategy (no cache available)
2021-08-26T06:40:02.543Z Fetching bookmarks
2021-08-26T06:40:03.161Z Syncing failed with E018: Konnte nicht am Server authentifiziert werden.

Additional context

The sync is not working from my company laptop+network which requires a proxy with authentication for web access. It works though form my private laptop/network where I don't need a proxy. So I guess it has something to do with proxy settings. Firefox itself does work with the proxy. I have the same problem with Chrome - doesn't work from company laptop/network.

[MicMun]

I have the same problem, but I don't believe that this can be fixed by Floccus.
Our proxy (Blue Coat, now Broadcom ...) deletes the Authentication header and that is the cause of the error.
I guess, your proxy do the same.

[trupf]

I don't know what the proxy does and if it deletes the header (and also don't know how to debug). But I can login to Nextcloud via the web page with Chrome or Firefox and the nextcloud sync agent does also work. The sign in for account setup in floccus does also work but the actual sync doesn't.

[mnalis]

@trupf can you see what nextcloud logs on the server (eg. /var/lib/nextcloud/data/nextcloud.log or wherever it is on your server) say for that same timestamps when login fails?

[trupf]

I can see the nextcloud logs, but nothing is logged there. But I can also see sslh logs from the server and see that a connection attempt was made, Nginx is configured to not log every access attempt so I think it is really the problem as described by MicMun, that the nextcloud auth header is missing due to the proxy, although I don't understand than how it can work with the nextcloud sync client. Is there a way to authenticate a client without that header?

[mnalis]

@trupf I'm not that familiar with the floccus code, but it looks like src/lib/adapters/NextcloudBookmarks.ts uses HTTP Basic auth. It would be great if you could:

• enable nginx logs for short time, in order to capture logs when floccus tries to connect. That should provide more information
• provide some info on the kind of proxy you're using (not the hostname / personal details of course, but is it HTTP/HTTPS proxy, or SOCKS v4/v5, or perhaps if you know who is manufacturer of proxy software etc). For firefox some info should be found here
• is your nextcloud instance HTTP or HTTPS ? Having it on HTTPS might avoid some transparent proxy issues (and you can get free HTTPS certificates via https://letsencrypt.org/)

Alternatively as a workaround, perhaps you can make exception for your nextcloud instance so it does not go via proxy (see No proxy for in firefox config docs above)? That would only work if it is manual/automatic proxy, not in case it is transparent proxy, though.

[trupf]

• Nextcloud instance is https
• proxy is http/https. I can not bypass this proxy as I'm not the admin and I have no web access without it
• only the debug level of error log provides some info

error_debug.log

[mnalis]

@trupf OK, I see that is also using Bluecoat proxy as @MicMun.

If BlueCoat removes the HTTP Basic auth headers as alleged (and your nginx error_debug.log indeed does not show Authorization header), it would explain why GET /owncloud/index.php/apps/bookmarks/public/rest/v2/bookmark?page=0&limit=300 at line 397 fails with 401 Unauthorized at line 626.

But what is also interesting is that GET /owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash at line 30 fails with HTTP/1.1 400 Bad request at line 256 before that, which is not 401 Unauthorized.

Can you try to (after logging in to your nextcloud instance) open in your normal browser tabs:

• https://your_nextcloud.example.com/owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash
• https://your_nextcloud.example.com/owncloud/index.php/apps/bookmarks/public/rest/v2/bookmark?page=0&limit=300

(where your_nextcloud.example.com is replaced with your instance FQDN)

and see if they show valid JSON, or return 400/401 error messages? Also share what a nginx.log looks like when you try to open those URLs from browser.

Also, could you try if (while using that proxy) you can connect to simple site using HTTP basic auth, like this one:

• https://authenticationtest.com/HTTPAuth/ (there should be a window popping up, asking for username/password which are User and Pass), and after that redirect to page saying if it was successful or not.

[trupf]

I can not see any issues with both statements in the browser, they complete with no error and return valid json data.

The basic auth does also work. I have even another instance on my web-server where I'm actually using basic out with nginx and that works too.

error_1.log
error_2.log

[mnalis]

Few observations, this seems to be the difference in headers sent to server when fetching /owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash

--- head_floccus.txt	2021-09-19 20:53:16.976555816 +0200
+++ head_browser.txt	2021-09-19 20:42:50.594542889 +0200
@@ -6,0 +7 @@
-http header: "Cache-Control: max-stale=0"
+http header: "Cache-Control: max-age=0"
+http header: "Upgrade-Insecure-Requests: 1"
-http header: "Accept: */*"
+http header: "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
-http header: "Sec-Fetch-Mode: cors"
+http header: "Sec-Fetch-Mode: navigate"
-http header: "Sec-Fetch-Dest: empty"
+http header: "Sec-Fetch-Dest: document"
+http header: "Sec-Fetch-User: ?1"
+http header: "Cookie: nc_sameSiteCookielax=true; nc_sameSiteCookiestrict=true; nc_username=trupf; oc_sessionPassphrase=.....; oc85ae6e4789=l79941cr1es8f5v32h7t72u5il; nc_token=hdrk1SPSsXuodE%2BTwziX9C2Ap1CeKcgD; nc_session_id=l79941cr1es8f5v32h7t72u5il"

While any difference might cause suspect proxy to react differently, the most glaring difference is that headers sent by Floccus lack any Cookies, which seem to be already be used for authentication at this stage (your nginx debug logs do not mention http header: "Authorization: basic" anywhere).

So I would guess the problem occurs somewhere before error_debug.log that you've sent before, @trupf

Could you try:

1. exiting the web browser
2. starting debug log in nginx (or remembering its position if its already on)
3. starting the browser
4. attempting the sync
5. waiting for failure
6. sending debug log produced in time between step (2) and step (5).

That should capture the authentication attempt.

[trupf]

Above you wrote "after logging in to your nextcloud instance" so it is clear authentication is already done. Anyway as long as I don't explicitly log off or delete cookies I don't need new log in to nextcloud, even if I restarted my computer or started a new browser session. So authentication is probably done with cookies and only if no valid cookie found with a new login. I can follow your procedure above but you will find multiple connections than, as it will take longer an I cannot filter for just floccus. Just to make you aware I don't fill in the password field in floccus. Instead I'm using the button right at the end of the user name field to sign in with the browser (which is working, but the sync afterwards doesn't). This way I would expect floccus to create a cookie for authentication and use it (same way as the nextcloud sync client does). This way floccus does work on my home computer (without company proxy). I also tried to use a separate app password for floccus (as I have 2 factor auth enabled) but this doesn't work either.

[marcelklehr]

Instead I'm using the button right at the end of the user name field to sign in with the browser (which is working, but the sync afterwards doesn't). This way I would expect floccus to create a cookie for authentication and use it (same way as the nextcloud sync client does)

Fyi, using the nextcloud login flow produces an app password which is sent via the Basic Auth header just like a normal password.

To test the authentication I recommend using curl as follows:

curl -u "username:apppassword" -vvv https://your_nextcloud.example.com/owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash

[trupf]

here is the requested debug log. This time i have not deleted previous (as I think unrelated) requests.

error_debug3.log

I can not use curl on my company laptop (and if I would need a way to use the proxy). But with my private and without the proxy the request ist working.

[mnalis]

Above you wrote "after logging in to your nextcloud instance" so it is clear authentication is already done.

Yes I did, for the 2 URLs you manually entered in web browser. What I'm trying to find out here is how floccus internal method (which doesn't work for you) differs from the way normal browser tabs works. If they worked in exactly the same way, they would both work (or both fail). The interesting part is that one method (web login in normal browsing tab, which produces cookie) works, while the other (floccus addon auth) fails.

Request generated by floccus in my case (with separate app password) looks like:

Request received from client: GET /owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash HTTP/1.1
 Headers received from client:
   Host: my.nextcloud.example.com
   User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0
   Accept: */*
   Accept-Language: en-US,en;q=0.7,hr;q=0.3
   Accept-Encoding: gzip, deflate, br
   Authorization: Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
   DNT: 1
   Connection: keep-alive

So in my case Floccus uses HTTP basic auth on each request, instead of "form login submit sets cookie" like web-browser login on nextcloud does. Since you say you were able to login on https://authenticationtest.com/HTTPAuth/ which also use same HTTP basic authentication method, floccus HTTP basic auth should work too.

If it doesn't, and BlueCoat strips authentication header before it reaches your nextcloud instance, but does not strip it for https://authenticationtest.com/HTTPAuth/, there must be a reason, and that reason is determined by request that is sent to proxy server. If requests were exactly the same, the result must be exactly the same too. But they never are completely the same, so BlueCoat might strip authentication header (if it really does that) depending on whether:

• it doesn't like some header the floccus sends (maybe Accept: */* or DNT: 1 ?) - which might be helped by new floccus version, if we find out exactly what it doesn't like (having curl would be very helpful here)
• it has some sites blacklisted by name (for example, if it contains nextcloud in the name), in which case you can workaround that by changing (or adding alias) FQDN for your site name
• it has same sites blacklisted by IP (errr, move the site to different hosting?)
• it only whitelists some sites by IP or FQDN (no workaround except asking for your site to be added as exception)
• it doesn't like something about HTTPS certificate or something (expired? self-signed? CA untrusted by BlueCoat? maybe could try different certificate or plain HTTP. curl would also help here)
• it doesn't like some path part of the URL (for example, strips authentication header if there is public in path name, or /-1/ or something - not much help here without kludging both floccus and nextcloud code if that is the case!). Not very likely to be the issue, though.

Or in all cases, you can try contacting your proxy administrator, and see if the restrictions can be changed/lifted, of course.
Knowing what exactly is the problem would likely help with that too (if it is an option at all, of course).

Just to make you aware I don't fill in the password field in floccus. Instead I'm using the button right at the end of the user name field to sign in with the browser (which is working, but the sync afterwards doesn't). This way I would expect floccus to create a cookie for authentication and use it (same way as the nextcloud sync client does). This way floccus does work on my home computer (without company proxy). I also tried to use a separate app password for floccus (as I have 2 factor auth enabled) but this doesn't work either.

Ah, I'm not sure how that way works; but is possibly harder to debug if it works differently. Could you try removing that floccus account on your company laptop, and setup a new account with separate app password for floccus, and send me that 6-step logs for that? Mixed requests should not be a huge problem, as all that pass through your company proxy seem to have X-BlueCoat-Via header.

Also, when you say you can't use curl, is it because you can't download/run an executable from https://curl.se/windows/ ? Or simply because you must use company proxy to access the web? Because curl has --proxy https://your.proxy.server.example.com:port option which should work with your company proxy.
If you can download and run curl, it would greatly simplify the debug process.

[mnalis]

@trupf if you can install curl, the output of the following would be useful:

curl --proxy https://your.proxy.server.example.com:port -u "username:apppassword" -vvv https://your_nextcloud.example.com/owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash
curl --proxy https://your.proxy.server.example.com:port -u "username:apppassword" -vvv -H "DNT: 1" -H "Accept: */*" https://your_nextcloud.example.com/owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash
curl --proxy https://your.proxy.server.example.com:port -vvv -u User:Pass https://authenticationtest.com/HTTPAuth/
curl --proxy https://your.proxy.server.example.com:port -vvv -H "DNT: 1" -H "Accept: */*" -u User:Pass https://authenticationtest.com/HTTPAuth/

It would hopefully show if any of the extra headers have effect on proxy, and are there certificate differences that might be problematic. Depending on the results, I can setup (if needed) few URLs that would test if it is path values that are problematic instead.

[MicMun]

https://authenticationtest.com/HTTPAuth/ is blocked at my company, but it is interesting that folder and bookmark is acting different.

1. curl --proxy https://your.proxy.server.example.com:port -u "username:apppassword" -vvv https://your_nextcloud.example.com/owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash
   HTTP/1.1 400 Bad request
   {"status":"error","data":"Could not find folder"}

2. curl --proxy https://your.proxy.server.example.com:port -u "username:apppassword" -vvv https://your_nextcloud.example.com/owncloud/index.php/apps/bookmarks/public/rest/v2/bookmark?page=0&limit=300
   HTTP/1.1 401 Unauthorized
   {"status":"error","data":"Unauthorized"}

The request is identical: Authorization, User-Agent and "Accept: */*" is send to nextcloud. The DNT header makes no difference in output (I tried with and without DNT).

The main differences in the response I can see:
1.: Cache-Control: no-cache, no-store, must-revalidate
2.: Cache-Control: no-cache, no-store, must-revalidate, proxy-revalidate
Only in 2.:
Authentication problem. Ignoring this.
WWW-Authenticate: Basic realm="Nextcloud", charset="UTF-8"
Set-Cookie: BCSI-CS-4935385577b06649=1; Path=/
Proxy-support: Session-based-authentication

The whole output has too much critical infos, so I cannot provide this to you. But perhaps the diff helps.
The first request .../folder/-1/hash returns the same message in browser before login, after login to nextcloud the request returns valid data without error.

I'm really surprised, that the requests differs in the result. I thought, the proxy does always the same thing and the problem must also be always the same.

[MicMun]

If it helps, I have a curl execution with success:
curl -ks -vvv -L --anyauth -u "$AUTH" -b cookie-store -c cookie-store 'https://my.nextcloud.instance/owncloud/index.ph/apps/bookmarks/public/rest/v2/bookmark?page=0&limit=3'

The output:
request.log

The first request is without basic authentication, the second request (curl does this automatic) is with cookies and basic authentication. The request get the right result.

[MicMun]

$AUTH is my "ocuser:app-password" and I have the proxy settings in environment variables. So don't mind that my curl is shorter than your examples.

[marcelklehr]

That would be a bug in the bookmarks app :/ The API should not allow authentication using cookies to prevent cross-site request forgery. 🙈

[trupf]

enclosed you'll find the output of the curl statements (see number 1 and 4)
I have repeated number one with my own server to capture nginx log too (see number 2), here you can see that no Authentication: Basic header is received although it was sent by curl.
Number 3 shows nginx log for the same request done from the web browser. You can see that Authentication: Basic header is received in nginx if it is the response to an error 401 answer from nginx (be aware that at first try I entered a wrong password, so that is repeated again)

curl_http_Basic_auth1.txt
curl_http_Basic_auth2.txt
nginx_error_basic_auth2.log
nginx_error_basic_auth3.log
curl_http_Basic_auth4.txt

To me it is pretty clear, that BlueCoat deletes the Authorization: Basic header if no 401 response was sent to the client before with same connection id. So I guess floccus browser plugin should be changed to sent auth data only after it received a 401 response. And only if that second request fails it should show the E018 error. May be some modificaion of floccus itself is also required to answer an unauthorized request always with error 401 not with error 400 like it does for example for https://your_nextcloud.example.com/owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash

[trupf]

MicMun:
The first request is without basic authentication, the second request (curl does this automatic) is with cookies and basic authentication. The request get the right result.

I think second request went through as first one was answered with an error 401, so BlueCoat let the Authentication: Basic header through for second request, same result as my log files show, so probably not due to the coockies.

[mnalis]

Yeah, I think @trupf is correct here, and nextcloud cookies are unrelated in @MicMun working connection - what is important is that the client first tries with no auth, and only when it gets 401 Unauthorized response it retries with HTTP basic auth (instead of trying with unsolicited basic auth immediately).

To confirm that, would you both @trupf and @MicMun please try this two commands (add your --proxy options as needed, but retain other options in commands below):

curl -vvv -L --anyauth -u "$AUTH"  'https://my.nextcloud.instance/owncloud/index.php/apps/bookmarks/public/rest/v2/bookmark?page=0&limit=3'
curl -vvv -L --anyauth -u "$AUTH"  'https://my.nextcloud.instance/owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash'

If the assumption is correct, the first command will succeed for first command (in automatic 2nd try), but will still fail with second command. When I see that two curl outputs, if they'll look like I suspect they will, I'll have some nextcloud changes for you to try.

[MicMun]

There is a typo in your url: index.ph -> index.php :)

[MicMun]

@mnalis The second request is 400 Bad request, this was suspected.
The first request still failed, but with Cookie "BCSI-CS-<Number>=1" it succeeded. Here <Number> is 4935385577b06649, but this can be different with other proxy instances. I think, the proxy need this cookie to know, that he has scanned this request already.
The <Number> was yesterday and today the same at every request, so I think it is an instance sign.

See also https://knowledge.broadcom.com/external/article/200712/what-is-the-http-header-bcsics-cookie-se.html

[mnalis]

@MicMun so the second request does not retry second time (after that 400 Bad request), correct?
Can you share redacter version of what exactly is sent/received for both attempts of that first request, as reported by curl?

[MicMun]

@mnalis yes, the second request does only one request and fails with 400 Bad request.
I'm not sure about the wanted output, so you get both working and not working output.

Your curl request:
request1.log

With the Cookie "BCSI-CS-4935385577b06649=1":
requestWorking.log

[mnalis]

Ah, thanks @MicMun, makes sense now: curl will completely ignore cookies even in --anyauth auto-retry mode if -b/-c are not provided. So working solution is only when you manually force cookie with curl -H "Cookie: BCSI-CS-4935385577b06649=1" (or when you use -b/-c), correct?

Can you also check for me:

• if adding that -H "Cookie: BCSI-CS-4935385577b06649=1" header works in single-request mode, ie. without --anyauth parameter?
• Also, if single-request above works, when you change that BCSI-CS id to something else, eg. "Cookie: BCSI-CS-01234567890abcdef=1", does it perchance also works, or it fails?

[trupf]

@mnalis
the first curl line faills with first ty and I don't se a second one, It is successfull although if i include a cookie-file with the BCSI-CS cookie (with -b coockiefile -c cookiefile at second request, at following calls with same cookie file on first request already).
Your second line gets forwarded to the nextcloud login url (error 302), send the basic auth with next request which gets accepted and data send error 500 javascript missing.

[trupf]

@Mnails:
your first curl request works only with --anyauth and the cookie (and it also works with a cookie header and the correct cookie id as you wrote above, but not with a modified cookie id). But I have to delete the &limit=3 part of the url (server does not know this command) and still get error 500 at the end (complaining about missing javascrit, I guess this is to be expected in this situation).
Your second curl request following the first one works without --anyauth and without any cookie setting. I can even delete the -u "authstring" and it still works. Reason for this behavior is not clear to me.

[mnalis]

@trupf Uh, I'd really appreciate if you can send logs for that behaviour, like you did before (what was exact curl command you were running, and what was its output). There should not be error 500, not any complaining about missing javascript. Nor should it work without -u! This is quite a strange turn of events. I know it is more work to provide full logs, but it really helps a lot for debugging - thanks in advance for sending them!

As for &limit=3, is that perhaps this error Der Befehl "limit" ist entweder falsch geschrieben oder konnte nicht gefunden werden ( which I've seen it before in your curl_http_Basic_auth4.txt) ? If so, I think it is likely your local cmd.exe complaining that URL is not properly quoted. In my request I've used ' around the URL (which works fine in my zsh shell), but windows cmd might require " to be be around URL instead, in order for & to be sent to the server, instead as being interpreted as end of curl command. eg:

curl -vvv -L --anyauth -u "$AUTH"  "https://my.nextcloud.instance/owncloud/index.php/apps/bookmarks/public/rest/v2/bookmark?page=0&limit=3"
curl -vvv -L --anyauth -u "$AUTH"  "https://my.nextcloud.instance/owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash"

[MicMun]

@trupf do you fixed the index.ph to index.php? The original curl command has a typo (missing 'p'), then it will redirect to the login page.

[MicMun]

@mnalis I can not test until tomorrow, because I'm already at home. But I don't believe that an other number than from proxy will work.

[mnalis]

Sure, no problem, whenever you can. I too have little hope that using other number would work - I only asked that second question because if by a stoke of luck it did work, it would make a fix incredibly simple and readily available 😄
The first one (does it work in single-request mode; eg. with -H Cookie.... but without --anyauth) is also important for the more complex solution that will probably be needed.

[trupf]

The missing "p" at index.php might have been the problem. So here are the logs
I do not really like to provide all the logs as the risk for me is high to miss editing some sensitive information.

curl_http_11.txt
curl_http_12.txt
curl_http_21.txt
curl_http_22.txt
error_11.log
error_12.log
error_21.log
error_22.log

[MicMun]

@mnalis

1. curl -vvv -L -u "$AUTH" -b "BCSI-CS-4935385577b06649=1" 'https://my.nextcloud.instance/owncloud/index.php/apps/bookmarks/public/rest/v2/bookmark?page=0&limit=3' (without --anyauth) -> it works : request01.log
2. curl -vvv -L -u "$AUTH" -b "BCSI-CS-01234567890abcdef=1" 'https://my.nextcloud.instance/owncloud/index.php/apps/bookmarks/public/rest/v2/bookmark?page=0&limit=3' (with other number in the cookie name) -> 401 Unauthorized : request02.log

I would fix this with saving the cookie name after the first failing request (cookie is in the response) and after that, the cookie can be send with each request. If the name changed after some time, you can update the name after the response of a failing request.
Thank you so much for your effort.

[mnalis]

Yes, thank you @MicMun and @trupf, I think we've got to the bottom of this - as they say, finding a problem is half work done!
So we finally have something actionable:

• firstly and obviously, I would suggest to everybody who sent logs to revoke their old floccuss app passwords, and issue new ones, just in case some data was not properly redacted in logs - it is always a good practice.

• If one were in a hurry to get quickest workaround/kludge, they could edit their https://github.com/floccusaddon/floccus/blob/develop/src/lib/adapters/NextcloudBookmarks.ts#L922 (and maybe https://github.com/floccusaddon/floccus/blob/develop/src/lib/adapters/NextcloudBookmarks.ts#L855 ?) and hardcode their own Cookie: BCSI-CS-4935385577b06649=1 header in there, and it should work.

  Of course, while such idea should work as short-term workaround, it obviously has a number of deficiencies: it does not help others with same problem, one needs to rebuild the floccus add-on from source, each upgrade would break it again, if company changes proxy it would break again too etc.

• Nextcloud bookmarks app has a issue with not being a nice citizen, ie. it replies with 400 Bad request on some requests when username/password are required but not sent, while it should be 401 Unauthorized (as it is one some other requests). So other clients aside from floccus might have problems there too. Quick check shows that making a backup of nextcloud/apps/bookmarks/lib/Controller/FoldersController.php and replacing all 12 instances of:

  -  ['status' => 'error', 'data' => 'Could not find folder'], Http::STATUS_BAD_REQUEST);
  +  ['status' => 'error', 'data' => 'Could not find folder'], Http::STATUS_UNAUTHORIZED);

  made that https://your_nextcloud.example.com/owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash URL react with proper 401 Unauthorized instead of 400 Bad request. Of course it should be done smarter. I'll file bug for that at nextcloud bookmarks and link to it here later, as it affects all clients using that API.

• Finally, floccus should IMHO be modified to support such proxies, which I'll detail in next comment

[mnalis]

Floccus should IMHO be modified to support proxies which drop unsolicited Authorization headers (ie. when authorization is sent immediately, and not after 401), by retrying same request with new cookies received. RFC 7235 (which is by now over 7 years old), does seem to allow for proxies to drop Authorization headers (it was previously required that proxies must forward the WWW-Authenticate and Authorization headers untouched, but that restriction seems to be relaxed now to just forbid modifying them)

This is what I would suggest to be done in sendRequestNative() (or is it sendRequestWeb() ?) in NextcloudBookmarks.ts:

• if the fetch() with Authorization: header fails with res.status === 401, it should check if response had any Set-Cookie: and if so, request should be retried with all previous data + said cookies (I'd strip those cookies starting with oc or nc for cleanliness and not to interfere later, but it probably might not do any harm even if all cookies are left in).
  Note that this case (401 received when Authorization: basic was sent) should happen only with filtering proxies such as BlueCoat (wrong username/pass would normally produce 403), so doing it this way would ensure common case would remain the same as it was up to now, so no introducing new bugs for majority of existing users.
• ideally, when in such case it was detected that request must be retried with cookies, that cookies should be saved for that account for that browsing session until browser closes (ie. they should not be permanent on disk, but just in memory), so on next request that cookies would be used immediately. This is mostly performance issue, as it should work even without that cashing, but anybody with BlueCoat and similar proxy would have to send twice the amount of request.

As this is now an actionable item, if @marcelklehr agree I would make a new issue (out of just this comment)?

If I were forced to, I'd probably be able to scrounge something that works (with massive help from testers), but as I neither really speak typescript, and especially all that object-oriented, try/catch and async/Promise model is alien to me, it would probably look quite horrible (like an perl programmer added a global variable, copy/pasted some code he only had a vague idea what it does, and then hacked at it until it started to work -- which is actually exactly what would be happening in such case 😄 )
So I'd prefer to leave actual programming part to someone much more qualified, unless @marcelklehr is so overwhelmed with other work that BlueCoat victims would be left out in the cold for months/years.

[marcelklehr]

Thanks @mnalis for the troubleshooting and getting to the bottom of this. Much appreciated!

Unfortunately, sending cookies with floccus is problematic, because

a) we cannot choose which cookies to send (the browser API doesn't allow that) and
b) the cookies jar is shared with the normal browsing context, automatically logging people into some nextcloud and often also logging them out on some errors

I'm also a bit annoyed by the fact that a proxy just right out strips important headers from a request, to be honest. Floccus won't be the last app that has problems with such behavior. :/

[mnalis]

Wow, I've been overly optimistic again - amazing that one goes through the effort to design API that allow custom headers to be sent in fetch request and then immediately forbid vast majority of useful headers from being used!

But there do seem to be ways around that; for example I've just tried this Firefox AddOn https://github.com/didierfred/SimpleModifyHeaders/blob/master/background.js#L114 and according to https://httpbin.org/#/Cookies test site it seems to allow to rewrite cookies in outgoing requests according to programmatic rules using browser.webRequest.onBeforeSendHeaders.addListener().

Although if using standard cookie jar is possible in that fetch() API, that should be much more clean solution IMHO.

In any case, cookie workaround should be enabled only if our attempt to contact target site while using basic auth fails with 401, so even if there are side effects it would only ever affect such users who would be completely left out in the cold otherwise. (And yeah I too was not initially happy with proxies dropping headers, but they seem to be allowed to do that and have a convincing enough reason for doing it).

Now that reminds me (looking at the floccus code in more details...): floccus seems to already have Send client certificates checkbox in options; is that the one that actually enables saving/sending cookie jar (among other things)?
(ie. the code does fetch (.... credentials: this.server.includeCredentials ? 'include' : 'omit')

Because if that cookie jar is saved across requests when credentials: include, would then just checking that option and retrying sync two times (since we don't have auto-retry yet) makes things work for @trupf and @MicMun even with current floccus version??

[marcelklehr]

Nice find! I forgot about that one. I can adjust the description of that option a bit to make it more clear it's applicable to this use case as well.

[trupf]

If you just store the BSCI-CS... this shouldn't be a problem the cookie is just for the proxy and does not even reach nextcloud. On the other hand all web sites go through the proxy, so the cookie will be set for every one anyway. You don't even need to explicitly store a cookie file, you could just store a text string and add it as header. And you are right, floccus is not the last app that has problems with this behavior.

[trupf]

Because if that cookie jar is saved across requests when credentials: include, would then just checking that option and retrying sync two times (since we don't have auto-retry yet) makes things work for @trupf and @MicMun even with current floccus version??

Yes I can confirm, checking the "Send client certificates" checkbox makes floccus work for me.

[mnalis]

That is great news @trupf ! Does the first sync attempt after browser startup still fail with an error, or does floccus now (when that Send client certificates is checked) work correctly without errors?

[mnalis]

Also, @MicMun, how does checking Send client certificates work for you?

[MicMun]

@mnalis I can test it Monday at work. Sorry, it is not sooner possible.

[mnalis]

Nextcloud bookmarks bug with returning 400 instead of 401 has now been reported at nextcloud/bookmarks#1658

[trupf]

That is great news @trupf ! Does the first sync attempt after browser startup still fail with an error, or does floccus now (when that Send client certificates is checked) work correctly without errors?

First sync never completed. I had to stop and restart it, than it worked. At browser restarts the first sync is immediately successful as long as the cookies are kept (I usually delete all when I close the browser, but now I've added an execption for nextcloud)

[MicMun]

@mnalis I can confirm, Send client certificates works for me. The sync completed with success.
Thats really great.

@trupf I think, floccus needs the cookies to succeed and thats the reason for the need of restart.
I had the nextcloud cookies already, the first sync works immediately.

[mnalis]

@MicMun that is great news, I'm glad it helped!

So, now that you've set this Send client certificates checkbox, does Floccus still sometimes throw any errors at all (for example after you restart a browser)? Or is the situation now completely fixed for you, and no further changes need to be done to floccus?

[MicMun]

@mnalis as I testet after restart, it works for me. So I don't see any further changes in floccus to be done.
Thank you very much for the help.

[marcelklehr]

Thanks so much @mnalis for getting to the bottom of this and coming up with the fix :)

[trupf]

@marcelklehr
Just to answer this from my side: it is working correct, after I have made the exception not to clear cookies for my nextcloud. I think it would be a good idea to write somewhere, that cookies need to be accepted by the browser and must persist over sessions, or the better option would be to change something so that after the first error 401 (due to no cookie in cache) a second retry is made with BSCI-CS cookie which was returned from the first try.