Dictionary index was missing in checking idle users

[maximmasiutin]

There was an exception that the key was not found when opening the /pending URL

[ppigazzini]

I need some time to review the whole userdb logic, but reading the code on the fly the error should require that the user_cache contains an user not present in the users collection (and this is strange).

Please let me know your steps to replicate the problem.
I just tested the wiki setup script (only change I set python 3.11.3) on a clean Ubuntu 18.04.
After login as user00 (the admin) I clicked the "Pending Users" link without problems.

Even after dropping all the collections I can click on "Pending users" without problems.

[maximmasiutin]

Thank you for trying to reproduce this error. I was using Python 3.10.7 on Ubuntu 22.10.

I first logged in as user00 and changed the password, then logged out, then created a new user maximmasiutin, then logged in as user00 in another tab and approved the new user maximmasiutin, then was idle for about 5 minutes, then visited "Users - Pending & Idle" (as user00) and the error appeared.

Similar error is when creating a new run but did not setup a Github token. Github denies the request after a few attempts (e.g. specify improper signature a few times) but Fishtest does not report that but fails (internal server error) due to a missing directory index.

The error is in the line

if b["path"].endswith((".epd.zip", ".pgn.zip"))

it tells that the string index must be integer

it stops giving "internal server error" in this line if I change it to

if isinstance(b, dict) and b["path"].endswith((".epd.zip", ".pgn.zip"))

to reproduce this error, delete Github token or use invalid token.

(however, it then gives similar error in another place.

[maximmasiutin]

Also, the script creates 10 users, but there is no userdb API function to delete a user - might have been useful.

[ppigazzini]

Please mind that MongoDB Community Edition supports only Ubuntu LTS, and until a couple of months ago Ubuntu 22.04 was not supported (I need to test the fishtest setup script on Ubuntu 22.04)

https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/

Deleting an user will corrupt the relations between the data/collections.

[maximmasiutin]

If the user is just created (never authenticated) than there is no action done by this user yet so it is safe to delete that user.

I installed MongoDB Community Edition on Ubuntu 22.10 using the setup script for 22.04.

[ppigazzini]

The never active user is deleted automatically by the scheduled script.
https://github.com/glinscott/fishtest/blob/38a858fd5d86c63f110f3eeee96078416aaf8cda/server/utils/delta_update_users.py#L205-L229

[ppigazzini]

The server setup script works on a clean Ubuntu 22.04 VM, no errors during the install and all the services are up and running, but I have a forbidden access to all the static content. eg http://172.17.195.40/favicon.ico
And trying to login I got "check_csrf_token(): Invalid token"

The setup on Ubuntu 18.04 is fine (and on Ubuntu 20.04 when tested some months ago).

[vdbergh]

I also installed some time ago Fishtest on 22.04. I recall that there were some issues with file or directory permissions. I don't remember how I fixed this.

[ppigazzini]

I also installed some time ago Fishtest on 22.04. I recall that there were some issues with file or directory permissions. I don't remember how I fixed this.

Thank you, so this is a general problem, and we need to fix the setup script.
EDIT_000: setting the ownership to www-data didn't help.

[ppigazzini]

Be sure that the script delta_update_users.py is running on your instance.
I added some comments, deleting a idle user should be always possible.

    # initialize idle with all the users
    for u in request.userdb.get_users():
        idle[u["username"]] = u
    # remove the active users
    for u in request.userdb.user_cache.find():
            del idle[u["username"]]

https://github.com/glinscott/fishtest/blob/163f84d12e83de768d74446f588a29cadd4765bc/server/utils/delta_update_users.py#L194-L198

[ppigazzini]

user_cache should have or a subset or the same users contained in users

mongosh << EOF
use fishtest_new
db.users.find()
db.user_cache.find()
EOF

run:

${VENV}/bin/python3 fishtest/server/utils/delta_update_users.py all

[maximmasiutin]

Here is my crontab:

VENV=/home/fishtestmaximmasiutin/fishtest/server/env
UPATH=/home/fishtestmaximmasiutin/fishtest/server/utils

# Backup mongodb database and upload to s3
# keep disabled on dev server
# 3 */6 * * * /usr/bin/cpulimit -l 50 -f -m -- sh ${UPATH}/backup.sh

# Update the users table
1,16,31,46 * * * * ${VENV}/bin/python3 ${UPATH}/delta_update_users.py

# Purge old pgn files
33 3 * * * ${VENV}/bin/python3 ${UPATH}/purge_pgn.py

# Clean up old mail (more than 9 days old)
33 5 * * * screen -D -m mutt -e 'push D~d>9d<enter>qy<enter>'

VENV=/home/fishtestmaximmasiutin/fishtest/server/env
UPATH=/home/fishtestmaximmasiutin/fishtest/server/utils

# Backup mongodb database and upload to s3
# keep disabled on dev server
# 3 */6 * * * /usr/bin/cpulimit -l 50 -f -m -- sh ${UPATH}/backup.sh

# Update the users table
1,16,31,46 * * * * ${VENV}/bin/python3 ${UPATH}/delta_update_users.py

# Purge old pgn files
33 3 * * * ${VENV}/bin/python3 ${UPATH}/purge_pgn.py

# Clean up old mail (more than 9 days old)
33 5 * * * screen -D -m mutt -e 'push D~d>9d<enter>qy<enter>'

Probably, the error appeared before the task run.

[ppigazzini]

The delta_update_users.py is the only point where an user is dropped from the "users" collection (if never active), but the code deletes from idle (list of candidates to be deleted) every user in the user_cache.

user_cache collection is supposed to contain only users contained in the users collection as well, so I would like to keep the code as is. IMO is dangerous to silence that type on inconsistence.

https://github.com/glinscott/fishtest/blob/163f84d12e83de768d74446f588a29cadd4765bc/server/utils/delta_update_users.py#L214-L229

[ppigazzini]

I also installed some time ago Fishtest on 22.04. I recall that there were some issues with file or directory permissions. I don't remember how I fixed this.

Wiki server setup script update to Ubuntu 22.04, nginx permission fixed with:

usermod -aG ${user_name} www-data

https://stackoverflow.com/questions/16808813/nginx-serve-static-file-and-got-403-forbidden