Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated Quota Support to Latest Version of MiaB and resolving code review comments #2387

Open
wants to merge 16 commits into
base: main
Choose a base branch
from

Conversation

chadfurman
Copy link

@chadfurman chadfurman commented Apr 27, 2024

This is an implementation of mailbox quotas using dovecot's quota capabilities. This pull request supersedes a previous pull request that contained some changes not pertinent to the main repository.

^ @jrsupplee


This PR is an update of #1568 ported to the latest version of MiaB.

TODO:

  • Move conf files to editorconf tool
    • 15-mailboxes.conf
    • 20-imap.conf
    • 90-quota.conf
  • get confirmation that we're okay with how the quota column is being added in the setup script - is this maintainable?
  • get confirmation we're okay with recalculating quotas on setup -- seems fine to me?
  • Figure out why JRSupplee had his own bootstrap script and resolve any discrepencies -- I'm guessing it's largely related to the conf files but could also be initializing dovecot somehow tbd
  • get approval from MiaB team and get merged into mainline

@chadfurman chadfurman mentioned this pull request Apr 27, 2024
@chadfurman
Copy link
Author

Looks like 15-mailbox.conf is just relocated and the script is already updated so it's fine as-is.

@chadfurman
Copy link
Author

Does not appear to be any reason why https://raw.githubusercontent.com/jrsupplee/mailinabox/master/setup/bootstrap.sh was being used instead of https://mailinabox.email/setup.sh?ping=1 -- my best guess is for reasons of pinning the version

.gitignore Outdated
@@ -5,5 +5,6 @@ tools/__pycache__/
externals/
.env
.vagrant
.idea/
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small change happy to remove if you want

@chadfurman
Copy link
Author

Woops looks like basing off main was a bad idea. Gonna redo this and base instead off of v68

@chadfurman chadfurman force-pushed the master branch 3 times, most recently from f82be1b to 1795f8a Compare April 27, 2024 22:54
@chadfurman
Copy link
Author

So the conflict is trivial to resolve (just bring in the first line of the if block and also the elseif block) but when I do, it also pulls in the latest commits from main and it causes the setup to break (due to the no password provided bug) so I'm not resolving conflicts just yet.

@chadfurman
Copy link
Author

strangely, I get a 500 error the first time I run setup/start.sh but the second time it works?

@chadfurman
Copy link
Author

When it 500s after setting the password, it seems it fails to create the admin user.

Creating the admin user by hand seems to work?

ubuntu@ip-172-31-36-105:~/mailinabox$ sudo management/cli.py user make-admin '[email protected]'
sudo: unable to resolve host box.agnai.guide: No address associated with hostname
OK
ubuntu@ip-172-31-36-105:~/mailinabox$ sudo management/cli.py alias add [email protected] [email protected]
sudo: unable to resolve host box.agnai.guide: No address associated with hostname
alias added

@chadfurman
Copy link
Author

Fixed the 500, it was related to missing the subprocess module. Fixed some conflicts with the spamassassin.sh script also

@chadfurman
Copy link
Author

I've resolved the conflict -- the remote chadfurman/mailinabox has a somewhat stable and tested version of this tagged as v68a (i.e. git checkout v68a), but the head now lines up with mail-in-a-box/mailinabox to make this easier to merge / manage.

@@ -117,6 +123,14 @@ def setup_key_auth(mgmt_uri):
if "admin" in user['privileges']:
print(user['email'])

elif sys.argv[1] == "user" and sys.argv[2] == "quota" and len(sys.argv) == 4:
# Set a user's quota
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this comment correct? I don't understand how this sets a user's quota.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is going to display not set a users current quota.

@stylnchris
Copy link

stylnchris commented May 6, 2024

Not sure if it helps or not but I've installed this with the v68a 'tag' on my production MiaB server and can confirm it works.

Steps I've taken to install it:

ssh into your box -

cd /root/
mv mailinabox mailinabox.old
git clone https://github.com/chadfurman/mailinabox.git
cd mailinabox
git checkout v68a
sudo setup/start.sh

echo "CREATE TABLE aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 "$db_path";
echo "CREATE TABLE mfa (id INTEGER PRIMARY KEY AUTOINCREMENT, user_id INTEGER NOT NULL, type TEXT NOT NULL, secret TEXT NOT NULL, mru_token TEXT, label TEXT, FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE);" | sqlite3 "$db_path";
echo "CREATE TABLE auto_aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 "$db_path";
elif sqlite3 $db_path ".schema users" | grep --invert-match quota; then
echo "ALTER TABLE users ADD COLUMN quota TEXT NOT NULL DEFAULT '0';" | sqlite3 $db_path;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You've opted to make the quota column type TEXT versus making it a numeric column. Just curious if there was a reason for this choice.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure what is stored in sqlite is basically for the webui to 'display' with what is actually set with maildirsize file for dovecot.

Also don't forget this is based on proven code from @jrsupplee, so some of this code is obviously not written by Chad.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I see is we're storing actually a string like "3G" (3 gigabyte limit), not an integer number of bytes. So this is ok.

@@ -20,10 +20,12 @@ db_path=$STORAGE_ROOT/mail/users.sqlite
# Create an empty database if it doesn't yet exist.
if [ ! -f "$db_path" ]; then
echo "Creating new user database: $db_path";
echo "CREATE TABLE users (id INTEGER PRIMARY KEY AUTOINCREMENT, email TEXT NOT NULL UNIQUE, password TEXT NOT NULL, extra, privileges TEXT NOT NULL DEFAULT '');" | sqlite3 "$db_path";
echo "CREATE TABLE users (id INTEGER PRIMARY KEY AUTOINCREMENT, email TEXT NOT NULL UNIQUE, password TEXT NOT NULL, extra, privileges TEXT NOT NULL DEFAULT '', quota TEXT NOT NULL DEFAULT '0');" | sqlite3 $db_path;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, '0' means unlimited and non-zero represents the quota. But there's no symbolic representation for "system default' from what I can see. So for example, if I made the system default 1GB, then the users would all get their DB value set to 1GB. If I later were to increase the system default to 10GB, the existing users would stay at 1GB and not inherit the new default.

Correct me if I've overlooked something and that's not how it works, but assuming that description is correct, then I'm wondering if the users that are getting the "system default quota" should have a symbolic representation of this in the DB, so if the default were to be changed, these users inherit the new value automatically.

Copy link

@stylnchris stylnchris May 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quota is not applied, by default so it is basically set to 0. 0 represents "no quota or 'unlimited' quota" (obviously the limit is the size of the hard-drive for miab server.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

creating a new user will give you the option to specify a quota with the new user.

miab_quota

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here is an example of setting the quota:

miab_quota3

Then enter the number and and click the "set quota" button

miab_quota2

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think my confusion stems from the fact that 0 means unlimited, but I also see a system default-quota value that can be set, and I'm not sure what role that default value plays in this. How would I set a user to the "default-quota" value?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay I understand what you would like to see, but I believe this is a limitation of Dovecot / Postfix.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dms00's question is right. The purpose of the default quota is going to cause confusion. What I see is that it's a default for new users. It's not a default for existing users (prior to this PR) or users without a quota set - they all get no quota. I think the way to resolve this is to just remove the default quota functionality to avoid confusion.

@MrMirhan's point is probably not solvable with this projects architecture because it's up to the mail client to do the send and then save the mail. As @stylnchris alluded to.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand the confusion around the default quota bits. I have to look into this more. As I understand it, default quota could either be the default quota for new users or the default quota for all users that don't have a quota set. But since "not having a quota set" means "having your quota set to '0' because it's unlimited" then the question becomes: if, in fact, "default quota" is the quota for all users who don't have a quota set, then is it actually just ignored? And, alternatively, if "default quota" is just for the new users, then is it actually more confusing than useful?

I agree, removing default quota seems to make sense. I can look into this. Likely just a UI adjustment.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So there's a bit of a question here -- if we remove the default value of '0' from the table, it will need to be replaced with something otherwise all users will require a quota. Replacing it with empty string or Null makes the most sense, but i'm not sure how this will ripple out, so this is TBD?

Alternatively, if we remove the "system default" API endpoints, CLI commands, etc, this would reduce confusion while allowing users to continue to default to unlimited which I think makes the most sense.

That said, ideally I could make the system default "unlimited" and then have a per-user quota and where users aren't specifically given a quota then when the system default changes the user's quota would also change for any user where the quota isn't set. That said, I'm not at all sure how to do this with dovecot. I can likely do this with the python / shell commands without too much trouble, but how dovecot will handle it is something I need to understand a bit better...

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed here: 7c7b744 -- note that for this we just remove the ability to configure the system-default because it's confusing and default everyone to unlimited because it's simple.

Copy link
Member

@JoshData JoshData left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I appreciate the effort and the patience while I found time to review. I'm doing this from airplane wifi! (It's the only time I don't have other responsibilities!)

Since there are some confusing things, I'd like to try to reduce the complexity of the PR to just side step those possible confusions. I'm not sure if using the old maildir backend is a blocker. It would require several changes. Moving the db migration to our migration script is a blocker so that all migrations are in the same place.

Thanks!

@@ -117,6 +123,14 @@ def setup_key_auth(mgmt_uri):
if "admin" in user['privileges']:
print(user['email'])

elif sys.argv[1] == "user" and sys.argv[2] == "quota" and len(sys.argv) == 4:
# Set a user's quota
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is going to display not set a users current quota.

"box_quota": box_quota,
"box_size": sizeof_fmt(box_size) if box_size != '?' else box_size,
"percent": '%3.0f%%' % percent if type(percent) != str else percent,
"box_count": box_count,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this is the number of messages. Since we don't enforce a limit for this, it's a little confusing to report. I'd suggest removing it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's used in the management users.html template mostly. I'm not sure how useful this is to have there, but maybe @stylnchris could shed some light on this for us?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Number of messages can be removed. I do find it somewhat helpful but I do understand that its not part of the scope of adding quotas. This would be "another feature" I suppose.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The counting of messages has been removed here: c91db7b

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this file is mistakenly added. I think it duplicates a file two directories up. I didn't check if the files are different.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolved in 8a1e803

@@ -66,7 +66,33 @@ tools/editconf.py /etc/dovecot/conf.d/10-mail.conf \
first_valid_uid=0

# Create, subscribe, and mark as special folders: INBOX, Drafts, Sent, Trash, Spam and Archive.
cp conf/dovecot-mailboxes.conf /etc/dovecot/conf.d/15-mailboxes.conf
cp conf/dovecot/conf.d/15-mailboxes.conf /etc/dovecot/conf.d/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this change is mistakenly included along with the added file.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolved in 8a1e803

@@ -20,10 +20,12 @@ db_path=$STORAGE_ROOT/mail/users.sqlite
# Create an empty database if it doesn't yet exist.
if [ ! -f "$db_path" ]; then
echo "Creating new user database: $db_path";
echo "CREATE TABLE users (id INTEGER PRIMARY KEY AUTOINCREMENT, email TEXT NOT NULL UNIQUE, password TEXT NOT NULL, extra, privileges TEXT NOT NULL DEFAULT '');" | sqlite3 "$db_path";
echo "CREATE TABLE users (id INTEGER PRIMARY KEY AUTOINCREMENT, email TEXT NOT NULL UNIQUE, password TEXT NOT NULL, extra, privileges TEXT NOT NULL DEFAULT '', quota TEXT NOT NULL DEFAULT '0');" | sqlite3 $db_path;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dms00's question is right. The purpose of the default quota is going to cause confusion. What I see is that it's a default for new users. It's not a default for existing users (prior to this PR) or users without a quota set - they all get no quota. I think the way to resolve this is to just remove the default quota functionality to avoid confusion.

@MrMirhan's point is probably not solvable with this projects architecture because it's up to the mail client to do the send and then save the mail. As @stylnchris alluded to.

echo "CREATE TABLE aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 "$db_path";
echo "CREATE TABLE mfa (id INTEGER PRIMARY KEY AUTOINCREMENT, user_id INTEGER NOT NULL, type TEXT NOT NULL, secret TEXT NOT NULL, mru_token TEXT, label TEXT, FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE);" | sqlite3 "$db_path";
echo "CREATE TABLE auto_aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 "$db_path";
elif sqlite3 $db_path ".schema users" | grep --invert-match quota; then
echo "ALTER TABLE users ADD COLUMN quota TEXT NOT NULL DEFAULT '0';" | sqlite3 $db_path;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a database migration script. This needs to move there.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 1c66f69

echo "CREATE TABLE aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 "$db_path";
echo "CREATE TABLE mfa (id INTEGER PRIMARY KEY AUTOINCREMENT, user_id INTEGER NOT NULL, type TEXT NOT NULL, secret TEXT NOT NULL, mru_token TEXT, label TEXT, FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE);" | sqlite3 "$db_path";
echo "CREATE TABLE auto_aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 "$db_path";
elif sqlite3 $db_path ".schema users" | grep --invert-match quota; then
echo "ALTER TABLE users ADD COLUMN quota TEXT NOT NULL DEFAULT '0';" | sqlite3 $db_path;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I see is we're storing actually a string like "3G" (3 gigabyte limit), not an integer number of bytes. So this is ok.

if ! grep -q "quota_status_success = DUNNO" /etc/dovecot/conf.d/90-quota.conf; then
cat > /etc/dovecot/conf.d/90-quota.conf << EOF;
plugin {
quota = maildir
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a newer quota backend called "count" that is recommended. I'm reluctant to introduce a new feature that uses an out of date practice.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely gonna take a bit of investigation to make this happen.

The count backend:
https://github.com/dovecot/core/blob/076cf225aa8d18e8abb0cbdc04f68abdf1c62a5e/src/plugins/quota/quota-count.c#L8

The maildir backend:
https://github.com/dovecot/core/blob/076cf225aa8d18e8abb0cbdc04f68abdf1c62a5e/src/plugins/quota/quota-maildir.c

My C is quite rusty (no pun intended), so getting this in will be a bit of a trick. I have a path forward that would involve switching to the new backend, seeing what breaks, and then debugging it by looking at the C code and the database.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #2387 (comment) for why I think it may be fine to leave the maildir backend

@chadfurman
Copy link
Author

chadfurman commented Jul 9, 2024

@JoshData I've addressed these comments:

the migration issue here: 1c66f69

duplicate conf issue here: 8a1e803

CLI comment thing here: 4e9c564 (note that this was actually supposed to be a GET and a SET)

Removal of the default fixed here: 7c7b744 -- note that for this we just remove the ability to configure the system-default because it's confusing and default everyone to unlimited because it's simple.

The counting of messages has been removed here: c91db7b

The only comment that I haven't addressed yet is around switching to the new count backend -- this is definitely a more involved change, if only because I need to test it thoroughly to make sure there's no unexpected side-effects. As far as I can tell, it might "just work" but I need to test it and make sure. Anything that starts acting up, I'll have to diff the two C files I linked and try to understand why/how/where etc. Not opposed to this, but waiting to hear from you if this is a blocker or not before I put in the time.

@chadfurman
Copy link
Author

Looking at https://doc.dovecot.org/configuration_manual/quota/quota_maildir/ it appears that, given we are using the maildir format, unless we want to shift away from using the maildir format in the future (which may be a bigger lift?) then we are likely fine to keep with the quota = maildir backend. The count backend will give us more flexibility which we don't need at the moment, and so given that the maildir backend for quota is the most commonly used backend for quota with the maildir format I'm thinking that it is safe to move forward without changing this.

@chadfurman
Copy link
Author

Note for the community: If anyone has used this branch before the migration change, you'll want to set /home/user-data/mailinabox.version to 15 manually so as to skip the migration which was applied outside the migration system previously.

@chad-fossa
Copy link

@JoshData just checking in, here. Anything you still would like to see on this before merging?

@downtownallday
Copy link
Contributor

FWIW, here are a few items that came to mind when I was checking this out today.

  1. The new argument method added to cli.py::mgmt() seriously breaks other important commands that require POST (like 'user add', 'password', etc). Removing the method argument and it's use inside mgmt() works fine for me. I don't think it's needed.
  2. cli.py user outputs unlimited for quotas set to 0 however, cli.py user quota [email protected] outputs 0 for a quota set to 0, which is inconsistent.
  3. Including quota output in cli.py user output may break scripts that are already expecting a certain output format.
  4. It is confusing to allow a quota value in the admin UI to be just a number without a unit (M or G).
  5. The quota_grace setting should be "10%%" not "10%" becuse "%" is a special character to dovecot.
  6. I think it's important to add the quota_warning dovecot feature. Currently, when a user exceeds their quota, incoming messages are bounced by postfix (the sender receives an automated email from the server saying the recipients mailbox is full). The problem is the user never knows they're about to exceed their mailbox quota until it's too late. The quota_warning feature of the dovecot quota plugin can alert the user that they're close to exceeding their quota before they start losing messages. See an example here: https://unix.stackexchange.com/questions/363658/how-to-send-warning-with-dovecot-to-users-when-over-quota.

@chad-fossa
Copy link

chad-fossa commented Sep 7, 2024 via email

@downtownallday
Copy link
Contributor

I also think Drafts and Sent folders should not count toward quota.

Drafts because Roundcube automatically saves to Drafts every 5 minutes and you might be surprised that your draft wan't there if the message causes over quota, or an incoming message causes over quota. Roundcube does give a brief toast notification about the inability to save to draft, though.

Sent becuase Roundcube automatically saves to Sent and it's probably more important to most that sent messages are retained. As-is you can send messages in an over quota situation, but the sent message fails to copy to Sent folder.

quota_rule2=Drafts:ignore
quota_rule3=Sent:ignore

box_quota = 0
percent = ''
try:
dirsize_file = os.path.join(env['STORAGE_ROOT'], 'mail/mailboxes/%s/%s/maildirsize' % (domain, user))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of reading the "maildirsize" file directly, which is specific to the maildir quota backend, why not execute doveadm -f json quota get -u <email> and parse the json, which contains both quota ("limit") in KiB, current size ("value") also in KiB, and percent used ("percent"). The "STORAGE" type is what you want. "MESSAGE" is a count of messages. The json looks like this:

[{"root":"User quota","type":"STORAGE","value":"2","limit":"5120","percent":"0"},{"root":"User quota","type":"MESSAGE","value":"1","limit":"-","percent":"0"}]```

@stylnchris
Copy link

stylnchris commented Sep 7, 2024 via email

@chadfurman
Copy link
Author

I've updated to v70 and fixed a few bugs:

  • Removed the quota output from the user cli command
  • Removed the 'method' param that resulted from my misreading of the get_method function and which broke stuff
  • set quota_grace to 10%% since % is a special character

Open questions:

  • Is the "count" backend a blocker? I'm thinking the answer here is probably "yes", though I'm a bit nervous about this change
  • Do we want to ignore sent and draft folders? I'm thinking the answer here is "no" since that might cause headaches for system admins
  • Do we want quota warning? I'm thinking the answer here is "yes" so I'll probably need to figure this out, though I'd like to hear back from Josh before I make this change
  • Is it fine to allow quotas without units? I'm thinking this is small potatoes, though the UI could probably clarify what the default unit is if not specified. Mandating a unit might be a good move here, though we'd want to make it clear how to "remove" a quota (just delete it in the UI maybe?)

I'm going to hold off on all the above changes until I hear from @JoshData -- for now, this PR is in maintenance mode and I will fix bugs and work to update it to new versions periodically; however, new features and major changes will only be made if it has a high probability of resulting in this PR getting merged to mainline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants