Gtk4 + New Core + Listview Megaport

[diegogangl]

---

This PR rebases and solves the conflicts in @ranchester2 's Gtk4 port (sometimes succesfully). It will also include my work on listviews and will finally plug the new core completely into the new UI.

🔥 Note that everything in this branch is broken and on fire. It will take quite a while before this is even testable. 🔥

Current Status

Will fill this up more as I go along.

• The Gtk4 port is complete
• The new widgets are mostly done, except for the horde of hidden bugs and regressions
• The port to the new core is missing several actions, plugins and backends (also tons of bugs probably)
• I plan on going back to the new core once everything is plugged in to streamline and simplify things (look at the cleanup section below)

Other Changes

• The Tags section can now be collapsed, like saved searches. They also look more like the sidebar items in Gnome Builder
• Tags can now be renamed without destroying their parent relationship or doing "weird stuff"
• Saved searches have a name that is independent of the query
• The saved search query can be edited in a new dialog
• The saved search context menu is different from the tags too
• Tasks can be dragged over tags in the sidebar to tag them
• Multiple tags can be selected in the sidebar, the selected tasks are the ones that have all the selected tags. (eg. if you select "@errands" and "@Prio1", only tasks that have both of those tags show up).
• There is a new sort menu above the tasks list
• Tasks can be sorted by added/modified/start/due dates, alphabetically and by tags

Clean-up and other todos

• Add new unit tests for: replace_tag and remove_tag (in tasks)
• Use TagPill in the sidebar
• Stop using ScrolledWindow for taskpane and sidebar, they are already inside ScrolledWindow
• Put unwrap() in a utils somewhere
• Add proper icons for due and start date
• Change the way color is stored in the file, document and add versioning.
• Remove all the stupid [:2] hacks for colors
• Move the code to add row color to a method
• Look into the padding of the rows in the task list
• Consider extending Gio.ListModel in the store classes directly
• Make tags in tasks a set instead of list (duh)
• Use bind() in task pane
• Figure out how to make bg_func in plugins now that we use css
• Figure out how to connect signals for tasks being modified (probably through the model?)

Fixes, etc for Github

fixes #307
fixes #874
closes #308
fixes #904
fixes #817
fixes #302
fixes #240
fixes #420
fixes #123
fixes #859

[diegogangl]

For anyone following this: the sidebar is really coming together. It loads and displays all tags, does the context menu and the tag editor also works 100%. Drag and drop parenting mostly works, but unparenting does some funny things.
Also, I haven't figured out how to do the expander arrows yet. This is not a use case that was considered in the API, so we'll have to find ways around some issues. Might take some ideas from ranchester's code too 🤔
Task counts in the sidebar also work, and the code will be able to support updating counts according to the selected pane.

One interesting effect from using a listview is that now all tags are visible. Even if they have zero tasks, I'm wondering if this should be fixed or we should keep the new behaviour. It can be useful to always be able to select a tag and use the quick add input (even if you completed all the tasks in it). @nekohayo any thoughts?

Next step is to figure out how to fix unparenting and the expanders, then clean the unholy mess that is the code (and the commits). After that, is on to saved searches. Which will probably take a while since they need a new editor and menu.

[nekohayo]

One interesting effect from using a listview is that now all tags are visible. Even if they have zero tasks, I'm wondering if this should be fixed or we should keep the new behaviour. It can be useful to always be able to select a tag and use the quick add input (even if you completed all the tasks in it).

Hmm, I'm not against this per se, and this is partially how 0.3.x used to work outside workview/actionable view; if you were not in workview mode, it would show all the tags, and in workview mode it would only show the tags set to show "only in workview mode"... but I know that what you're talking about here is something different: also showing tags with 0 tasks associated with them.

Ideally I'd prefer if the user has some control over that (like maybe 2 or 3 states that can be controlled from some pref widget in the sidebar or the hamburger menu?), though it could be argued that if a user doesn't want to see a particular tag with 0 tasks in it, either they need to create tasks for it or they need to delete the tag, perhaps...

In any case, always showing all tags "even unused ones" will force you to fix issue #605 as you will quickly go insane with the amount of erroneous/duplicate tags that get created and never get cleared ;)

[diegogangl]

Ideally I'd prefer if the user has some control over that (like maybe 2 or 3 states that can be controlled from some pref widget in the sidebar or the hamburger menu?)

Do you mean the panes/tabs?

In any case, always showing all tags "even unused ones" will force you to fix issue #605 as you will quickly go insane with the amount of erroneous/duplicate tags that get created and never get cleared ;)

Might be worth keeping until I get to the editor then :P

Note to my future self: add a filter to toggle which tags are visible (also for panes)

[diegogangl]

Another quick update: I've finished the saved searches section of the sidebar. The only missing thing is the task count, we'll have to figure out how to do that efficiently at some point. Next up, I'm going to add basic click handlers for all the things in the sidebar. This means bumping the minimum Gtk to 4.6 since BitsetIter was broken for Python before.
After that, it's time to organize and squash commits. Then it's onto the task list and trying to plug everything together.

[diegogangl]

Another update:

I managed to squeeze more stuff in the week, mostly copy-pasting previous stuff. Selections in the sidebar are now working. This means the minimum Gtk now is 4.6, previous versions will crash as soon as you click on a tag.
I've also squashed and reorganized all the commits regarding the sidebar. I've just started working on the task panes, and I've got the list view to show up. Tasks are unfiltered and unsorted and nothing is really working yet though.
Next up is adding all the filters and sorters and plugging them to the tasks. Also making the tasks open the editor and show the right click menu. Then, extending the task pane class so it can take filters. Maybe also going back to the sidebar to add filtering for tags and clean up the code some more.

Screenshot of the work so far (missing a ton of polish)

[diegogangl]

Welp, couldn't do much during the week but still managed a few things. Dismissing, closing and re-opening tasks with the keyboard shortcuts is working with the new core. I also added a css class for closed tasks.
The editor is working-ish, still plenty of bugs and things to change. How on earth did we end up with two unused callbacks in taskview? Anyways, I've also hammered in a few more ui things and cleaned up the code in the editor class a bit.

The crisis around here is kicking into high(er) gear, so I had to rush to buy things and ended up running out of time for the weekend. I'll try to complete the editor, fix the delete tasks and batch tag dialogs, and re-organize the commits next weekend. Then it's onto sorters and filters.

[diegogangl]

Another update: let's see if I can remember everything. Squashed and organized some of the commits. Quick add input is now working with the new core. The checkbox is now toggling active/done status. The editor is mostly stable now. Added the sorters and filters. Clicking on a tag(s) in the sidebar works now. The sorting menu in the taskview is also working. The tasks context menu shows up but every option is a trap that slides you into a parallel universe.

Up next:
Fix the context menu. Make search work again. Make panes work again. Make the title of the task list change. Make the all tasks, and no tags buttons work. Add a filter for no tags too btw. Hide the separator in the list if the task has no tags. Make everything update (spam items-changed?). Make saved searches work with new core. Also make a task filter for a saved search (I think?).
Make saved searches in the sidebar apply the filter.
After that is time to jump into the preferences dialog

[diegogangl]

Another update:

Searching and saved searches are in. Panes are working, the task pane title is changing. The task context menu is working
Some small fixes. "All tasks" is working too.

I'm going to take a week or two to keep the burnout at bay and do something else. Once I get back I'll continue with:

• Porting the plugins
• Defining what to do with backends (and porting the caldav backend)
• Use new core for purging tasks and updating actionables
• Use bind property in the task pane
• Add the workaround for filters (expand all tags)
• Plug the background row setting to the task pane. Find some way to make the list update
• Enabling normal read and save from the same file (and unplug the old datastore)
• Figure out how to select a task when right clicking

[nekohayo]

Thank you for your very extensive work thus far, take care and I will look forward to seeing further action!

Given how big this is, I would say, try to get it to the point where it 90-99% works (without critical issues) rather than 100%, to be able to merge somewhat early and let other contributors help you fix the remainder with a checklist, if possible :)

[diegogangl]

Thank you for your very extensive work thus far, take care and I will look forward to seeing further action!

Given how big this is, I would say, try to get it to the point where it 90-99% works (without critical issues) rather than 100%, to > be able to merge somewhat early and let other contributors help you fix the remainder with a checklist, if possible :)

Thanks for checking in :D
Yeah, I'm definitely leaving some stuff rough to complete this faster. There's a crazy amount of performance and code improvements that can be done. Will update the 0.7 tracking issue, and maybe split it into sub issues later.

Fixed some bugs and visual issues this weekend. Started to make the task list use bind() for almost everything (had to add a ton of properties to tasks for that, egh), added a way to notify the task list when tags change (so they update immediately), fixed collapse/expand all. Also organized all the previous commits.

Up next, bring back ranchester's tag pill widget so we can bind everything (in the sidebar too), emit expand all when clicking on something in the sidebar, add a filter for tasks with no tags, fix dark mode and colored rows in preferences

[diegogangl]

Managed to get all that done, next up is plugins

[diegogangl]

Plugins are ported, probably full of bugs tho
Fixed a couple of simple things in the ui too.
Next up, going back to the ui big time. I want to get parenting working properly once and for all. I also want to clamp the listview horizontally so long task names get ellipsized rather than adding a horizontal scrollbar.
I also have to move task counts to the tags themselves in a gprop so they can be used with bind(). Also I need to check why tags aren't working with fill_with_samples().

Also there are some weird issues with the task view when using fill_with_samples and expanding the tasks (they go away though).

Besides the bug fixing and workaround-ing, the only two big missing pieces are backends and recurring

[diegogangl]

Managed to push forward some more. Task counts now update properly, the tasklist hides the expander arrows and has better padding.

Remaining issues:

• Backends
• Filtering deep subtasks when changing tags
• Tasklist still not refreshing when adding a parent or subtask, or when dragging tasks over a tag

[diegogangl]

Hi guys, sorry this is still moving at a glacial pace -_-
I only have a few mins per weekend to pour on this. And I'm hitting a wall with gtk lists.
Backends are working (at least caldav is initializing). Also noteworthy: the old datastore and requester are not being initialized anymore \o/

[diegogangl]

Latest changes:

• Added some useful functions to task pane
• Found a somewhat reliable way to trigger a refresh in the listview (though it's not working for already expanded parent tasks sigh)
• Ported the code for repeating tasks
• FIxed a crash and some bugs in the RMB menu

@zeddo123 I took the liberty of doing a quick n dirty port so we can land this in master ASAP. Feel free to hack on it once it's merged, or replace it with the new stuff you've been working on

The last big problem is proper filtering. ATM, subtasks that whose parent task isn't expanded "dont exist" in the model, so when a filter (a tag) is applied they are not taken into account.

[nekohayo]

Tested the current state of the branch tonight; it at least launches on Python 3.12 / Fedora 39, and overall works pretty well! I had to clear my tmp/ subdirectory to ensure I was indeed loading the 0.6 data file from my samples directory, instead of the previously-incorrectly-written file from the last time I ran the branch, so that I could see the tag's hierarchy preserved.

---

In the laundry list of issues I reported above, this issue was marked as done:

• 🔰 "Previously set tag colors are not imported / preserved"

...but it has a weird bug going on still. The colors are imported, but they are rendered incorrectly every 1 out 2 startups.

In the GTG 0.6.x data format, I had this:

<tag id="some_ID_hash" name="opensource" color="3465A4" nonactionable="False"/>

Color #3465A4 is blue, as you can see, but in your branch it gets rendered as some sort of green on the 1st startup:

In the resulting gtg_data.xml, the line showed up like this (same color code, but with a hashtag):

<tag id="some_ID_hash" name="opensource" color="#34a465"/>

But here's the crazy part... if you restart GTG, then it shows up as the correct blue, and the other tags also show up with the correct colors… until you restart, then it will flip all the colors around again… and when you restart yet again it will show correct colors, etc. It really seems to alternate on every launch, providing correct colors 50% of the time and incorrect colors the other 50% of the time.

---

Here are a few other "real bugs" I found now, that I hadn't noticed last time:

• ⚠️ The nonactionable="False" tag properties are not preserved in importing from 0.6 to your branch, as you can see in the markup difference above.
  That one might be a blocker since it touches the file format?
• ⚠️ If you have a (sub)tag selected in the sidebar, have the "Open" tasks view selected, and drag a subtask onto out of its parent and onto the general view/canvas (to make it a toplevel/non-child), it will duplicate the task.
  Uh oh! I don't think we're supposed to be able to duplicate items just by drag-and-drop, very accident-prone.
• Sidebar tag gets deselected, view resets to "All Tasks" when dragging a child (subtask) onto another task in "Open" tasks view mode #1017
• Reparenting children tasks is generally broken, and doing so before and after switching between Open and Actionable views will lead to a freeze hang #1018
• Setting yearly recurring tasks from the Task Editor's menu sets them as repeating monthly #1015

Newly found minor bugs:

• 🔰 When the searchbar is active with an existing search term, switching between the Open / Actionable / Done tabs does not take the search query into account, so results are unfiltered/nonsensical.
• Sorting results often seem to have semi-random/nonsensical ordering.
  Even in "Actionable" view mode (to take out the possibility of the "Open" tasks view's hierarchy influencing), sorting by Start date, or by Title (which would make it really obvious if something is not alphabetical) shows results in a seemingly random order. Sorting by Tag doesn't quite seem to work either? Sorting by Due date seems to mostly work, but it's a bit confusing/hard to tell because I keep visually confusing it with the Start dates in the view, which brings me to...
• 🔰 "Actionable" and "Closed" views should not show Start dates
  (it's a regression compared to 0.6, and it visually overloads those views, particularly the Actionable view that is presumed to be only "tasks that don't start in the future" anyway)
• Selected view mode (Open / Actionable / Closed) is not remembered & restored on startup #1019
• Sorting order not remembered & restored on startup #1020
• Sorting does not set an arrow/triangle (such as ▼/▲) to indicate the sorting order in the popover menu #1021

---

I presume there are probably some other bugs lurking around, but probably not showstoppers and I will probably have to do another round of close-up inspection to find them (and arguably, maybe it's best to do that only after once we fix the existing inventory of bugs found in the previous comment and this one, bugfixing that is presumably only going to happen after merging?)

[nekohayo]

New issues discovered now that the nonactionable attr for tags gets imported thanks to commit f37d842 :

• "Actionable" tags are not preserved as such in the UI (in the sidebar tags' properties);
  If you go to the "Actionable" view mode, no tasks will be shown, because no tags are actually set as actionable...

• Even if you use the properties UI to edit a tag and mark it as "Actionable", the tag will be hidden from the sidebar when you switch to the "Actionable" view mode

Update:

• The actionable status gets imported and applied incorrectly for some tags but not others, due to nonactionable="True" being assumed

comparing the data files with Meld, I see what's wrong now: the file format used in 0.5-0.6 did not explicitly set nonactionable="False" in some cases (I presume, tags where this was never toggled?), and the latest code assumes the default to be nonactionable="True" which is not true! The resulting XML file has nonactionable="True" in tags that had nothing defined before, but we should probably expect nonactionable="False" as the default instead

[nekohayo]

Edited the latest comment above to clarify the currently inverted nonactionable flag.

An additional issue that was found in recent days is:

• ⚠️ 100% CPU usage busy loop on 1st run with large data file from GTG 0.6

…with large datasets (such as my anonymized personal dataset from december 2021), on the 1st startup (where the import and presumably conversion from 0.6 -> 0.7 happens), GTG will enter a busy loop eating 100% of one of the logical CPUs.

The cProfile for this led us to generate this flameprof flame chart:

The open_task method in GTG/gtk/browser/main_window.py gets called and hits the except KeyError: return True silent exception, and it continues indefinitely, wasting CPU.

[nekohayo]

Task counts in the sidebar also work, and the code will be able to support updating counts according to the selected pane.

There's a bug with that currently: #1023

After going to the Actionable view within a task, and marking a bunch of tasks as done, task count does not match reality
As shown below, there sure aren't 40 tasks shown in that current view combination (the counts for the Open and Closed views are different, too):

[piio]

Hi,

I try to test new GTG develop version. I use https://github.com/diegogangl/gtg.git - branch all_ports.

GTG opened correctly with start data, I noticed that I can't delete tasks:

• removing task on parent task return error, but after closing GTG and opening again task is removed:
  Context: Global generic exception

Traceback (most recent call last):
  File "/home/piio/.archive/src/gtg-newcore-all-ports/.local_build/install/lib/python3.12/site-packages/GTG/gtk/browser/delete_task.py", line 40, in on_response
    self.on_delete_confirm(tasklist)
  File "/home/piio/.archive/src/gtg-newcore-all-ports/.local_build/install/lib/python3.12/site-packages/GTG/gtk/browser/delete_task.py", line 49, in on_delete_confirm
    self.ds.tasks.remove(task.id)
  File "/home/piio/.archive/src/gtg-newcore-all-ports/.local_build/install/lib/python3.12/site-packages/GTG/core/tasks.py", line 914, in remove
    item = self.lookup[item_id]
           ~~~~~~~~~~~^^^^^^^^^
KeyError: '2d427a77-3077-4277-904c-073fcfcb4842'

Software versions:

• GTG (debug version) v0.5-597-g9bc092d2
• CPython 3.12.1 (main, Dec 18 2023, 00:00:00) [GCC 13.2.1 20231205 (Red Hat 13.2.1-6)]
• GTK 4.13.5, GLib 2.79.1
• PyGLib 3.46.0, PyGObject 3.46.0
• Linux-6.8.0-0.rc1.12.fc40.x86_64-x86_64-with-glibc2.38.9000

• removing sub task removes task from list but after closing GTG and opening again task is still visible

Best regards,
Piotrek

[nekohayo]

Thanks for catching an extra bug or two, @piio!

---

From my side, here are today's bugs I mentioned via chat, but in detail / better organized here...

General parenting bugs (including ones reported before) - refiled as #1018 :

• ⚠️ If you deparent a child by dragging (i.e. dragging it between tasks) in a view of Open tasks with many tasks (ex: a hundred), the app freezes (it did that previously, but for a moment yesterday we thought maybe this was due to the infinite loop that happened on startup… well nope, that was not the cause, it was just in addition to the hang!)
• ⚠️ If you drag a child out of a parent, the selected tag in the sidebar resets to "All Tasks" view
• ⚠️ If you drag a standalone task into a parent (turning the task into a child), the selected tag in the sidebar resets to "All Tasks" view
• ⚠️ If you drag a child from one parent to another task (re-parent), it duplicates it in the view

Other bugs:

• ⚠️ When switching between tags selection in the sidebar, the existing filtering search query in the searchbar is not applied to the newly chosen tag view
  Piggybacked on Clicking / Switching between saved searches is broken (does not re-set the search field, stacks the searches, or breaks filtering in general) #390

• ⚠️ Third-level children are not shown when selecting the 2nd level parent
  (ex: if you have nested tags where the hierarchy goes like this: travel -> conferences -> guadec, and you have task foobar tagged only @guadec, when you click the conferences tag in the sidebar, task foobar (tagged guadec) won't be shown in the view. At least, that is the case with my dataset of personal tasks imported from GTG 0.6
  Migrated to Third and fourth-level children are not shown when selecting the 2nd/3rd level parent tag filter #1024

• Sorting by title (alphabetically) works only partially
  It will get confused by any special characters (or numbers), such as task titles that start with @, [, (, "; it will also get confused by any accented characters (ex: "à" and "é" will get sorted at the end while "a" (and "e") gets sorted at the beginning); maybe there is a standard python function/lib for that... but how would you handle non-latin languages, such as arabic, persian, chinese/japanese/korean, klingon, etc.?

• Sorting by tags seems to behave a bit unpredictably in most cases
  Many times I see tasks with the same colored single tag... not grouped together / separated by other tasks with unrelated tags.

[nekohayo]

I have now refiled all of the remaining noteworthy issues I reported above, into the main official bug tracker, most of them with the "regression" label and generally targetted to the 0.7 milestone if they are core UX regressions.