Search history completions

[MareStare]

Before you begin

• I understand my contributions may be rejected for any reason
• I understand my contributions are for the benefit of Derpibooru and/or the Philomena software
• I understand my contributions are licensed under the GNU AGPLv3

• I understand all of the above

---

Closes #419

This is WIP, I'll add some details about the mechanics of the feature to the PR description once I'm ready, I'll request a review.

FIXME

• Selecting the history item should overwrite all input, not just the last term after the last coma
• Clicking on the history suggestion when the input is empty doesn't insert the item
• If there is already some input in the search field and you reload the page it keeps the input the same, if you focus the input it won't show tag completions
• Server-side suggestions render different labels. Instead, make the server return the necessary data in machine readable format without any display logic and render the results fully on frontend

TODO

• Improve UI as much as possible. Make sure history is enabled on all relevant search inputs
• Consider moving history lazy loading into the completions events subscriptions entirely
• Ctrl shortcuts, establish the behavior
• Add Ctrl+Enter/LMB shortcut that submits the form with the suggestion immediately, no matter what type it is
• Make request ID truly unique, add use idempotency token instead for retried requests
• Test input validation on backend
• Add a setting to hide the search history
• Handle big numbers formatting in image counts
• Test watching for changes, multiple tabs scenario
• Better cookies handling of missing values
• Test clicking into random places inside of the input
• Test various empty inputs with different number of whitespace characters
• Fix selection on refocus (e.g. completion in the middle)
• Load-test the history at the maximum number of inputs.
• Do some automated testing. Update existing tests
• Post-refactoring
• Eliminate unnecessary logs.
• After Release (+ after history items deletions / purging support) Prepare a demo for users. (Screencast keyboard shortcuts, maybe post on derpi itself?)

---

Search History

Added support for search history recording and completions to several inputs. When the input is empty and focused we display 10 last recently used queries. If the user types at least one non-whitespace character we display at most 3 history suggestions at the top based on the prefix match.

Updated the UI look of the completions that you'll see on various screenshots below. Of the not I replaced fat arrow character ⇒ with a thin arrow → for aliases, and to make that work I changed the font to the monospace font used in other places on the frontend.

I also did an extensive rewrite of the existing autocomplete code to make it more organized and type-coherent.

Data Attribute Config

Search history can be enabled for certain <input> or <textarea> elements with the new data attribute data-autocomplete-history-id={history_id}. The {history_id} should be some human-readable word that will be used a prefix in the local storage key, which is formatted as {history_id}-history. I enabled this only for the main search input in the header page with the history_id = 'search', which results in a search-history key in localStorage.

Several inputs can share the same search history if the use the same history_id.

Storage Format

The history is stored as a single JSON object with a flat list of history records of the following form in localStorage:

{
  "schemaVersion": 1,
  "records": ["foo", "bar", "..."]
}

The more recently used history records appear at the head of the array, and the least recently used items are at the tail. The ordering is maintained when new items are inserted or existing items are used repeatedly. I considered storing some additional useful info like created_at, updated_at, number of submits etc for each record, but then decided that to be an unnecessary overhead at least at this stage.

Shortucts

Form Submission

Clicking or pressing Enter on the history item submits the form right away. If the user wants to continue typing after the history item and prevent submitting the form, they can do so by using Shift+Enter or with a Shift+LMB. This is a big deviation from the tag's completions because I believe most of the time the intention of selecting the history item is to submit it immediately. This decision is arguable, but we can tweak this behavior as needed based on feedback.

An opposite shortcut Ctrl+Enter or Ctrl+LMB can now be used both with history and tag suggestions to submit the form immediately with the provided suggestion inserted.

Navigation

• Ctrl+Down - Jump to the next block of suggestions. This is intended to skip history suggestions and jump to the first tag suggestion.

• Ctrl+Up - Jump to the prev block of suggestions. E.g. if you are in the tag suggestions, that will jump to the last history suggestion. I don't see a real use case for this key binding, added it just for the symmetry.

Deviations from Current Behavior

Now there are tag suggestions when the cursor is right before the first character of the term:

---

Hovering over the item in the completions list doesn't move the selection. Instead, it renders a visual-only highlight. This behaviour was borrowed from VSCode's completion behaviour. The left side is the old implementation and the right side is the new implementation:

completions-comparison.mp4

Server-Side Completions

I enabled server-side tag completions on all inputs (including the main search). Plus I had to update its backend implementation to return a structured list of suggestions of the following form:

{
    "suggestions": [
         {
             // Name of the alias if this tag aliases some other one, can be absent or null
             "alias": "ts",
             // Canonical name of the tag, if this suggestion is for the aliases, then this is the alias destination
             "canonical": "twilight sparkle",
             // Number of images that have this tag
             "images": 9999
         }
    ]
}

where previous server-side suggestions API was returning this shape of the response:

[
    {
        "label": "twilight sparkle (image_count)",
        "value": "twilight sparkle"
]

The problem with that API was that it was doing the suggestion label rendering on the backend side. So we had to keep the frontend rendering of suggestions produced by the local top-50K autocomplete index in sync with the backend which we obviously failed to do for example, when we added the change to display aliased tags with an arrow.

So the new API can now be invoked with the vsn=2 parameter (analogous to the local autocomplete index), while the old API is left in code for backwards compatibility for some time to let users re-download the new frontend version.

Debouncing and Caching

Today, there is an annoying implementation behaviour of debouncing and caching for server-side completions. We debounce user input with a 300ms threshold, then send a server-side completions request and cache its result. However, if the user changes the input and then returns back to the cached query, the user still needs to wait for a 300ms debounce threshold before the cached results are displayed.

I fixed this behaviour and also moved this logic into a generic DebouncedCache class. It also manages an AbortController/AbortSignal for the server-side request, so that it can cancel it if the user typed some more characters after the server-side request was initiated. So this change eliminates this race condition, where previously the user could see stale sever-side completion results while they were typing. It isn't a critical bug though, but still nice to see it fixed.

Here is an example of how it works. You can see here how two requests were aborted because I typed in the middle of the active request:

Some debug-level logs are output when this condition is experienced (ignore the refresh logs, they were temporary used for debugging):

HTTP Client wrapper

I added a new http-client.ts file with a generic HTTP Client implementation that may be useful for any request from the frontend. It provides a nice API with inbuilt error handling, exponential retries with backoff and jitter, escaping of URL query parameters, and it also adds some useful diagnostic info into the request headers like the X-Request-Id, X-Retry-Sequence-Id and X-Retry-Attempt.

Other Changes

---

• Bumped TS standard lib types to ES2021 to get support for Error causes

• Added autocomplete to the /tags page. Doesn't record history. Conditionally enabled (disabled by default) for now just like the main search for images in the header.
  

Image Count Rendering

I've decided to format the number in full with groups of 3 digits separated by a short space. I think this way of rendering looks nicer than using M / K contractions, since this way all the digits are aligned and it's easy to see the difference between the order of numbers between different tags. I think this way of formatting numbers in full may scale even to 100 000 000 images. Also, this way of displaying the image count will make derpibooru's completions UI style more unique (less of a copy-paste from danbooru).

However, I understand that this is subjective and given enough opinions, we can change the formatting easily.

UI Extremes

Overly long suggestions are truncated with an ellipsis. The overall max width of the completions box can be at most 70% of the viewport.

Here is the demo of handling extreme suggestion / image counts lengths:

extremes-demo-1080p.mp4

Note that the position of the completions box stays static when browser is resized. It also works this way in the implementation in master, and I don't think it's a big issue. The box is rerendered next time the user types something. I haven't found a simple solution for how to fix this yet.

Future Extensions

These changes aren't implemented and are left for the scope of the followup PRs:

• Support deleting items from the history and purging it
• Fix local settings state handling. Remove cookies and the save button there
• Integrate the inserted suggestion into the user's text undo stack. I.e. Ctrl+Z should cancel the previous inserted suggestion. This requires document.execCommand, but that API is deprecated with no alternatives (!)
• Prioritize canonical tag in server-side completions. I.e. basically we need the solution implemented in Fixed aliases still displaying for namespaced tags, fixing the actual cause of the issue inside local autocomplete #420, but also implemented in the server-side completions
• Support pinning (📌) history items to make sure they are always listed at the top of suggestions when the input is empty