Skip to content

Latest commit

 

History

History
116 lines (92 loc) · 7.13 KB

ROADMAP.md

File metadata and controls

116 lines (92 loc) · 7.13 KB
Raw

Roadmap

The full span of functionality that Datacore will eventually encompass.

Views

Views in Datacore are editable and no longer use the Dataview Query Language. You can still write complex-looking statements / columns using the Dataview Expression Language (a subset of the query language which supports math, string operations, list operations, and so on).

  • Table: The standard columnar table view.
  • List: The standard list view, which just lists elements.
  • Embed List: A special list which just renders a list of embedded pages
  • Card: A card-based view which supports a title and card content.
  • Grouped: A special view which allows for splitting query results into subgroups, and render a separate table / list / card view for each group.

Each view will have the following general functionality:

  • Adding / Removing Fields: You can easily add/remove fields. Autocompletion is supported.
  • Picking How Fields Render: You can either render field values directly, or choose how to render them (as progress, as stars/ratings, as one of a fixed set of values, etc.).
  • Sorting by Expression / Field: You can sort on arbitrary fields.
  • Filter by Expression / Field: You can filter on arbitrary fields.
  • Inline Editing: Fields can be directly edited by clicking on them; most editing is just textual inline editing, though ratings / dropdowns will have special behavior.
  • Pagination: For performance reasons, all views are paginated (configurable to 10, 20, 50, 100, 250 or a custom granularity).
  • Live Reloading: Views automatically update whenever the result of their query changes.
  • Embed Support: Most views should play nice with embeds (images, PDFs, other markdown files).
  • Add New Element: Views will support adding new pages automatically via a template; this template can be configured or use an existing Obsidian template file.

Additionally, there is potential for a few extra nice features:

  • View Pages: Special page types which are just Datacore views and which can be put into the side bar or as regular pages.

Data Input

Obsidian has some native support for editing metadata at the page level, and has recommended that Datacore move away from inline fields (since they are much more bespoke). Inline fields will still be supported for users who prefer it, but the following ways to add metadata will be added:

  • Inline YAML: You will be able to place YAML objects anywhere in the document and mark it inline to have it add additional metadata to the page / section.
  • Inline Objects: You will be able to define YAML codeblocks as "objects", which are searchable independent of the page they are on (for example, "exercise" objects).
  • Sections: Sections can be treated as objects and are directly queryable.
  • Tasks: Tasks can be directly queried and are more performant; metadata can be added using the Task plugin emoji or inline fields.

Comprehensibility

Dataview can be very frustrating to debug if it fails or lags; some new features are being added to make things more clear.

  • Auto-limit: Queries will autromatically be limited to the top 50/100 results to prevent serious rendering lagging. Rendering large objects (such as markdown pages) will show abbreviated views that are much more performant.
  • Query Time: All views will have timing information showing total time it took to query, render, and update.
  • Indexing Activity: A new indicator on the bottom of the screen will show how indexing is going visually and average index time per file.
  • Log: Datacore will log any actions it takes (re-renders, indexes) with timing information for debugging. This data will be available in a simple log view which supports redacting file names for safely posting to support forums.

Indexing

The kind of metadata that Datacore will index.

  • Markdown Pages: As expected. Should index sections, tasks, frontmatter, inline fields, and so on.
  • Inline Fields: Inline fields will be opt-in instead of on by default, so performance sensitive vaults can opt for inline YAML blocks or frontmatter instead, which are faster to parse.
  • Sections: Every section in markdown pages will be parsed, including the implicit "root" section (the first part of the page before any section headers).
  • Tasks: Tasks inside markdown pages will be parsed, along with various metadata from the Tasks plugin.
  • Canvas Files: Canvas files will be indexed; each text node will be available to load, as well as all linked pages inside of the file.
  • Images: Basic metadata - name, dimensions, format.
  • Videos: Basic metadata - name, dimensions, format.
  • PDF Files: Basic metadata - name, pages, format.

Javascript API

  • React-based Views: Javascript swaps to using React for views, which eliminates the "flickering" during re-renders, and also allows for access to Datacore visual components.
  • Index Queries: Skip the "query rendering" step and directly access the backing database, which allows for much faster queries in exchange for less flexibility.
  • Script Imports: Support require() inside of codeblocks; note that this will generally only works for scripts inside of the vault.

Technical

Background pre-requisites; this is generally not user facing.

  • Rate-limiting Background Importer. Allow specifying a rate at which new files should be ingested, as well as how long in between consecutive file updates. Starting a vault with Datacore should introduce NO lag (by slowing down throttling).
    • This can generally be computed by estimating how long the average file import takes (potentially correlated with file length), and then scheduling file imports to reduce expected CPU time to X%.
  • Performance Controls: Support choosing which files to index, what file types to index, and what subfeatures to index all to speed up performance.
    • All of these features will have timings which indicate how long they take, on average.
  • Throttle Controls: Set a percent allowance of how much CPU to use up for all Datacore tasks.
  • File-based Persistence: IndexedDB appears to have hard metadata limits and does not work for large vaults (where caching is most desparately needed). An alternative multi-file database (of sorted JSON documents), like LowDB or LevelDB, will work better.
  • Rendering Ratelimiting: Datacore should never crash Obsidian due to rendering a huge query. All queries have implicit limits which must be EXPLICITLY turned off. Renders automatically pause after 10 seconds pass.
  • NPM Publishing: Publish the library and indexer to npm; other plugins can directly depend on it without requiring users to install Datacore itself.

In What Order Will This Be Done?

Technical tasks will be completed first as they are generally required to implement everything else. Frontend improvements take priority over extra features; Datacore is principally concerned with feeling like a natural experience inside of Obsidian, closer to Notion tables.