CUBE directory-first considerations/implications

[rudolphpienaar]

The existing CUBE instance that is in production as of summer 2023 largely bounds the data-behavioural space requirements (i.e. needs that clients of CUBE have, or alternatively the experience the ChRIS provides clients). In other words, the current CUBE offers (albeit inconsistently) a largely feature-complete set of behaviours.

However, the effects of a subtle, but fundamental disconnect between the philosophy of CUBE and its current internal design are now quite readily apparent. Philosophically, ChRIS is a directory based system -- ChRIS applications function on directories: input files are presented in a directory; output files are grouped in a directory. The compute flow data structure is organised into directories (a Feed -- the fundamental grouping of a computational flow) is really nothing more than a set of nested directories; users can "upload" new files into CUBE and such an upload is captured in a directory.

Historically, however, the internal CUBE data-view has been pervasively file first and this file first paradigm is reflected in the entire model of database design. Moreover, the CUBE file-first approach was also tightly coupled to its storage model, and files had to exist in storage to be registered to CUBE. At design time, the possible shortcomings of this approach were not immediately apparent.

However in 2023 these cumulative effects of these shortcomings are becoming evident. Generally speaking, two fundamental issues exist: (1) inconsistent attempts at "modeling" directories that are strongly contextual ; (2) duplication of files throughout the whole system (i.e. no concept of hard/soft linking).

This discussion document proposes a new redesign set of requirements, organised about explicit representations of directories within CUBE.