WIP

Signed-off-by: Andrew Stein <[email protected]>

.github/workflows/build.yaml

@@ -168,8 +168,6 @@ jobs:
                       packages/perspective-viewer-d3fc/dist
                       packages/perspective-viewer-datagrid/dist
                       packages/perspective-viewer-openlayers/dist
-                      packages/perspective-esbuild-plugin/dist
-                      packages/perspective-webpack-plugin/dist
                       packages/perspective-cli/dist
                       packages/perspective-workspace/dist
 
@@ -947,12 +945,6 @@ jobs:
             - run: pnpm pack --pack-destination=../..
               working-directory: ./packages/perspective-cli
 
-            - run: pnpm pack --pack-destination=../..
-              working-directory: ./packages/perspective-webpack-plugin
-
-            - run: pnpm pack --pack-destination=../..
-              working-directory: ./packages/perspective-esbuild-plugin
-
             - run: pnpm pack --pack-destination=../..
               working-directory: ./packages/perspective-jupyterlab
 

.gitignore

@@ -112,7 +112,7 @@ packages/*/cjs
 cppbuild
 docsbuild
 tools/perspective-build/lib
-packages/perspective-esbuild-plugin/lib
+tools/perspective-esbuild-plugin/lib
 rust/perspective-python/perspective/nbextension/static
 rust/perspective-python/perspective/labextension
 boost_*.tar.gz
@@ -215,7 +215,6 @@ results.debug.json
 
 tools/perspective-build/lib
 docs/.docusaurus
-packages/perspective-esbuild-plugin/lib
 docs/static/blocks
 test-results/
 playwright-report/
@@ -252,3 +251,4 @@ rust/perspective-python/LICENSE_*
 rust/perspective-viewer/docs/exprtk.md
 rust/perspective-server/docs/lib_gen.md
 rust/perspective-client/docs/expression_gen.md
+docs/book

docs/book.toml

@@ -0,0 +1,39 @@
+#  ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
+#  ┃ ██████ ██████ ██████       █      █      █      █      █ █▄  ▀███ █       ┃
+#  ┃ ▄▄▄▄▄█ █▄▄▄▄▄ ▄▄▄▄▄█  ▀▀▀▀▀█▀▀▀▀▀ █ ▀▀▀▀▀█ ████████▌▐███ ███▄  ▀█ █ ▀▀▀▀▀ ┃
+#  ┃ █▀▀▀▀▀ █▀▀▀▀▀ █▀██▀▀ ▄▄▄▄▄ █ ▄▄▄▄▄█ ▄▄▄▄▄█ ████████▌▐███ █████▄   █ ▄▄▄▄▄ ┃
+#  ┃ █      ██████ █  ▀█▄       █ ██████      █      ███▌▐███ ███████▄ █       ┃
+#  ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
+#  ┃ Copyright (c) 2017, the Perspective Authors.                              ┃
+#  ┃ ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ ┃
+#  ┃ This file is part of the Perspective library, distributed under the terms ┃
+#  ┃ of the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). ┃
+#  ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
+
+[book]
+authors = ["Andrew Stein"]
+language = "en"
+multilingual = false
+src = "md"
+title = "Perspective"
+
+
+[output.html]
+# theme = "my-theme"
+# default-theme = "light"
+# preferred-dark-theme = "navy"
+# smart-punctuation = true
+# mathjax-support = false
+# copy-fonts = true
+additional-css = [
+    "md/perspective.css",
+    "node_modules/@finos/perspective-viewer/dist/css/themes.css",
+]
+# additional-js = []
+# no-section-label = false
+# git-repository-url = "https://github.com/rust-lang/mdBook"
+# git-repository-icon = "fa-github"
+# edit-url-template = "https://github.com/rust-lang/mdBook/edit/master/guide/{path}"
+# site-url = "/example-book/"
+# cname = "myproject.rs"
+# input-404 = "not-found.md"

docs/docusaurus.config.js

@@ -69,7 +69,7 @@ const config = {
         defaultLocale: "en",
         locales: ["en"],
     },
-    plugins: ["./plugins/perspective-loader"],
+    // plugins: ["./plugins/perspective-loader"],
     presets: [
         [
             "classic",

docs/md/SUMMARY.md

@@ -0,0 +1,44 @@
+# Summary
+
+# Overview
+
+-   [`Table`](./explanation/table.md)
+    -   [Schema and column types](./explanation/table/schema.md)
+    -   [Loading data](./explanation/table/loading_data.md)
+    -   [`index` and `limit` options](./explanation/table/options.md)
+    -   [`update()` and `remove()` streaming methods](./explanation/table/update_and_remove.md)
+    -   [`clear()` and `replace()` start-over methods](./explanation/table/clear_and_replace.md)
+-   [JavaScript](./explanation/javascript.md)
+    -   [Module Structure](./explanation/javascript_module_structure.md)
+    -   [Build options](./explanation/javascript_builds.md)
+-   [Python](./explanation/python.md)
+
+# Getting Started
+
+-   [Construct a `Table`](./how_to/table.md)
+-   [Rust](./how_to/rust.md)
+-   [JavaScript](./how_to/javascript.md)
+    -   [Installation via NPM](./how_to/javascript/installation.md)
+    -   [Importing with a bundler or without a bundler](./how_to/javascript/importing.md)
+    -   [`perspective` data engine library](./how_to/javascript/worker.md)
+        -   [Serializing data](./how_to/javascript/serializing.md)
+        -   [Cleaning up](./how_to/javascript/deleting.md)
+        -   [Hosting a `WebSocketServer` in Node.js](./how_to/javascript/nodejs_server.md)
+    -   [`perspective-viewer` Custom Element library](./how_to/javascript/viewer.md)
+        -   [Theming](./how_to/javascript/theming.md)
+        -   [Loading data from a `Table`](./how_to/javascript/loading_data.md)
+        -   [Loading data from a virtual `Table`](./how_to/javascript/loading_virtual_data.md)
+        -   [Saving and restoring UI state](./how_to/javascript/save_restore.md)
+        -   [Listening for events](./how_to/javascript/events.md)
+-   [Python](./how_to/python.md)
+    -   [Installation](./how_to/python/installation.md)
+    -   [Loading data into a `Table`](./how_to/python/table.md)
+    -   [Callbacks and events](./how_to/python/callbacks.md)
+    -   [Multithreading](./how_to/python/multithreading.md)
+    -   [Hosting a WebSocket server](./how_to/python/websocket.md)
+    -   [`PerspectiveWidget` for JupyterLab](./how_to/python/jupyterlab.md)
+    -   [Tutorial: A `tornado` server with virtual `perspective-viewer`](./tutorials/python/tornado.md)
+
+# API
+
+-   [Crate documentation on `docs.rs` ](./api_reference.md)

docs/md/api_reference.md

@@ -0,0 +1,9 @@
+# API Reference
+
+Perspective's complete API is hosted on `docs.rs`:
+
+-   [`perspective-client`](https://docs.rs/perspective-client/latest/perspective_client/index.html)
+    covers `Table` and `View` data engine API methods common for Rust,
+    JavaScript and Python.
+-   [`perspective-rs`](https://docs.rs/perspective-client/latest/perspective_client/index.html)
+    adds Rust-specific documentation for the Rust crate entrypoint.

docs/md/explanation.md

@@ -0,0 +1 @@
+# Explanation

docs/md/explanation/javascript.md

@@ -0,0 +1,15 @@
+Perspective's JavaScript library offers a configurable UI powered by the same
+fast streaming data engine, just re-compiled to WebAssembly. A simple example
+which loads an [Apache Arrow](https://arrow.apache.org/) and computes a "Group
+By" operation, returning a new Arrow:
+
+```javascript
+import perspective from "@finos/perspective";
+
+const table = await perspective.table(apache_arrow_data);
+const view = await table.view({ group_by: ["CounterParty", "Security"] });
+const arrow = await view.to_arrow();
+```
+
+[More Examples](https://github.com/finos/perspective/tree/master/examples) are
+available on GitHub.

docs/md/explanation/javascript_builds.md

@@ -0,0 +1,42 @@
+# JavaScript Builds
+
+Perspective requires the browser to have access to Perspective's `.wasm`
+binaries _in addition_ to the bundled `.js` files, and as a result the build
+process requires a few extra steps. To ease integration, Perspective's NPM
+releases come with multiple prebuilt configurations.
+
+## Browser
+
+### ESM Builds
+
+The recommended builds for production use are packaged as ES Modules and require
+a _bootstrapping_ step in order to acquire the `.wasm` binaries and initialize
+Perspective's JavaScript with them. However, because they have no hard-coded
+dependencies on the `.wasm` paths, they are ideal for use with JavaScript
+bundlers such as ESBuild, Rollup, Vite or Webpack.
+
+### CDN Builds <!-- Explanation -->
+
+Perspective's CDN builds are good for non-bundled scenarios, such as importing
+directly from a `<script>` tag with a browser-side `import`. CDN builds _do not_
+require _bootstrapping_ the WebAssembly binaries, but they also generally _do
+not_ work with bundlers such as `WebPack`.
+
+### Inline Builds <!-- Explanation -->
+
+<span class="warning">Inline builds are deprecated and will be removed in a
+future release.</span>
+
+Perspective's _Inline_ Builds are a last-ditch effort at compatibility. They
+work by _inlining_ WebAssembly binary content as a base64-encoded string. While
+inline builds work with most bundlers and _do not_ require bootstrapping, there
+is an inherent file-size and boot-performance penalty when using this
+inefficient build method.
+
+Prefer your bundler's inlining features and Perspective ESM builds to this one
+where possible.
+
+## Node.js
+
+There is a Node.js build as well for `@finos/perspective` data engine, which
+shouldn't require any special instructions to use.

docs/md/explanation/javascript_module_structure.md

@@ -0,0 +1,57 @@
+# Module Structure
+
+Perspective is designed for flexibility, allowing developers to pick and choose
+which modules they need for their specific use case. The main modules are:
+
+-   `@finos/perspective`  
+    The data engine library, as both a browser ES6 and Node.js module. Provides
+    a WebAssembly, WebWorker (browser) and Process (node.js) runtime.
+
+-   `@finos/perspective-viewer`  
+    A user-configurable visualization widget, bundled as a
+    [Web Component](https://www.webcomponents.org/introduction). This module
+    includes the core data engine module as a dependency.
+
+`<perspective-viewer>` by itself only implements a trivial debug renderer, which
+prints the currently configured `view()` as a CSV. Plugin modules for popular
+JavaScript libraries, such as [d3fc](https://d3fc.io/), are packaged separately
+and must be imported individually.
+
+Perspective offers these plugin modules:
+
+-   `@finos/perspective-viewer-datagrid`  
+    A custom high-performance data-grid component based on HTML `<table>`.
+
+-   `@finos/perspective-viewer-d3fc`  
+    A `<perspective-viewer>` plugin for the [d3fc](https://d3fc.io) charting
+    library.
+
+When imported after `@finos/perspective-viewer`, the plugin modules will
+register themselves automatically, and the renderers they export will be
+available in the `plugin` dropdown in the `<perspective-viewer>` UI.
+
+## Which modules should I import?
+
+Depending on your requirements, you may need just one, or all, Perspective
+modules. Here are some basic guidelines to help you decide what is most
+appropriate for your project:
+
+-   For Perspective's high-performance streaming data engine (in WebAssembly),
+    or for a purely Node.js based application, import:
+
+    -   `@finos/perspective`, as detailed [here](#perspective-library)
+
+-   For Perspective as a simple, browser-based data visualization widget, you
+    will need to import:
+
+    -   `@finos/perspective`, detailed [here](#perspective-library)
+    -   `@finos/perspective-viewer`, detailed
+        [here](#perspective-viewer-web-component)
+    -   `@finos/perspective-viewer-datagrid` for data grids
+    -   `@finos/perspective-viewer-d3fc` for charting
+
+-   For more complex cases, such as
+    [sharing tables between viewers](#sharing-a-table-between-multiple-perspective-viewers)
+    and
+    [binding a viewer to a remote view in Node.js](#remote-perspective-via-workerhost),
+    you will likely need all Perspective modules.

docs/md/explanation/python.md

@@ -0,0 +1,77 @@
+# What is `perspective-python`
+
+Perspective for Python uses the exact same C++ data engine used by the
+[WebAssembly version](https://docs.rs/perspective-js/latest/perspective_js/) and
+[Rust version](https://docs.rs/crate/perspective/latest). The library consists
+of many of the same abstractions and API as in JavaScript, as well as
+Python-specific data loading support for [NumPy](https://numpy.org/),
+[Pandas](https://pandas.pydata.org/) (and
+[Apache Arrow](https://arrow.apache.org/), as in JavaScript).
+
+Additionally, `perspective-python` provides a session manager suitable for
+integration into server systems such as
+[Tornado websockets](https://www.tornadoweb.org/en/stable/websocket.html),
+[AIOHTTP](https://docs.aiohttp.org/en/stable/web_quickstart.html#websockets), or
+[Starlette](https://www.starlette.io/websockets/)/[FastAPI](https://fastapi.tiangolo.com/advanced/websockets/),
+which allows fully _virtual_ Perspective tables to be interacted with by
+multiple `<perspective-viewer>` in a web browser. You can also interact with a
+Perspective table from python clients, and to that end client libraries are
+implemented for both Tornado and AIOHTTP.
+
+## Example
+
+A simple example which loads an [Apache Arrow](https://arrow.apache.org/) and
+computes a "Group By" operation, returning a new Arrow.
+
+```python
+from perspective import Server
+
+client = Server().new_local_client()
+table = client.table(arrow_bytes_data)
+view = table.view(group_by = ["CounterParty", "Security"])
+arrow = view.to_arrow()
+```
+
+[More Examples](https://github.com/finos/perspective/tree/master/examples) are
+available on GitHub.
+
+## What's included
+
+The `perspective` module exports several tools:
+
+-   `Server` the constructor for a new isntance of the Perspective data engine.
+-   The `perspective.widget` module exports `PerspectiveWidget`, the JupyterLab
+    widget for interactive visualization in a notebook cell.
+-   The `perspective.handlers` modules exports web frameworks handlers that
+    interface with a `perspective-client` in JavaScript.
+    -   `perspective.handlers.tornado.PerspectiveTornadoHandler` for
+        [Tornado](https://www.tornadoweb.org/)
+    -   `perspective.handlers.starlette.PerspectiveStarletteHandler` for
+        [Starlette](https://www.starlette.io/) and
+        [FastAPI](https://fastapi.tiangolo.com)
+    -   `perspective.handlers.aiohttp.PerspectiveAIOHTTPHandler` for
+        [AIOHTTP](https://docs.aiohttp.org),
+
+### Virtual UI server
+
+As `<perspective-viewer>` or any other Perspective `Client` will only consume
+the data necessary to render the current screen (or wahtever else was requested
+via the API), this runtime mode allows large datasets without the need to copy
+them entirely to the Browser, at the expense of network latency on UI
+interaction/API calls.
+
+### Jupyterlab
+
+`PerspectiveWidget` is a JupyterLab widget that implements the same API as
+`<perspective-viewer>`, allows running such a viewer in
+[JupyterLab](https://jupyterlab.readthedocs.io/en/stable/) in either server or
+client (via WebAssembly) mode. `PerspectiveWidget` is compatible with Jupyterlab
+3 and Jupyter Notebook 6 via a
+[prebuilt extension](https://jupyterlab.readthedocs.io/en/stable/extension/extension_dev.html#prebuilt-extensions).
+To use it, simply install `perspective-python` and the extensions should be
+available.
+
+`perspective-python`'s JupyterLab extension also provides convenient builtin
+viewers for `csv`, `json`, or `arrow` files. Simply right-click on a file with
+this extension and choose the appropriate `Perpective` option from the context
+menu.

docs/md/explanation/table.md

@@ -0,0 +1,14 @@
+# Table
+
+`Table` is Perspective's columnar data frame, analogous to a Pandas `DataFrame`
+or Apache Arrow, supporting append & in-place updates, removal by index, and
+update notifications.
+
+A `Table` contains columns, each of which have a unique name, are strongly and
+consistently typed, and contains rows of data conforming to the column's type.
+Each column in a `Table` must have the same number of rows, though not every row
+must contain data; null-values are used to indicate missing values in the
+dataset. The schema of a `Table` is _immutable after creation_, which means the
+column names and data types cannot be changed after the `Table` has been
+created. Columns cannot be added or deleted after creation either, but a `View`
+can be used to select an arbitrary set of columns from the `Table`.

docs/md/explanation/table/clear_and_replace.md

@@ -0,0 +1,25 @@
+# `Table::clear` and `Table::replace`
+
+Calling `Table::clear` will remove all data from the underlying `Table`. Calling
+`Table::replace` with new data will clear the `Table`, and update it with a new
+dataset that conforms to Perspective's data types and the existing schema on the
+`Table`.
+
+<div class="javascript">
+
+```javascript
+// JavaScript
+
+table.clear();
+table.replace(json);
+```
+
+</div>
+<div class="python">
+
+```python
+# Python
+
+table.clear()
+table.replace(df)
+```