-
+
+
+@finos/perspective JavaScript API`,
},
{
type: "html",
- value: "Rust",
- },
- {
- href: "https://docs.rs/perspective/latest/perspective/",
- label: "`perspective`, Rust API",
- },
- {
- href: "https://docs.rs/perspective-client/latest/perspective_client/struct.Table.html",
- label: "`Table` API",
- },
- {
- href: "https://docs.rs/perspective-client/latest/perspective_client/struct.View.html",
- label: "`View` API",
+ value: `perspective-python Python API`,
},
{
type: "html",
- value: "Appendix",
- },
- {
- href: "https://docs.rs/perspective-server/latest/perspective_server/",
- label: "Data Binding",
- },
- {
- href: "https://docs.rs/perspective-client/latest/perspective_client/config/expressions/",
- label: "Expression Columns",
+ value: `perspective Rust API`,
},
],
},
diff --git a/docs/md/SUMMARY.md b/docs/md/SUMMARY.md
new file mode 100644
index 0000000000..3a161513cc
--- /dev/null
+++ b/docs/md/SUMMARY.md
@@ -0,0 +1,60 @@
+# Summary
+
+[What is Perspective](./perspective.md)
+
+# Overview
+
+- [Data Architecture](./explanation/architecture.md)
+ - [Client-only](./explanation/architecture/client_only.md)
+ - [Client/Server replicated](./explanation/architecture/client_server.md)
+ - [Server only](./explanation/architecture/server_only.md)
+- [`Table`](./explanation/table.md)
+ - [Construct an empty `Table` from a schema](./explanation/table/constructing_schema.md)
+ - [Schema and column types](./explanation/table/schema.md)
+ - [Loading data](./explanation/table/loading_data.md)
+ - [`index` and `limit` options](./explanation/table/options.md)
+ - [`update()` and `remove()` streaming methods](./explanation/table/update_and_remove.md)
+ - [`clear()` and `replace()` start-over methods](./explanation/table/clear_and_replace.md)
+- [`View`](./explanation/view.md)
+ - [Querying data](./explanation/view/querying.md)
+ - [`group_by`](./explanation/view/config/group_by.md)
+ - [`split_by`](./explanation/view/config/split_by.md)
+ - [`aggregates`](./explanation/view/config/aggregates.md)
+ - [`columns`](./explanation/view/config/columns.md)
+ - [`sort`](./explanation/view/config/sort.md)
+ - [`filter`](./explanation/view/config/filter.md)
+ - [`expressions`](./explanation/view/config/expressions.md)
+ - [Flattening a View into a Table](./explanation/view/config/flattening.md)
+- [JavaScript](./explanation/javascript.md)
+ - [Module Structure](./explanation/javascript_module_structure.md)
+ - [Build options](./explanation/javascript_builds.md)
+- [Python](./explanation/python.md)
+
+# Getting Started
+
+- [Rust](./how_to/rust.md)
+- [JavaScript](./how_to/javascript.md)
+ - [Installation via NPM](./how_to/javascript/installation.md)
+ - [Importing with or without a bundler](./how_to/javascript/importing.md)
+ - [`perspective` data engine library](./how_to/javascript/worker.md)
+ - [Serializing data](./how_to/javascript/serializing.md)
+ - [Cleaning up resources](./how_to/javascript/deleting.md)
+ - [Hosting a `WebSocketServer` in Node.js](./how_to/javascript/nodejs_server.md)
+ - [`perspective-viewer` Custom Element library](./how_to/javascript/viewer.md)
+ - [Theming](./how_to/javascript/theming.md)
+ - [Loading data from a `Table`](./how_to/javascript/loading_data.md)
+ - [Loading data from a virtual `Table`](./how_to/javascript/loading_virtual_data.md)
+ - [Saving and restoring UI state](./how_to/javascript/save_restore.md)
+ - [Listening for events](./how_to/javascript/events.md)
+- [Python](./how_to/python.md)
+ - [Installation](./how_to/python/installation.md)
+ - [Loading data into a `Table`](./how_to/python/table.md)
+ - [Callbacks and events](./how_to/python/callbacks.md)
+ - [Multithreading](./how_to/python/multithreading.md)
+ - [Hosting a WebSocket server](./how_to/python/websocket.md)
+ - [`PerspectiveWidget` for JupyterLab](./how_to/python/jupyterlab.md)
+ - [Tutorial: A `tornado` server with virtual `perspective-viewer`](./tutorials/python/tornado.md)
+
+# API
+
+- [Crate documentation on `docs.rs` ](./api_reference.md)
diff --git a/docs/md/api_reference.md b/docs/md/api_reference.md
new file mode 100644
index 0000000000..c5836cf30b
--- /dev/null
+++ b/docs/md/api_reference.md
@@ -0,0 +1,9 @@
+# API Reference
+
+Perspective's complete API is hosted on `docs.rs`:
+
+- [`perspective-client`](https://docs.rs/perspective-client/latest/perspective_client/index.html)
+ covers `Table` and `View` data engine API methods common for Rust,
+ JavaScript and Python.
+- [`perspective-rs`](https://docs.rs/perspective-client/latest/perspective_client/index.html)
+ adds Rust-specific documentation for the Rust crate entrypoint.
diff --git a/docs/md/explanation.md b/docs/md/explanation.md
new file mode 100644
index 0000000000..b1303b40c8
--- /dev/null
+++ b/docs/md/explanation.md
@@ -0,0 +1 @@
+# Explanation
diff --git a/docs/md/explanation/architecture.md b/docs/md/explanation/architecture.md
new file mode 100644
index 0000000000..2739c61fd4
--- /dev/null
+++ b/docs/md/explanation/architecture.md
@@ -0,0 +1,15 @@
+# Data Architecture
+
+Application developers can choose from
+[Client (WebAssembly)](./architecture/client_only.md),
+[Server (Python/Node)](./architecture/server_only.md) or
+[Client/Server Replicated](./architecture/client_server.md) designs to bind
+data, and a web application can use one or a mix of these designs as needed. By
+serializing to Apache Arrow, tables are duplicated and synchronized across
+runtimes efficiently.
+
+Perspective is a multi-language platform. The examples in this section use
+Python and JavaScript as an example, but the same general principles apply to
+any `Client`/`Server` combination.
+
+
diff --git a/rust/perspective-server/docs/architecture.dot b/docs/md/explanation/architecture/architecture.dot
similarity index 100%
rename from rust/perspective-server/docs/architecture.dot
rename to docs/md/explanation/architecture/architecture.dot
diff --git a/rust/perspective-server/docs/architecture.sub1.dot b/docs/md/explanation/architecture/architecture.sub1.dot
similarity index 100%
rename from rust/perspective-server/docs/architecture.sub1.dot
rename to docs/md/explanation/architecture/architecture.sub1.dot
diff --git a/rust/perspective-server/docs/architecture.sub1.svg b/docs/md/explanation/architecture/architecture.sub1.svg
similarity index 100%
rename from rust/perspective-server/docs/architecture.sub1.svg
rename to docs/md/explanation/architecture/architecture.sub1.svg
diff --git a/rust/perspective-server/docs/architecture.sub2.dot b/docs/md/explanation/architecture/architecture.sub2.dot
similarity index 100%
rename from rust/perspective-server/docs/architecture.sub2.dot
rename to docs/md/explanation/architecture/architecture.sub2.dot
diff --git a/rust/perspective-server/docs/architecture.sub2.svg b/docs/md/explanation/architecture/architecture.sub2.svg
similarity index 100%
rename from rust/perspective-server/docs/architecture.sub2.svg
rename to docs/md/explanation/architecture/architecture.sub2.svg
diff --git a/rust/perspective-server/docs/architecture.sub3.dot b/docs/md/explanation/architecture/architecture.sub3.dot
similarity index 100%
rename from rust/perspective-server/docs/architecture.sub3.dot
rename to docs/md/explanation/architecture/architecture.sub3.dot
diff --git a/rust/perspective-server/docs/architecture.sub3.svg b/docs/md/explanation/architecture/architecture.sub3.svg
similarity index 100%
rename from rust/perspective-server/docs/architecture.sub3.svg
rename to docs/md/explanation/architecture/architecture.sub3.svg
diff --git a/rust/perspective-server/docs/architecture.svg b/docs/md/explanation/architecture/architecture.svg
similarity index 100%
rename from rust/perspective-server/docs/architecture.svg
rename to docs/md/explanation/architecture/architecture.svg
diff --git a/docs/md/explanation/architecture/client_only.md b/docs/md/explanation/architecture/client_only.md
new file mode 100644
index 0000000000..7db35eca0e
--- /dev/null
+++ b/docs/md/explanation/architecture/client_only.md
@@ -0,0 +1,39 @@
+# Client-only
+
+
+
+_For static datasets, datasets provided by the user, and simple server-less and
+read-only web applications._
+
+In this design, Perspective is run as a client Browser WebAssembly library, the
+dataset is downloaded entirely to the client and all calculations and UI
+interactions are performed locally. Interactive performance is very good, using
+WebAssembly engine for near-native runtime plus WebWorker isolation for parallel
+rendering within the browser. Operations like scrolling and creating new views
+are responsive. However, the entire dataset must be downloaded to the client.
+Perspective is not a typical browser component, and datset sizes of 1gb+ in
+Apache Arrow format will load fine with good interactive performance!
+
+Horizontal scaling is a non-issue, since here is no concurrent state to scale,
+and only uses client-side computation via WebAssembly client. Client-only
+perspective can support as many concurrent users as can download the web
+application itself. Once the data is loaded, no server connection is needed and
+all operations occur in the client browser, imparting no additional runtime cost
+on the server beyond initial load. This also means updates and edits are local
+to the browser client and will be lost when the page is refreshed, unless
+otherwise persisted by your application.
+
+As the client-only design starts with creating a client-side Perspective
+`Table`, data can be provided by any standard web service in any Perspective
+compatible format (JSON, CSV or Apache Arrow).
+
+#### Javascript client
+
+```javascript
+const worker = await perspective.worker();
+const table = await worker.table(csv);
+
+const viewer = document.createElement("perspective-viewer");
+document.body.appendChild(viewer);
+await viewer.load(table);
+```
diff --git a/docs/md/explanation/architecture/client_server.md b/docs/md/explanation/architecture/client_server.md
new file mode 100644
index 0000000000..f6df4408eb
--- /dev/null
+++ b/docs/md/explanation/architecture/client_server.md
@@ -0,0 +1,58 @@
+# Client/Server replicated
+
+
+
+_For medium-sized, real-time, synchronized and/or editable data sets with many
+concurrent users._
+
+The dataset is instantiated in-memory with a Python or Node.js Perspective
+server, and web applications create duplicates of these tables in a local
+WebAssembly client in the browser, synchonized efficiently to the server via
+Apache Arrow. This design scales well with additional concurrent users, as
+browsers only need to download the initial data set and subsequent update
+deltas, while operations like scrolling, pivots, sorting, etc. are performed on
+the client.
+
+Python servers can make especially good use of additional threads, as
+Perspective will release the GIL for almost all operations. Interactive
+performance on the client is very good and identical to client-only
+architecture. Updates and edits are seamlessly synchonized across clients via
+their virtual server counterparts using websockets and Apache Arrow.
+
+#### Python and Tornado server
+
+```python
+from perspective import Server, PerspectiveTornadoHandler
+
+server = Server()
+client = server.new_local_client()
+client.table(csv, name="my_table")
+routes = [(
+ r"/websocket",
+ perspective.handlers.tornado.PerspectiveTornadoHandler,
+ {"perspective_server": server},
+)]
+
+app = tornado.web.Application(routes)
+app.listen(8080)
+loop = tornado.ioloop.IOLoop.current()
+loop.start()
+```
+
+#### Javascript client
+
+Perspective's websocket client interfaces with the Python server. then
+_replicates_ the server-side Table.
+
+```javascript
+const websocket = await perspective.websocket("ws://localhost:8080");
+const server_table = await websocket.open_table("my_table");
+const server_view = await server_table.view();
+
+const worker = await perspective.worker();
+const client_table = await worker.table(server_view);
+
+const viewer = document.createElement("perspective-viewer");
+document.body.appendChild(viewer);
+await viewer.load(client_table);
+```
diff --git a/docs/md/explanation/architecture/server_only.md b/docs/md/explanation/architecture/server_only.md
new file mode 100644
index 0000000000..09084c4b70
--- /dev/null
+++ b/docs/md/explanation/architecture/server_only.md
@@ -0,0 +1,33 @@
+# Server-only
+
+
+
+_For extremely large datasets with a small number of concurrent users._
+
+The dataset is instantiated in-memory with a Python or Node.js server, and web
+applications connect virtually. Has very good initial load performance, since no
+data is downloaded. Group-by and other operations will run column-parallel if
+configured.
+
+But interactive performance is poor, as every user interaction must page the
+server to render. Operations like scrolling are not as responsive and can be
+impacted by network latency. Web applications must be "always connected" to the
+server via WebSocket. Disconnecting will prevent any interaction, scrolling,
+etc. of the UI. Does not use WebAssembly.
+
+Each connected browser will impact server performance as long as the connection
+is open, which in turn impacts interactive performance of every client. This
+ultimately limits the horizontal scalabity of this architecture. Since each
+client reads the perspective `Table` virtually, changes like edits and updates
+are automatically reflected to all clients and persist across browser refresh.
+Using the same Python server as the previous design, we can simply skip the
+intermediate WebAssembly `Table` and pass the virtual table directly to `load()`
+
+```javascript
+const websocket = await perspective.websocket("ws://localhost:8080");
+const server_table = await websocket.open_table("my_table");
+
+const viewer = document.createElement("perspective-viewer");
+document.body.appendChild(viewer);
+await viewer.load(server_table);
+```
diff --git a/docs/md/explanation/javascript.md b/docs/md/explanation/javascript.md
new file mode 100644
index 0000000000..ffea687dde
--- /dev/null
+++ b/docs/md/explanation/javascript.md
@@ -0,0 +1,15 @@
+Perspective's JavaScript library offers a configurable UI powered by the same
+fast streaming data engine, just re-compiled to WebAssembly. A simple example
+which loads an [Apache Arrow](https://arrow.apache.org/) and computes a "Group
+By" operation, returning a new Arrow:
+
+```javascript
+import perspective from "@finos/perspective";
+
+const table = await perspective.table(apache_arrow_data);
+const view = await table.view({ group_by: ["CounterParty", "Security"] });
+const arrow = await view.to_arrow();
+```
+
+[More Examples](https://github.com/finos/perspective/tree/master/examples) are
+available on GitHub.
diff --git a/docs/md/explanation/javascript_builds.md b/docs/md/explanation/javascript_builds.md
new file mode 100644
index 0000000000..26efe0ae4c
--- /dev/null
+++ b/docs/md/explanation/javascript_builds.md
@@ -0,0 +1,42 @@
+# JavaScript Builds
+
+Perspective requires the browser to have access to Perspective's `.wasm`
+binaries _in addition_ to the bundled `.js` files, and as a result the build
+process requires a few extra steps. To ease integration, Perspective's NPM
+releases come with multiple prebuilt configurations.
+
+## Browser
+
+### ESM Builds
+
+The recommended builds for production use are packaged as ES Modules and require
+a _bootstrapping_ step in order to acquire the `.wasm` binaries and initialize
+Perspective's JavaScript with them. However, because they have no hard-coded
+dependencies on the `.wasm` paths, they are ideal for use with JavaScript
+bundlers such as ESBuild, Rollup, Vite or Webpack.
+
+### CDN Builds
+
+Perspective's CDN builds are good for non-bundled scenarios, such as importing
+directly from a `
+```
+
+## Node.js builds
+
+The Node.js runtime for the `@finos/perspective` module runs in-process by
+default and does not implement a `child_process` interface. Hence, there is no
+`worker()` method, and the module object itself directly exports the full
+`perspective` API.
+
+```javascript
+const perspective = require("@finos/perspective");
+```
+
+In Node.js, perspective does not run in a WebWorker (as this API does not exist
+in Node.js), so no need to call the `.worker()` factory function - the
+`perspective` library exports the functions directly and run synchronously in
+the main process.
diff --git a/docs/md/how_to/javascript/installation.md b/docs/md/how_to/javascript/installation.md
new file mode 100644
index 0000000000..0083fecc9d
--- /dev/null
+++ b/docs/md/how_to/javascript/installation.md
@@ -0,0 +1,31 @@
+# JavaScript NPM Installation
+
+Perspective releases contain several different builds for use in most
+environments.
+
+## Browser
+
+Perspective's WebAssembly data engine is available via NPM in the same package
+as its Node.js counterpart, `@finos/perspective`. The Perspective Viewer UI
+(which has no Node.js component) must be installed separately:
+
+```bash
+$ npm add @finos/perspective @finos/perspective-viewer
+```
+
+By itself, `@finos/perspective-viewer` does not provide any visualizations, only
+the UI framework. Perspective _Plugins_ provide visualizations and must be
+installed separately. All Plugins are optional - but a `
-
-Application developers can choose from [Client (WebAssembly)](#client-only),
-[Server (Python/Node)](#server-only) or
-[Client/Server Replicated](#clientserver-replicated) designs to bind data, and a
-web application can use one or a mix of these designs as needed. By serializing
-to Apache Arrow, tables are duplicated and synchronized across runtimes
-efficiently.
-
-Perspective is a multi-language platform. The examples in this section use
-Python and JavaScript as an example, but the same general principles apply to
-any `Client`/`Server` combination.
-
-## Client-only
-
-