Merge pull request #213 from ulixee/docs

feat: update documentation for payments

client/docs/interface/remote-client.md

@@ -1,27 +1,37 @@
 # Remote Client
 
-This is the default use-case for using Ulixee Client. You supply a connection URI or Object to the datastore you want to query. 
+This is the default use-case for using Ulixee Client. You supply a connection URI or Object to the datastore you want to query.
 
 You can also initialize clients with a local Datastore, Table, Extractor or Crawler instance, however, these clients provide a more limited set of properties and methods than what is shown on this page. See [Local Client](./local-client.md).
 
 ## Constructor
 
-### new Client _(uriOrObject)_ {#constructor}
+### new Client _(uriOrObject, config?)_ {#constructor}
 
 Creates a new Client instance.
 
 #### **Arguments**:
 
 - uri `string` | `Object`. A connection string in the format of `ulx://USERNAME:PASSWORD@HOST:PORT/DB`. You can also supply
-an object with the following properties:
-  - username `string` 
+  an object with the following properties:
+  - username `string`
   - password `string`
   - host `string`
   - port `number`
   - database `string`
+- config `Object`. Optional. Configuration options
+  - paymentService `IPaymentService`. Optional. A [payment service](/docs/datastore/basics/payments#payment-services) to use for transactions.
+  - argonMainchainUrl `string`. Optional. The RPC URL of an Argon Mainchain node to use for looking up notary information for micropayment channel holds.
+  - authentication `Object`. Optional. An object with the following properties:
+  - identity `string`. A bech32 encoded Ed25519 key
+  - signature `string`. A signature of the identity
+  - nonce `string`. A nonce to prevent replay attacks
+  - affiliateId `string`. Optional. An affiliate ID to allow your queries to be tracked (often used in cloned Datastores to request funding)
+  - onQueryResult `(IDatastoreQueryResult) => void`. Optional. A callback function that will be called with the result of every query. The result has the latestVersion, metadata, and output/error.
+  - queryId `string`. Optional. A unique identifier for the query. If not provided, one will be generated.
 
 ```js
-import Client from '@ulixee/client-playground';
+import Client from '@ulixee/client';
 
 const client = new Client({
   username: 'test',
@@ -31,6 +41,22 @@ const client = new Client({
   database: 'test',
 });
 ```
+
+Example using a Payment Service:
+
+```typescript
+import { DefaultPaymentService } from '@ulixee/databroker';
+import Client from '@ulixee/client';
+
+const paymentService = await DefaultPaymentService.fromBroker('wss://broker.testnet.ulixee.org', {
+  pemPath: 'path to your Identity pem file',
+});
+const client = new Client('ulx://UsCPI.Stats/v1.0.0', {
+  paymentService,
+  argonMainchainUrl: 'wss://rpc.testnet.argonprotocol.org',
+});
+```
+
 ## Properties
 
 ### client.username {#username}
@@ -39,35 +65,30 @@ The username authenticated with the remote server.
 
 #### **Type**: `string`
 
-
 ### client.password {#password}
 
 The password used to authenticate with the remote server.
 
 #### **Type**: `string`
 
-
 ### client.host {#host}
 
 The host of the remote server. Defaults to localhost.
 
 #### **Type**: `string`
 
-
 ### client.port {#port}
 
 The port sent of the remote server. Defaults to 1818.
 
 #### **Type**: `number`
 
-
 ### client.database {#database}
 
 The name of the datastore the client is connected.
 
 #### **Type**: `string`
 
-
 ## Methods
 
 ### client.query _(sql, boundValues)_ {#query}
@@ -118,7 +139,6 @@ const records = await client.fetch('daysUntilWorldDomination', { probability: 5
 
 #### **Returns**: `Promise<Record[]>`
 
-
 ### client.crawl _(crawlerName, inputFilter)_ {#crawl}
 
 Trigger one of the Datastore's crawlers:

datastore/docs/advanced/credits.md

@@ -12,7 +12,7 @@ To embed credits, you can configure the [remoteDatastoreEmbeddedCredits](../basi
 
 ## Denominations
 
-Ulixee Payments (in this case, Credits) come in the following denominations:
+Argon Payments (in this case, Credits) come in the following denominations:
 
 - _Argon_: ~1 USD adjusted for inflation.
 - _Milligon_: ~1 thousand of a USD adjusted for inflation ($0.001). 

datastore/docs/basics/payments.md

@@ -0,0 +1,108 @@
+# Datastore Payments
+
+Ulixee Datastores accept two forms of micropayments out of the box: Argons and a Credit system. A Payment Service allows you to customize how payments are allocated for your queries.
+
+You can create your own payment service by implementing the [IPaymentService](https://github.com/ulixee/platform/tree/main/datastore/main/interfaces/IPaymentService.ts) interface.
+
+## Concepts
+
+### Denominations
+
+Argon Payments come in the following denominations:
+
+- _Argon_: ~1 USD adjusted for inflation.
+- _Milligon_: ~1 thousand of a USD adjusted for inflation (about $0.001).
+- _Microgon_: ~1 millionth of a USD adjusted for inflation (about $0.000,001). This is the denomination used by a query (eg, [Extractor.basePrice](./extractor.md#constructor).
+
+### Micropayment ChannelHolds
+
+A micropayment ChannelHold is a temporary hold on a Localchain that reserves a set amount of Argons (let's say, 10 Argon). When a user sets aside funds in their Localchain for a ChannelHold, their account cannot be modified for 1 hour (in Argon, these are 60 "ticks" representing an agreed upon minute of clock time).
+
+The smallest unit in Argon is a milligon, which is 1/1000th of an Argon. But in Ulixee Micropayments, payments are allowed to go as low as a microgon, which is 1/1,000,000th of an Argon. This allows for a price per query model to work with huge volumes, while still keeping the cost per query reasonable.
+
+During the hour, the ChannelHold sender and Datastore can exchange data for payment. There is no way to break the agreement early. Every time another milligon is spent (1/1000th of an argon), the Datastore will require an updated "settlement" indicating 1 more milligon is authorized for payment. This way, at all times, the Datastore is assured that the 10 Argons are legitimate, and the user knows the Datastore can only claim the funds that have been authorized.
+
+After the hour is up, the recipient of the funds (the datastore) submits the signed settlement to a notary and moves those funds to their Localchain.
+
+The Datastore and User were able to exchange data for payment with volumes only limited by each other's machines and network connections. Only the beginning and final settlement need to be sent to the broader Argon network. This allows Ulixee to achieve a high volume of micropayments without overwhelming the Argon network.
+
+### Credits
+
+Datastores come built-in with a credits model. This is a little like a free trial mode if you want to hand out data credits to people trying out your Datastore. Credits are not a payment method, but a way to allocate free queries. A payment service will keep track of how many credits are available and prioritize them before charging for Argons.
+
+## Payment Services
+
+Payment services are used by [Clients](/docs/client) and Datastores to manage payments. They can be used to allocate Argons, manage credits, and track payments.
+
+### DefaultPaymentService
+
+The [default payment service](https://github.com/ulixee/platform/blob/42bc301bb24f1697ea60bca2db9258fe469e0212/datastore/main/payments/DefaultPaymentService.ts#L25) combines an Argon payment service with a Credit payment service. Argon payments can come from a Localchain on the same computer, a [Data broker](./databrokers) or a Remote Service.
+
+You will interact with this class in two primary ways:
+
+#### 1. With a Localchain
+
+If you are running a Localchain on the same computer, you can use the `fromLocalchain` method to create a payment service that will automatically allocate Argons from the Localchain.
+
+```typescript
+import {
+  DefaultPaymentService,
+  IChannelHoldAllocationStrategy,
+  LocalchainWithSync,
+} from '@ulixee/datastore';
+
+// This strategy will create batches of 200 queries worth of argons per ChannelHold (1 hour).
+const channelHoldAllocationStrategy: IChannelHoldAllocationStrategy = {
+  type: 'multiplier',
+  queries: 200,
+};
+const bobchain = await LocalchainWithSync.load({
+  localchainName: 'bobchain',
+  channelHoldAllocationStrategy,
+});
+const paymentService = DefaultPaymentService.fromLocalchain(bobchain);
+```
+
+#### 2. With a Data Broker
+
+A [Databroker](./databrokers) is a service that manages Argons for you. You can use the `fromBroker` method to create a payment service that will automatically allocate Argons from the Data Broker.
+
+```typescript
+import { DefaultPaymentService, IChannelHoldAllocationStrategy } from '@ulixee/datastore';
+
+// This strategy will create batches of 200 queries worth of argons per channelHold (1 hour).
+const channelHoldAllocationStrategy: IChannelHoldAllocationStrategy = {
+  type: 'multiplier',
+  queries: 200,
+};
+const paymentService = await DefaultPaymentService.fromBroker(
+  'wss://broker.testnet.ulixee.org',
+  {
+    pemPath: 'path to your Identity pem file',
+  },
+  channelHoldAllocationStrategy,
+);
+```
+
+### EmbeddedPaymentService
+
+When you [Clone](./cloning) a Datastore that requires payment, your CloudNode needs to establish Micropayment Channels with any upstream datastore(s). The embedded payment service works with a local (or cluster) Localchain to establish payments with limited permissions. This service will automatically only whitelist the upstream Datastore sources and Datastore IDs listed in the cloned Datastore.
+
+To enable the EmbeddedPaymentService, you either need to have a configured Localchain on the same machine, or you'll need to configure a Hosted Service to manage the Localchain for you.
+
+Configure a Localchain with the [`Localchain` configurations](http://localhost:8080/docs/datastore/overview/configuration#payment-configuration), or if you're setting up a CloudNode in a cluster, you would set the `ULX_UPSTREAM_PAYMENTS_SERVICE_HOST` environment variable, pointing to your Hosted Services node. (NOTE: you can also just configure your child with the `ULX_SERVICES_SETUP_HOST` environment variable set to your Hosted Services node).
+
+Lead node:
+
+```bash
+$ npx @ulixee/cloud start --hosted-services-port 18181 \
+    --argon-localchain-path /path/to/localchain \
+    --argon-mainchain-url wss://rpc.testnet.argonprotocol.org \
+    --argon-block-rewards-address 5DRTmdnaztvtdZ56QbEmHM8rqUR2KiKh7KY1AeMfyvkPSb5S
+```
+
+Child node:
+
+```bash
+$ npx @ulixee/cloud start --setup-host <host ip>:18181
+```

datastore/docs/guides/close-argon-blocks.md

@@ -0,0 +1,63 @@
+# Close Argon Blocks
+
+> Your Localchain will automatically convert your settled micropayments into Argon Block votes, and you'll close some blocks! Find out here how to set it up.
+
+## Background
+
+Ulixee uses the Argon currency for Micropayments. When a datastore consumer runs a query, they will lock their Localchain with a set amount of Argons - this is called a Micropayment ChannelHold. Payment is now settled in tiny increments between the datastore and user. Once complete, the Datastore will send the Settled ChannelHold note to a notary and the Argon network.
+
+Argon is a blockchain that uses proof of "work" to close blocks. However, in this case the work is the Datastore queries that you just served to users. Proof of work is supplied in the form of tax revenue from Datastore Micropayments, which are able to be converted into a "vote" on which block to close. If your vote is chosen, you get a portion of the rewards for closing the block.
+
+Argon requires a tax on all transactions, which it uses to stabilize its value. For transactions over 1 Argon, this is set to ~20 cents (0.2 argons). For any micropayments (under 1 Argon), it is only 20% of the transaction value. So as your users pay for queries, you're actually automatically collecting a tax that you can use to close Argon blocks.
+
+## Closing Blocks
+
+When you setup your Datastore for payments, you'll configure the Argon Localchain that your Datastore will use. You'll also choose a specific Argon Miner that you believe is an honest operator.
+
+Your Localchain collects payments and automatically claims the appropriate amount of tax when it settles with your chosen Argon Notary (`ARGON_NOTARY_ID`).
+
+The Localchain can be setup to automatically create block votes if you set an `ARGON_BLOCK_REWARDS_ADDRESS` in your env, or you set it using the `@ulixee/cloud` command line. The Localchain will create votes with a strategy of voting anytime your collected tax exceeds the minimum vote threshold.
+
+NOTE: you can code a more sophisticated strategy if you want to change this process. The code for creating votes can be currently traced in the Argon Mainchain codebase in [Localchain/src/balance_sync.rs](https://github.com/argonprotocol/mainchain/blob/416812ac0c905295dcf76472a68bee16d02e5f3c/localchain/src/balance_sync.rs#L508). A new strategy doesn't have to be in rust. There's a nodejs library to interact with the mainchain `@argonprotocol/mainchain`, as well as node interaction with the notary and localchain at `@argonprotocol/localchain`. You'll find uses of that library in this [project](.
+
+## Securing you Block Rewards Address
+
+Your block rewards account does not need to have a Localchain yet (you can always create one later that attaches to this private key). For that reason, you should generate your block rewards address, but only create a key. You should have a seed phrase and an account address once you're done. Do not publish the seed phrase to any CloudNode or anything public (like a code repository). This will separate your block rewards from your Localchain account - kind of like automatically depositing them into a more secure vault.
+
+NOTE: In a production environment, you would create these with a hardware wallet or a secure offline computer.
+
+For the Testnet, it's perfectly valid to follow the online Polkadot.js process to create an account [here](https://github.com/argonprotocol/mainchain/blob/3a4bfab8cb296b85da0543d577a2a33e85b83b54/docs/account-setup.md) or even reuse your Localchain account if you don't want to bother with this step.
+
+## Watching for Closed Blocks
+
+You can watch for closed blocks by listening to the Argon Mainchain. You can use the `@argonprotocol/mainchain` library to listen for new blocks and check if your vote was chosen. If your vote was chosen, you'll receive a reward in the form of Argons.
+
+To monitor for blocks using your address, you could do something like this:
+
+```typescript
+import { getClient } from '@argonprotocol/mainchain';
+
+const client = await getClient(`wss://rpc.testnet.argonprotocol.org`);
+const eventMetadata = client.events.blockRewards.RewardCreated.meta;
+const rewardsIndex = eventMetadata.fields.findIndex(x => x.name.toString() === 'rewards');
+const unsub = mainchainClient.rpc.chain.subscribeNewHeads(async lastHeader => {
+  const blockHash = lastHeader.hash.toHex();
+  const blockNumber = lastHeader.number.toNumber();
+
+  const events = await mainchainClient.query.system.events.at(blockHash);
+  for (const { event } of events) {
+    if (event.section === 'blockRewards' && event.method === 'RewardCreated') {
+      const rewards = event.data[rewardsIndex].toJSON();
+      for (const reward of rewards) {
+        if (reward.accountId == '5DtCHcwuh7Mhp8cZtvinxDzSa36rh7m3TG9LFo3Tgxuyx889') {
+          console.log(`You closed a block! (${blockNumber}: ${blockHash})`, JSON.stringify(reward));
+        }
+      }
+    }
+  }
+});
+```
+
+Or you can use the Argon [developer console](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frpc.testnet.argonprotocol.org#/explorer) UI to watch for your rewards (this is a common utility built by Polkadot, which produces the Substrate framework that Argon is built on top of. You should see a BlockRewards event with your address if you closed a block.
+
+![Polkadot.js - Block Explorer](../images/pjs-block-explorer.png)

datastore/docs/guides/querying-datastores.md

@@ -0,0 +1,62 @@
+# Querying Datastores
+
+The primary way you'll interact with Datastores is the `@ulixee/client` library. The client library allows you to use an addressing system to lookup datastores by version and id, and then run queries against them using SQL. You can learn the details of the client library [here](../../client).
+
+## Using a Localchain
+
+Datastores that require payment use the [Argon currency](https://argonprotocol.org). You can learn more about the Argon [here](./using-localchain.md). You can directly use a Localchain account to pay for queries, as shown below. In this example, a Datastore with a `domain` of `Meals.Health` is queried for all recipes that are `paleo`. Version `0.0.1` of the Datastore is used.
+
+This payment services is using all default settings, which will use the `primary` Localchain on the machine installed into the default location. It will attempt to create [Channel Holds](../basics/payments.md#micropayment-channelholds) for 100 queries at a time - if the price is 1 milligon per query, this will load 100 milligons into the Channel Hold. You can choose different "Channel Hold" strategies by passing in a different `channelHoldAllocationStrategy` object.
+
+```typescript
+import { Client, DefaultPaymentService } from '@ulixee/client';
+
+(async () => {
+  const client = new Client(`ulx://Meals.Health/v0.0.1`, {
+    paymentService: await DefaultPaymentService.fromLocalchain(),
+  });
+  const results = await client.query(
+    `SELECT * from recipes where diet = 'paleo`,
+  );
+
+  console.log(results);
+
+  await client.disconnect();
+})().catch(console.error);
+```
+
+### Acquire Testnet Argons
+
+If you want to test out a Datastore using the Argon testnet, you can request them using the Discord Faucet. Find directions [here](https://github.com/argonprotocol/mainchain/blob/main/docs/account-setup.md#requesting-testnet-funds). 
+
+## Using the Testnet Databroker
+
+> Under construction.
+
+An easier way to query Datastores is to use the Ulixee Foundation's Databroker. The Databroker allows you to run queries against any Datastore on the Testnet without needing to manage a Localchain account. The Databroker will automatically pay for your queries using the Ulixee Foundation's Localchain account. You'll need to register on the Databroker Admin Panel to get an API key (see next step).
+
+```typescript
+import { Client, DefaultPaymentService } from '@ulixee/client';
+
+(async () => {
+  const client = new Client(`ulx://Meals.Health/v0.0.1`, {
+    paymentService: await DefaultPaymentService.fromBroker(
+      'wss://databroker.testnet.ulixee.org',
+      {
+        pemPath: 'path to your Identity pem file',
+      },
+    ),
+  });
+  const results = await client.query(
+    `SELECT * from recipes where diet = 'paleo`,
+  );
+
+  console.log(results);
+
+  await client.disconnect();
+})().catch(console.error);
+```
+
+### Register on the Databroker Admin Panel
+
+Normally, the admin panel for a Databroker is locked down to the owner of the Datastore. However, the Ulixee Foundation has opened up the Databroker for the Testnet to allow anyone to register and run queries. You can register for the Databroker [here](https://databroker.testnet.ulixee.org:18171/admin).