feat: add persistence and refetching (#19)

* add persistence * chore: update readme and bump veersioin * fix api client version * fix tests * adjust readme * rename bloomFilter to bloomFilterObject * refactor: storage api and save every domain blocklist * chore: fix typos and inconsistencies * chore: add react-native guide * chore: export fixes
blowfishxyz · Oct 18, 2023 · 143fd08 · 143fd08
1 parent d92ef06
commit 143fd08
Show file tree

Hide file tree

Showing 12 changed files with 1,177 additions and 721 deletions.
diff --git a/README.md b/README.md
@@ -1,9 +1,11 @@
 # Blowfish Local Blocklists
-This is a Javascript library that makes it easy to access the Blowfish Local Blocklist API: for example, to fetch the blocklist object from API, scan a domain against the blocklist and saved bloom filter.
+
+This is a Javascript/Typescript library that makes it easy to access the Blowfish Local Blocklist API: for example, to fetch the blocklist object from API, scan a domain against the blocklist and saved bloom filter.
 
 It's designed to support React Native, Chrome Extension and Node.js environments.
 
 ## Install
+
 ```bash
 npm install @blowfishxyz/blocklist
 ```
@@ -12,131 +14,293 @@ It's also recommended for React Native apps to install `react-native-url-polyfil
 
 ## Usage
 
-In order to execute lookups, you need to fetch a **blocklist object** and **bloom filter**. 
+In order to execute lookups, you need to fetch a **blocklist object** and **bloom filter**.
 After the first fetch, you should keep these objects updated. You can save the objects in a local database
 (for example, using local storage in Chrome extension).
 
-### Blocklist object
-This object includes a link to the bloom filter and the recently added/removed domains domains. 
-
 We recommend updating it every 5 minutes.
 
+### Basic usage
+
 ```javascript
-import { fetchDomainBlocklist, DEFAULT_BLOCKLIST_URL } from '@blowfishxyz/blocklist';
+import {
+  BlowfishLocalBlocklist,
+  ApiConfig,
+  BLOWFISH_API_BASE_URL,
+} from "@blowfishxyz/blocklist";
 
 const apiConfig: ApiConfig = {
-    domainBlocklistUrl: DEFAULT_BLOCKLIST_URL,
-    apiKey: "",
+  basePath: BLOWFISH_API_BASE_URL,
+  // It's highly encouraged to use a proxy server to not expose your API key on the client (see: https://docs.blowfish.xyz/docs/wallet-integration-guide#optional-proxy-server).
+  // When using a proxy server, replace basePath with your endpoint and set apiKey to `undefined`.
+  apiKey: "you-api-key",
 };
-const blocklist = await fetchDomainBlocklist(apiConfig);
+const blocklist = new BlowfishLocalBlocklist(apiConfig);
+
+// 1. Fetch the blocklist and persist it in the storage
+blocklist.fetchBlocklist();
 
-if (blocklist) {
-    [...] // save blocklist.recentlyAdded and blocklist.recentlyRemoved to a local database
+// 2. Re-refetch the blocklist every 5 minutes
+setInterval(() => blocklist.fetchBlocklist(), 1000 * 60 * 5);
+
+// 3. Once you have a blocklist object and a bloom filter saved, you can execute lookups
+const action = blocklist.scanDomain("https://scam-website.io");
+
+if (action === Action.BLOCK) {
+  // block the domain
 }
 ```
 
-You can skip `apiKey` and pass custom `domainBlocklistUrl` to route the query to your backend app.
+You can skip `apiKey` and pass custom `basePath` to route the query to your backend app or a proxy.
 
 ### Bloom filter
 
-Blocklist object links to a bloom filter. However, bloom filter is a 500 KB file, so your app should only
+Blocklist object links to a bloom filter. However, bloom filter is a 700 KB file, so your app should only
 re-download it when nessesary.
 
-To do that, consider tracking `blocklist.bloomFilter.hash` in your local database.
-If app doesn't have a stored hash, or stored hash doesn't match `blocklist.bloomFilter.hash`, download the blocklist from `blocklist.bloomFilter.url`.
+To do that, we are tracking bloom filter's hash and re-fetching it if necessary.
 
-Then, save the bloom filter object itself and its hash to your local database.
-
-```javascript
-import { fetchDomainBlocklist, fetchDomainBlocklistBloomFilter } from '@blowfishxyz/blocklist';
-
-const blocklist = await fetchDomainBlocklist(apiConfig);
-const storedHash = [...]; // fetch it from your storage
-if (storedHash != blocklist.bloomFilter.hash) {
-    const bloomFilter = await fetchDomainBlocklistBloomFilter(blocklist.bloomFilter.url);
-    [...] // save bloomFilter to a local database
-    [...] // save bloomFilter.hash or blocklist.bloomFilter.hash to a local database
-}
-```
+Then, we save the bloom filter object itself and its hash to the `storage`.
 
 We don't update blocklist hash more often than every 24 hours.
 
-### Executing lookups
+### Error handling
 
-Once you have a blocklist object and a bloom filter saved, you can execute lookups.
+Functions that depend on API an/or network can return `null` when I/O errors are encountered.
 
-```javascript
-import { scanDomain, Action } from '@blowfishxyz/blocklist';
+If you would like to track errors, you can pass optional `reportError` callback to `BlowfishLocalBlocklist` constructor.
 
-const recentlyAdded = [...]; // get from storage
-const recentlyRemoved = [...]; // get from storage
-const bloomFilter = [...]; // get from storage
+It could be called with an `Error` or with a string.
 
-const action = scanDomain(
-    bloomFilter,
-    recentlyAdded,
-    recentlyRemoved, 
-    "https://example.com/"
-);
+## Guides
 
-if (action === Action.BLOCK) {
-    // block the domain
-}
+### Browser extension
+
+1. Install Necessary Dependencies:
+
+```bash
+npm install @blowfishxyz/blocklist webextension-polyfill
 ```
 
-### Error handling
+2. Create Blocklist Module:
+
+```typescript
+// src/blocklist.ts
+import {
+  BlowfishLocalBlocklist,
+  BlowfishBlocklistStorageKey,
+  BlowfishBlocklistStorage,
+  BLOWFISH_API_BASE_URL,
+} from "@blowfishxyz/blocklist";
+
+const storage: BlowfishBlocklistStorage = {
+  async getItem<T>(key: BlowfishBlocklistStorageKey) {
+    const storage = chrome.storage.local.get([key]);
+    return storage[key] as T | undefined;
+  },
+  async setItem(key: BlowfishBlocklistStorageKey, data: unknown) {
+    return chrome.storage.local.set({
+      [key]: data,
+    });
+  },
+};
 
-Functions that depend on API an/or network can return `null` when I/O errors are encountered.
+export const blocklist = new BlowfishLocalBlocklist(
+  { basePath: BLOWFISH_API_BASE_URL, apiKey: undefined },
+  undefined,
+  storage
+);
+export { Action } from "@blowfishxyz/blocklist";
+```
 
-If you would like to track errors, you can pass optional `trackError` callback to `fetchBlocklist` and `fetchBloomFilter` functions. 
+3. Schedule Blocklist Updates:
 
-It could be called with an `Error` or with a string.
+```typescript
+// src/background.ts
+import Browser from "webextension-polyfill";
+import { blocklist } from "./blocklist";
 
+Browser.alarms.onAlarm.addListener((alarm) => {
+  if (alarm.name === "refetch-blocklist") {
+    blocklist.fetchBlocklist();
+  }
+});
 
-## Exported functions
+Browser.alarms.create("refetch-blocklist", {
+  periodInMinutes: 5,
+  delayInMinutes: 0,
+});
+```
 
-The following functions are exported by the library:
+4. Domain Scanning:
+
+```typescript
+// src/content-script.ts
+import Browser from "webextension-polyfill";
+import { blocklist, Action } from "./blocklist";
+
+blocklist.scanDomain(window.location.href).then((action) => {
+  if (action === Action.BLOCK) {
+    Browser.runtime.sendMessage({
+      type: "block-domain",
+      host: window.location.hostname,
+      href: encodeURI(window.location.href),
+    });
+  }
+});
+```
 
-### `fetchDomainBlocklist`
-Fetch blocklist JSON object from Blowfish API with recent domains and  a link to the bloom filter.
+5. Blocked Domain Screen:
 
-#### Arguments
-* `apiConfig: ApiConfig`: an object that contains the API configuration details, such as the URL for the domain blocklist and the API key.
-You can use it to pass API requests to a proxy that sits between your users and Blowfish API.
-* `priorityBlockLists: string[] | null`: An array of strings that contains the priority blocklists. (optional)
-* `priorityAllowLists: string[] | null`: An array of strings that contains the priority allowlists. (optional)
-* `reportError: (error: unknown) => void`: A callback function that library uses to track errors when result is `null`. (optional)
+```typescript
+// src/block-screen.tsx
+import { blocklist } from "./blocklist";
 
-#### Return type
+function proceedToBlockedDomainButtonClickHandler() {
+  blocklist.allowDomainLocally(window.location.href);
+}
 ```
-Promise<{ bloomFilter: BloomFilter, recentlyAdded: string[], recentlyRemoved: string[] } | null>
+
+### React Native
+
+1. Install Necessary Dependencies:
+
+```bash
+npm install @blowfishxyz/blocklist react-native-async-storage react-native-background-timer react-native-url-polyfill
 ```
 
-### `fetchDomainBlocklistBloomFilter`
+2. Create Blocklist Module:
+
+```typescript
+// src/blocklist.ts
+import {
+  BlowfishLocalBlocklist,
+  BlowfishBlocklistStorageKey,
+  BlowfishBlocklistStorage,
+  BLOWFISH_API_BASE_URL,
+} from "@blowfishxyz/blocklist";
+import AsyncStorage from "@react-native-async-storage/async-storage";
+
+const storage: BlowfishBlocklistStorage = {
+  async getItem<T>(key: BlowfishBlocklistStorageKey): Promise<T | undefined> {
+    const data = await AsyncStorage.getItem(key);
+    return data ? (JSON.parse(data) as T) : undefined;
+  },
+  async setItem(
+    key: BlowfishBlocklistStorageKey,
+    data: unknown
+  ): Promise<void> {
+    await AsyncStorage.setItem(key, JSON.stringify(data));
+  },
+};
 
-Fetches the bloom filter from specified URL and returns it.
+export const blocklist = new BlowfishLocalBlocklist(
+  { basePath: BLOWFISH_API_BASE_URL, apiKey: undefined },
+  undefined,
+  storage
+);
+export { Action } from "@blowfishxyz/blocklist";
+```
 
-#### Arguments
-* `url: string`: the URL for the bloom filter. 
-You can use URL returned from `fetchDomainBlocklist` function or proxy this URL through your own server.
+3. Schedule Blocklist Updates:
 
-#### Return type
-`Promise<{ bitVector: number[], k: number, hash: string, bits: number, salt: number } | null>`
+```typescript
+// src/background.ts
+import { blocklist } from "./blocklist";
+import BackgroundTimer from "react-native-background-timer";
 
-### `scanDomain`
-Scans a domain against the domain blocklist and returns the action to be taken (either `BLOCK` or `NONE`).
+let intervalId;
 
-#### Arguments
+const refetchBlocklist = () => {
+  blocklist.fetchBlocklist();
+};
+export const startBlocklistRefetch = () => {
+  intervalId = BackgroundTimer.setInterval(refetchBlocklist, 5 * 60 * 60);
+};
+
+export const stopBlocklistRefetch = () => {
+  BackgroundTimer.clearInterval(intervalId);
+};
+```
 
-* `bloomFilter: { bitVector: number[], k: number, hash: string, bits: number, salt: number }`: the bloom filter for the domain blocklist.
-* `recentlyAdded: string[]`: an array of domains that have recently been added to the blocklist.
-* `recentlyRemoved: string[]`: an array of domains that have recently been removed from the blocklist.
-* `domain: string`: the domain or an URL to be scanned.
+4. Domain Scanning:
 
-#### Return type
+```typescript
+// src/domainScanner.ts
+import { blocklist, Action } from "./blocklist";
+
+const scanCurrentDomain = async (url: string) => {
+  const action = await blocklist.scanDomain(url);
+  if (action === Action.BLOCK) {
+    // Handle domain blocking logic
+    console.warn("Blocked domain:", url);
+  }
+};
+
+export default scanCurrentDomain;
 ```
-enum Action {
-    BLOCK = "BLOCK",
-    NONE = "NONE",
+
+5. Blocked Domain Screen:
+
+```typescript
+// src/BlockScreen.tsx
+import React from "react";
+import { TouchableOpacity, Text } from "react-native";
+import { blocklist } from "./blocklist";
+
+function proceedToBlockedDomainHandler(url: string) {
+  blocklist.allowDomainLocally(url);
 }
+
+const BlockScreen: React.FC<{ url: string }> = ({ url }) => {
+  return (
+    <TouchableOpacity onPress={() => proceedToBlockedDomainHandler(url)}>
+      <Text>Proceed to Blocked Domain</Text>
+    </TouchableOpacity>
+  );
+};
+
+export default BlockScreen;
 ```
+
+## API Reference
+
+### `BlowfishLocalBlocklist`
+
+### Constructor arguments
+
+- `apiConfig: ApiConfig`
+  - `basePath: string`: the URL for the domain blocklist. You can use it to pass API requests to a proxy that sits between your users and Blowfish API.
+  - `apiKey: string | undefined`: the API key for the Blowfish API. Can be `undefined` when using a proxy.
+- `blocklistConfig: BlocklistConfig`
+  - `priorityBlockLists: PriorityBlockListsEnum[] | undefined`: Always block domain if it present on one of these lists, even if it's allow-listed on one of regular allow lists (ex: `PHANTOM`, `BLOWFISH`, `BLOWFISH_AUTOMATED`, `SOLFARE`, `PHISHFORT`, `SCAMSNIFFER`, `METAMASK`)
+  - `priorityAllowLists: PriorityAllowListsEnum[] | undefined`: Override domain blocking if domain is present on one of these lists, even if it's block-listed on of regular block lists (ex: `BLOWFISH`, `METAMASK`, `DEFILLAMA`)
+  - `blockLists: BlockListsEnum[] | undefined`: Override domain blocking if domain is present on one of these lists, even if it's block-listed on of regular block lists (ex: `PHANTOM`, `BLOWFISH`, `BLOWFISH_AUTOMATED`, `SOLFARE`, `PHISHFORT`, `SCAMSNIFFER`, `METAMASK`)
+  - `allowLists: AllowListsEnum[] | undefined`: Override domain blocking if domain is present on one of these lists, even if it's block-listed on of regular block lists (ex: `BLOWFISH`, `METAMASK`, `DEFILLAMA`)
+  - `bloomFilterTtl?: number`: How long a bloom filter and corresponding `hash` should remain static. By default, 24 hours. Minimum 24 hours, maximum 14 days. During this time, new domains will be added to `recentlyAdded` and removed from `recentlyRemoved` fields.
+- `storage: BlowfishBlocklistStorage` If storage is not specified we use in-memory storage. It is highly encouraged to provide the proper storage for your environemnt ([see guides](#guides)).
+  - `getItem<T>(key: BlowfishBlocklistStorageKey): Promise<T | undefined>`: get item by key from the environment storage.
+  - `setItem(key: BlowfishBlocklistStorageKey, data: unknown)`: set item by key to the environment storage.
+- `reportError: (error: unknown) => void`: A callback function that library uses to track errors when result is `null`. (optional)
+
+### Methods
+
+### `fetchBlocklist(): Promise<LocalBlocklist | undefined>`
+
+Fetches the blocklist metadata and saves it to the storage. If the fetched blocklist hash is different from one in the storage, it re-fetches the bloom filter and saves it to the storage.
+
+If the blocklist fetch fails, the method returns `undefined` and reports the error to `reportError`.
+
+### `scanDomain(url: string): Promise<Action>`
+
+Scans a domain against the stored domain blocklist and returns the action to be taken (either `BLOCK` or `NONE`).
+
+If there is no stored blocklist it fetches the blocklist using `fetchBlocklist` method and returns the resulting action.
+
+If the fetch fails, the method returns the action `NONE` and reports the error to `reportError`.
+
+### `allowDomainLocally(url: string): Promise<Action>`
+
+If the user wants to proceed to the blocked domain with an explicit action, the domain is added in the user allow list (locally in the storage).
+
+The `scanDomain` method will return `NONE` action for this domain even if it's in the blocklist.
diff --git a/jest.config.js b/jest.config.js
@@ -1,5 +1,6 @@
 /** @type {import('ts-jest').JestConfigWithTsJest} */
 export default {
-  preset: 'ts-jest',
-  testEnvironment: 'node',
+  preset: "ts-jest",
+  testEnvironment: "node",
+  setupFilesAfterEnv: ["./jestSetup.js"],
 };
diff --git a/jestSetup.js b/jestSetup.js
@@ -0,0 +1,2 @@
+// eslint-disable-next-line @typescript-eslint/no-var-requires, no-undef
+require("dotenv").config();
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		// eslint-disable-next-line @typescript-eslint/no-var-requires, no-undef
		require("dotenv").config();