-
Notifications
You must be signed in to change notification settings - Fork 18
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
KDB.AI destination connector: how to create a table with a compatible…
… schema and index (#367)
- Loading branch information
1 parent
84518cc
commit 5de0da7
Showing
1 changed file
with
61 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,64 @@ | ||
- A KDB.AI Cloud or server instance. [Sign Up for KDB.AI Cloud: Starter Edition](https://trykdb.kx.com/kdbai/signup/). [Set up KDB.AI Server](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-server-setup.html). | ||
- The instance's endpoint URL. [Get the KDB.AI Cloud endpoint URL](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-cloud-setup.html#connect-to-your-database). [Get the KDB.AI Server endpoint URL](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-server-setup.html). | ||
- An API key. [Create the API key](https://code.kx.com/kdbai/latest/gettingStarted/kdb-ai-cloud-setup.html#create-an-api-key). | ||
- The name of the target table to access. [Create the table](https://code.kx.com/kdbai/latest/gettingStarted/quickstart.html#create-a-new-table). | ||
- The name of the target table to access. [Create the table](https://code.kx.com/kdbai/latest/gettingStarted/quickstart.html#create-a-new-table). | ||
|
||
KDB.AI requires the target table to have a defined schema before Unstructured can write to the table. The recommended table | ||
schema for Unstructured contains the fields `id`, `element_id`, `document`, `metadata`, and `embeddings`, as follows. | ||
This example code demonstrates the use of the [KDB.AI Client for Python](https://pypi.org/project/kdbai-client/) to create | ||
a table with this recommended schema, along with creating a vector index that contains 3072 dimensions: | ||
|
||
```python Python | ||
import kdbai_client as kdbai | ||
import os | ||
|
||
session = kdbai.Session( | ||
endpoint=os.getenv("KDBAI_ENDPOINT"), | ||
api_key=os.getenv("KDBAI_API_KEY") | ||
) | ||
|
||
db = session.database("default") | ||
|
||
schema = [ | ||
{ | ||
"name": "id", | ||
"type": "str" | ||
}, | ||
{ | ||
"name": "element_id", | ||
"type": "str" | ||
}, | ||
{ | ||
"name": "document", | ||
"type": "str" | ||
}, | ||
{ | ||
"name": "metadata", | ||
"type": "general" | ||
}, | ||
{ | ||
"name": "embeddings", | ||
"type": "float32s" | ||
} | ||
] | ||
|
||
indexes = [ | ||
{ | ||
"name": "vectorIndex", | ||
"type": "flat", | ||
"params": { | ||
"dims": 3072, | ||
"metric": "L2" | ||
}, | ||
"column": "embeddings" | ||
} | ||
] | ||
|
||
table = db.create_table( | ||
table=os.getenv("KDBAI_TABLE"), | ||
schema=schema, | ||
indexes=indexes | ||
) | ||
|
||
print(f"The table named '{table.name}' now exists.") | ||
``` |