Bump Version 0.4.0

.github/workflows/ci_templates.yaml

@@ -24,10 +24,8 @@ jobs:
         id: setup
         run: |
           if [[ "${{ github.base_ref }}" == "main" ]]; then
-            echo "mode=changed"
             echo "mode=changed" >> $GITHUB_ENV
           else
-            echo "mode=all"
             echo "mode=all" >> $GITHUB_ENV
           fi
 
@@ -83,17 +81,13 @@ jobs:
           echo "matrix_json: $matrix_json"
           echo "matrix_json=$matrix_json" >> $GITHUB_ENV
 
-
       - name: Set job output
         id: final_output
         run: |
           echo "::set-output name=matrix::$matrix_json"
-          echo "::set-output name=templates_exist::$TEMPLATES_EXIST"
-
 
   test_template:
     needs: template_update_check
-    if: needs.template_update_check.outputs.templates_exist != '0'
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false

CHANGELOG.md

@@ -11,6 +11,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 #### Changed defaults / behaviours
 
+#### New Features & Functionality
+
+#### Bug Fixes
+
+## [0.3.0](https://github.com/superduper-io/superduper/compare/0.4.0...0.3.0])    (2024-Nov-02)
+
+#### Changed defaults / behaviours
+
 - Change images docker superduper/<image> to superduperio/<image> 
 - Change the image's user from `/home/superduperdb` to `/home/superduper`
 - Add message broker service config

README.md

@@ -79,55 +79,12 @@ query.execute()
 
 Superduper may be run anywhere; you can also [contact us](https://superduper.io/contact) to learn more about the enterprise platform for bringing your Superduper workflows to production at scale. 
 
-## What's new in the `main` branch?
-
-We are working on an upcoming release of `0.4.0`. In this release we have:
-
-### Revamped how `Component` triggers initial computations and data dependent computations using `@trigger`
-
-This will enable a large diversity of `Component` types in addition to the well established `Model`, `Listener`, `VectorIndex`.
-
-### Created a general `CDC` (change-data-capture) base class
-
-This will allow developers to create a range of functionality which reacts to incoming data changes
-
-### Developed the concept of `Template` to enable re-usable units of complete functionality
-
-Components saved as `Template` instances, will allow users to easily redeploy their already deployed and tested `Component` and `Application` implementations, on alternative data sources, and with key parameters toggled to cater to operational requirements.
-
-### Added concrete `Template` implementations to the project
-
-These `Template` instances may be applied with Superduper with a simple single command
-
-```
-superduper apply <template> '{"variable_1": "value_1",  "variable_2": ...}'
-```
-
-or:
-
-```python
-from superduper import templates
-
-app = template(variable_1='value_1', variable_2='value_2', ...)
-
-db.apply(app)
-```
-
-### Added a user interface and new REST implementation
-
-Now you may view your `Component`, `Application` and `Template` instances in the user-interface, and execute queries using `QueryTemplate` instances, directly against the REST server.
-
-```
-superduper start
-```
-
 ## What does Superduper support?
 
 Superduper is flexible enough to support a huge range of AI techniques and paradigms. We have a range of pre-built functionality in the `plugins` and `templates` directories. In particular, Superduper excels when AI and data need to interact in a continuous and tightly integrated fashion. Here are some illustrative examples, which you may try out from our templates:
 
 - Semantic multimodal vector search ([images](https://github.com/superduper-io/superduper/tree/main/templates/multimodal_image_search), [text](https://github.com/superduper-io/superduper/tree/main/templates/text_vector_search), [video](https://github.com/superduper-io/superduper/tree/main/templates/multimodal_video_search))
 - [Retrieval augmented generation](https://github.com/superduper-io/superduper/tree/main/templates/retrieval_augmented_generation) with specialized requirements (data fetching involves semantic search as well as business rules and pre-processing)
-- Object detection directly on database hosted images
 - [LLM finetuning on database hosted data](https://github.com/superduper-io/superduper/tree/main/templates/llm_finetuning)
 - [Transfer learning using multimodal data](https://github.com/superduper-io/superduper/tree/main/templates/transfer_learning)
 
@@ -168,6 +125,49 @@ Implement vector search and embedding generation without requiring a dedicated v
 
 Superduper's REST API, allows installed models to be served without additional development work. For enterprise grade scalability, fail safes, security and logging, applications and workflows created with Superduper, may be deployed in one click on [Superduper enterprise](https://superduper.io/contact).
 
+
+## What's new in the `main` branch?
+
+We are working on an upcoming release of `0.4.0`. In this release we have:
+
+### Revamped how `Component` triggers initial computations and data dependent computations using `@trigger`
+
+This will enable a large diversity of `Component` types in addition to the well established `Model`, `Listener`, `VectorIndex`.
+
+### Created a general `CDC` (change-data-capture) base class
+
+This will allow developers to create a range of functionality which reacts to incoming data changes
+
+### Developed the concept of `Template` to enable re-usable units of complete functionality
+
+Components saved as `Template` instances, will allow users to easily redeploy their already deployed and tested `Component` and `Application` implementations, on alternative data sources, and with key parameters toggled to cater to operational requirements.
+
+### Added concrete `Template` implementations to the project
+
+These `Template` instances may be applied with Superduper with a simple single command
+
+```
+superduper apply <template> '{"variable_1": "value_1",  "variable_2": ...}'
+```
+
+or:
+
+```python
+from superduper import templates
+
+app = template(variable_1='value_1', variable_2='value_2', ...)
+
+db.apply(app)
+```
+
+### Added a user interface and new REST implementation
+
+Now you may view your `Component`, `Application` and `Template` instances in the user-interface, and execute queries using `QueryTemplate` instances, directly against the REST server.
+
+```
+superduper start
+```
+
 ## Getting started
 
 **Installation**:

VERSION

@@ -1 +1 @@
-0.4.0.dev
+0.4.0

superduper/__init__.py

@@ -9,7 +9,7 @@
 
 logging = logger.Logging
 
-__version__ = '0.4.0.dev'
+__version__ = '0.4.0'
 
 
 from .base.decorators import code

templates/llm_finetuning/component.json

@@ -24,19 +24,19 @@
       "method": "dill",
       "encodable": "artifact"
     },
-    "a026ecb590e04810d1e0f9d4ef7195e74c3614c3": {
+    "3b76f9f4db5089b20c14142b58f7a485faa38c43": {
       "_path": "superduper.components.datatype.Artifact",
       "datatype": "?datatype:dill",
       "uri": null,
-      "blob": "&:blob:a026ecb590e04810d1e0f9d4ef7195e74c3614c3"
+      "blob": "&:blob:3b76f9f4db5089b20c14142b58f7a485faa38c43"
     },
     "dataset:llm_finetuning": {
       "_path": "superduper.components.dataset.RemoteData",
       "upstream": null,
       "plugins": null,
       "cache": true,
       "status": null,
-      "getter": "?a026ecb590e04810d1e0f9d4ef7195e74c3614c3"
+      "getter": "?3b76f9f4db5089b20c14142b58f7a485faa38c43"
     },
     "table:sample_llm_finetuning": {
       "_path": "superduper.components.table.Table",

templates/simple_rag/build.ipynb

@@ -42,7 +42,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "id": "3ef70f6d-a189-460a-8864-241a689624e2",
    "metadata": {
     "editable": true,
@@ -66,7 +66,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "id": "cb029a5e-fedf-4f07-8a31-d220cfbfbb3d",
    "metadata": {
     "editable": true,
@@ -75,22 +75,7 @@
     },
     "tags": []
    },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "2024-Oct-31 09:37:53.82| INFO     | superduper.misc.plugins:13   | Loading plugin: mongodb\n",
-      "2024-Oct-31 09:37:53.87| INFO     | superduper.base.datalayer:73   | Building Data Layer\n",
-      "2024-Oct-31 09:37:53.87| INFO     | superduper.base.build:184  | Configuration: \n",
-      " +---------------+--------------+\n",
-      "| Configuration |    Value     |\n",
-      "+---------------+--------------+\n",
-      "|  Data Backend | mongomock:// |\n",
-      "+---------------+--------------+\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "from superduper import superduper, CFG\n",
     "\n",
@@ -103,7 +88,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": null,
    "id": "4e7902bd",
    "metadata": {
     "editable": true,
@@ -129,7 +114,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "id": "1ef8dd07-1b47-4dce-84dd-a081d1f5ee9d",
    "metadata": {},
    "outputs": [],
@@ -151,7 +136,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": null,
    "id": "c5965fdf",
    "metadata": {},
    "outputs": [],
@@ -185,7 +170,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": null,
    "id": "2d20eaa0-a416-4483-938e-23f79845739a",
    "metadata": {},
    "outputs": [],
@@ -213,7 +198,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": null,
    "id": "93d21872-d4dc-40dc-abab-fb07ba102ea3",
    "metadata": {},
    "outputs": [],
@@ -232,7 +217,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": null,
    "id": "31900eec-b516-4bef-939e-2e8f46252b12",
    "metadata": {},
    "outputs": [],
@@ -280,7 +265,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": null,
    "id": "a9b1f538-65ca-499e-b6d0-2dd733f81723",
    "metadata": {},
    "outputs": [],
@@ -307,7 +292,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": null,
    "id": "4663fa4b-c2ec-427d-bf8b-b8b109cc2ccf",
    "metadata": {},
    "outputs": [],
@@ -330,7 +315,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": null,
    "id": "509c3505-54c5-4e68-84ec-3df8bea0fd74",
    "metadata": {},
    "outputs": [],
@@ -350,7 +335,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": null,
    "id": "f98e5ff4",
    "metadata": {},
    "outputs": [],
@@ -375,7 +360,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": null,
    "id": "44baeb09-6f35-4cf2-b814-46283a59f7e9",
    "metadata": {},
    "outputs": [],
@@ -402,7 +387,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": null,
    "id": "2d3a0d3a-da1c-41ec-b16c-f281c46ad794",
    "metadata": {},
    "outputs": [],
@@ -413,7 +398,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": null,
    "id": "2a82ea22-9694-4c65-b72f-c89ae49d1ab2",
    "metadata": {},
    "outputs": [],
@@ -432,7 +417,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": null,
    "id": "e6787c78-4b14-4a72-818b-450408a74331",
    "metadata": {},
    "outputs": [],
@@ -451,7 +436,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
+   "execution_count": null,
    "id": "e7c16557-af76-4e70-83d9-2984e19a9554",
    "metadata": {},
    "outputs": [],
@@ -478,19 +463,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
+   "execution_count": null,
    "id": "2e850c03-33c6-4c88-95d3-d14146a6a0af",
    "metadata": {},
-   "outputs": [
-    {
-     "ename": "SyntaxError",
-     "evalue": "invalid syntax. Perhaps you forgot a comma? (2968979026.py, line 39)",
-     "output_type": "error",
-     "traceback": [
-      "\u001b[0;36m  Cell \u001b[0;32mIn[18], line 39\u001b[0;36m\u001b[0m\n\u001b[0;31m    'databackend': {\u001b[0m\n\u001b[0m                   ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax. Perhaps you forgot a comma?\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "from superduper import Template, Table, Schema\n",
     "from superduper.components.dataset import RemoteData\n",
@@ -542,7 +518,7 @@
     "        'api_key': {\n",
     "            'type': 'str',\n",
     "            'optional': True,\n",
-    "            'default': 'ollama'\n",
+    "            'default': None,\n",
     "        }\n",
     "    }\n",
     ")"