[FSTORE-1509] Feature logging (#276)

* fraud batch logging * fix online fraud * async feature logging * online logging * fix batch * log online model * remove redundant import * use hw model
logicalclocks · Oct 15, 2024 · 724817d · 724817d
1 parent 5eeb9e3
commit 724817d
Show file tree

Hide file tree

Showing 5 changed files with 194 additions and 76 deletions.
diff --git a/fraud_batch/1_fraud_batch_feature_pipeline.ipynb b/fraud_batch/1_fraud_batch_feature_pipeline.ipynb
@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "f60c52ce",
+   "id": "5d5afc66",
    "metadata": {
     "tags": []
    },
@@ -27,7 +27,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7ec7724a",
+   "id": "d20eee58",
    "metadata": {},
    "source": [
     "## <span style='color:#ff5f27'> 📝 Imports"
@@ -36,7 +36,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "42efa933",
+   "id": "802d6425",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -46,7 +46,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "a4120a95",
+   "id": "bd55a691",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -64,7 +64,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "010286f1",
+   "id": "879a539d",
    "metadata": {},
    "source": [
     "## <span style=\"color:#ff5f27;\"> 💽 Loading the Data </span>\n",
@@ -84,7 +84,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "491a37b4",
+   "id": "586d4111",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -100,7 +100,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "9ed8c4f0",
+   "id": "0cdb9c3d",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -118,7 +118,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "2543c511",
+   "id": "5a6fb4f4",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -135,15 +135,15 @@
   },
   {
    "cell_type": "markdown",
-   "id": "717f9102",
+   "id": "fbb301ce",
    "metadata": {},
    "source": [
     "---"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "16357302",
+   "id": "bf6f9299",
    "metadata": {},
    "source": [
     "## <span style=\"color:#ff5f27;\"> 🛠️ Feature Engineering </span>\n",
@@ -158,7 +158,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "0b919bf8",
+   "id": "d1af85e4",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -181,7 +181,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "5d6fd9ae",
+   "id": "4956cb52",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -191,7 +191,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "92820fab",
+   "id": "bc4e505f",
    "metadata": {},
    "source": [
     "Next, you will create features that for each credit card aggregate data from multiple time steps.\n",
@@ -203,7 +203,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "47116f13",
+   "id": "ee689954",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -223,7 +223,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0b3abcfc",
+   "id": "ed7039bb",
    "metadata": {},
    "source": [
     "Next lets compute windowed aggregates. Here you will use 4-hour windows, but feel free to experiment with different window lengths by setting `window_len` below to a value of your choice."
@@ -232,7 +232,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "f9bf9dd7",
+   "id": "791708c4",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -248,7 +248,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b3021fa4",
+   "id": "391e478f",
    "metadata": {},
    "source": [
     "### <span style=\"color:#ff5f27;\">⚙️ Convert date time object to unix epoch in milliseconds </span>"
@@ -257,7 +257,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "837bafd5",
+   "id": "bc6bcae3",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -270,7 +270,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a3df26a9",
+   "id": "bc3fec23",
    "metadata": {},
    "source": [
     "## <span style=\"color:#ff5f27;\">👮🏻‍♂️ Great Expectations </span>"
@@ -279,7 +279,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "c9391579",
+   "id": "cbc3ca3c",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -299,7 +299,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "8969ff27",
+   "id": "b3891fe5",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -340,23 +340,23 @@
   },
   {
    "cell_type": "markdown",
-   "id": "adf03efd",
+   "id": "da2f174f",
    "metadata": {},
    "source": [
     "---"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7c069f5a",
+   "id": "c48bbff9",
    "metadata": {},
    "source": [
     "## <span style=\"color:#ff5f27;\"> 📡 Connecting to Hopsworks Feature Store </span>"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ac32437d",
+   "id": "4d68f207",
    "metadata": {
     "tags": []
    },
@@ -373,7 +373,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "287491fa",
+   "id": "4c259c35",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -386,7 +386,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "23b61089",
+   "id": "0f80ccc5",
    "metadata": {},
    "source": [
     "To create a feature group you need to give it a name and specify a primary key. It is also good to provide a description of the contents of the feature group and a version number, if it is not defined it will automatically be incremented to `1`."
@@ -395,7 +395,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "58353322",
+   "id": "c78c614a",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -412,7 +412,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "832333fa",
+   "id": "45fcf5c3",
    "metadata": {},
    "source": [
     "A full list of arguments can be found in the [documentation](https://docs.hopsworks.ai/feature-store-api/latest/generated/api/feature_store_api/#create_feature_group).\n",
@@ -423,7 +423,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "7f82e4a2",
+   "id": "e97b5aab",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -435,7 +435,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "0c6832a9",
+   "id": "2f6cc0c0",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -462,7 +462,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "36f7e30a",
+   "id": "a57846b4",
    "metadata": {},
    "source": [
     "At the creation of the feature group, you will be prompted with an URL that will directly link to it; there you will be able to explore some of the aspects of your newly created feature group.\n",
@@ -472,7 +472,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1250c3ce",
+   "id": "777e6d4e",
    "metadata": {},
    "source": [
     "You can move on and do the same thing for the feature group with our windows aggregation."
@@ -481,7 +481,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "6b8ef9f6",
+   "id": "35ca5bbb",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -498,7 +498,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "1970657b",
+   "id": "e16aa93e",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -510,7 +510,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "68b0c84f",
+   "id": "2d80f5e2",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -530,7 +530,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e22dce87",
+   "id": "eb2254b9",
    "metadata": {},
    "source": [
     "Both feature groups are now accessible and searchable in the UI\n",
@@ -540,7 +540,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4f8b5d8a",
+   "id": "c6e783a2",
    "metadata": {},
    "source": [
     "## <span style=\"color:#ff5f27;\">⏭️ **Next:** Part 02: Training Pipeline\n",
@@ -555,7 +555,7 @@
    "hash": "e1ddeae6eefc765c17da80d38ea59b893ab18c0c0904077a035ef84cfe367f83"
   },
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -569,7 +569,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.11"
+   "version": "3.10.13"
   }
  },
  "nbformat": 4,

diff --git a/fraud_batch/2_fraud_batch_training_pipeline.ipynb b/fraud_batch/2_fraud_batch_training_pipeline.ipynb
@@ -186,7 +186,7 @@
     "    version=1,\n",
     "    query=selected_features,\n",
     "    labels=[\"fraud_label\"],\n",
-    "    transformation_functions=transformation_functions,\n",
+    "    transformation_functions=[label_encoder(\"category\")],\n",
     ")"
    ]
   },
@@ -482,7 +482,7 @@
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   },
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -496,7 +496,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.11"
+   "version": "3.10.13"
   }
  },
  "nbformat": 4,