Skip to content

Commit

Permalink
Adding in new changes to support the new feature view type sortedFeat…
Browse files Browse the repository at this point in the history 
…ureView
  • Loading branch information
@Manisha4
Manisha4 committed Feb 23, 2025
1 parent 882c527 commit c5783b6
Show file tree
Hide file tree
Showing 13 changed files with 507 additions and 56 deletions.
78 changes: 40 additions & 38 deletions docs/reference/registry/registry-permissions.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,43 +3,45 @@

## API Endpoints and Permissions

| Endpoint | Resource Type | Permission | Description |
| ------------------------ |---------------------|------------------------| -------------------------------------------------------------- |
| ApplyEntity | Entity | Create, Update, Delete | Apply an entity to the registry |
| GetEntity | Entity | Read | Get an entity from the registry |
| ListEntities | Entity | Read | List entities in the registry |
| DeleteEntity | Entity | Delete | Delete an entity from the registry |
| ApplyDataSource | DataSource | Create, Update, Delete | Apply a data source to the registry |
| GetDataSource | DataSource | Read | Get a data source from the registry |
| ListDataSources | DataSource | Read | List data sources in the registry |
| DeleteDataSource | DataSource | Delete | Delete a data source from the registry |
| ApplyFeatureView | FeatureView | Create, Update, Delete | Apply a feature view to the registry |
| GetFeatureView | FeatureView | Read | Get a feature view from the registry |
| ListFeatureViews | FeatureView | Read | List feature views in the registry |
| DeleteFeatureView | FeatureView | Delete | Delete a feature view from the registry |
| GetStreamFeatureView | StreamFeatureView | Read | Get a stream feature view from the registry |
| ListStreamFeatureViews | StreamFeatureView | Read | List stream feature views in the registry |
| GetOnDemandFeatureView | OnDemandFeatureView | Read | Get an on-demand feature view from the registry |
| ListOnDemandFeatureViews | OnDemandFeatureView | Read | List on-demand feature views in the registry |
| ApplyFeatureService | FeatureService | Create, Update, Delete | Apply a feature service to the registry |
| GetFeatureService | FeatureService | Read | Get a feature service from the registry |
| ListFeatureServices | FeatureService | Read | List feature services in the registry |
| DeleteFeatureService | FeatureService | Delete | Delete a feature service from the registry |
| ApplySavedDataset | SavedDataset | Create, Update, Delete | Apply a saved dataset to the registry |
| GetSavedDataset | SavedDataset | Read | Get a saved dataset from the registry |
| ListSavedDatasets | SavedDataset | Read | List saved datasets in the registry |
| DeleteSavedDataset | SavedDataset | Delete | Delete a saved dataset from the registry |
| ApplyValidationReference | ValidationReference | Create, Update, Delete | Apply a validation reference to the registry |
| GetValidationReference | ValidationReference | Read | Get a validation reference from the registry |
| ListValidationReferences | ValidationReference | Read | List validation references in the registry |
| DeleteValidationReference| ValidationReference | Delete | Delete a validation reference from the registry |
| ApplyPermission | Permission | Create, Update, Delete | Apply a permission to the registry |
| GetPermission | Permission | Read | Get a permission from the registry |
| ListPermissions | Permission | Read | List permissions in the registry |
| DeletePermission | Permission | Delete | Delete a permission from the registry |
| Commit | | None | Commit changes to the registry |
| Refresh | | None | Refresh the registry |
| Proto | | None | Get the proto representation of the registry |
| Endpoint | Resource Type | Permission | Description |
|---------------------------|---------------------|------------------------|-------------------------------------------------|
| ApplyEntity | Entity | Create, Update, Delete | Apply an entity to the registry |
| GetEntity | Entity | Read | Get an entity from the registry |
| ListEntities | Entity | Read | List entities in the registry |
| DeleteEntity | Entity | Delete | Delete an entity from the registry |
| ApplyDataSource | DataSource | Create, Update, Delete | Apply a data source to the registry |
| GetDataSource | DataSource | Read | Get a data source from the registry |
| ListDataSources | DataSource | Read | List data sources in the registry |
| DeleteDataSource | DataSource | Delete | Delete a data source from the registry |
| ApplyFeatureView | FeatureView | Create, Update, Delete | Apply a feature view to the registry |
| GetFeatureView | FeatureView | Read | Get a feature view from the registry |
| ListFeatureViews | FeatureView | Read | List feature views in the registry |
| DeleteFeatureView | FeatureView | Delete | Delete a feature view from the registry |
| GetSortedFeatureView | SortedFeatureView | Read | Get a sorted feature view from the registry |
| ListSortedFeatureViews | SortedFeatureView | Read | List sorted feature views in the registry |
| GetStreamFeatureView | StreamFeatureView | Read | Get a stream feature view from the registry |
| ListStreamFeatureViews | StreamFeatureView | Read | List stream feature views in the registry |
| GetOnDemandFeatureView | OnDemandFeatureView | Read | Get an on-demand feature view from the registry |
| ListOnDemandFeatureViews | OnDemandFeatureView | Read | List on-demand feature views in the registry |
| ApplyFeatureService | FeatureService | Create, Update, Delete | Apply a feature service to the registry |
| GetFeatureService | FeatureService | Read | Get a feature service from the registry |
| ListFeatureServices | FeatureService | Read | List feature services in the registry |
| DeleteFeatureService | FeatureService | Delete | Delete a feature service from the registry |
| ApplySavedDataset | SavedDataset | Create, Update, Delete | Apply a saved dataset to the registry |
| GetSavedDataset | SavedDataset | Read | Get a saved dataset from the registry |
| ListSavedDatasets | SavedDataset | Read | List saved datasets in the registry |
| DeleteSavedDataset | SavedDataset | Delete | Delete a saved dataset from the registry |
| ApplyValidationReference | ValidationReference | Create, Update, Delete | Apply a validation reference to the registry |
| GetValidationReference | ValidationReference | Read | Get a validation reference from the registry |
| ListValidationReferences | ValidationReference | Read | List validation references in the registry |
| DeleteValidationReference | ValidationReference | Delete | Delete a validation reference from the registry |
| ApplyPermission | Permission | Create, Update, Delete | Apply a permission to the registry |
| GetPermission | Permission | Read | Get a permission from the registry |
| ListPermissions | Permission | Read | List permissions in the registry |
| DeletePermission | Permission | Delete | Delete a permission from the registry |
| Commit | | None | Commit changes to the registry |
| Refresh | | None | Refresh the registry |
| Proto | | None | Get the proto representation of the registry |

## How to configure Authentication and Authorization
Please refer the [page](./../../../docs/getting-started/concepts/permission.md) for more details on how to configure authentication and authorization.
Please refer the [page](./../../../docs/getting-started/concepts/permission.md) for more details on how to configure authentication and authorization.
2 changes: 2 additions & 0 deletions sdk/python/feast/__init__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
from .on_demand_feature_view import OnDemandFeatureView
from .project import Project
from .repo_config import RepoConfig
from .sorted_feature_view import SortedFeatureView
from .stream_feature_view import StreamFeatureView
from .value_type import ValueType

Expand All @@ -42,6 +43,7 @@
"FeatureService",
"FeatureStore",
"FeatureView",
"SortedFeatureView",
"OnDemandFeatureView",
"RepoConfig",
"StreamFeatureView",
Expand Down
133 changes: 133 additions & 0 deletions sdk/python/feast/expediagroup/pydantic_models/feature_view_model.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from pydantic import BaseModel, field_serializer, field_validator
from typing_extensions import Self

from feast import SortedFeatureView, ValueType
from feast.expediagroup.pydantic_models.data_source_model import (
AnyBatchDataSource,
KafkaSourceModel,
Expand All @@ -24,6 +25,8 @@
from feast.feature_view import FeatureView
from feast.feature_view_projection import FeatureViewProjection
from feast.on_demand_feature_view import OnDemandFeatureView
from feast.protos.feast.core.SortedFeatureView_pb2 import SortOrder
from feast.sort_key import SortKey
from feast.transformation.pandas_transformation import PandasTransformation
from feast.transformation.python_transformation import PythonTransformation
from feast.transformation.substrait_transformation import SubstraitTransformation
Expand Down Expand Up @@ -312,6 +315,136 @@ def from_substrait_transformation(
)


class SortedFeatureViewSortKeyModel(BaseModel):
"""
Pydantic Model for a SortedFeatureView's sort key.
- name: string
- value_type: string (e.g., "INT64", "FLOAT", etc.)
- default_sort_order: string ("ASC" or "DESC")
- tags: map<string, string>
- description: string
"""

name: str
value_type: str
default_sort_order: str
tags: Optional[Dict[str, str]] = None
description: Optional[str] = ""

def to_sort_key(self) -> SortKey:
sort_order = (
SortOrder.ASC
if self.default_sort_order.upper() == "ASC"
else SortOrder.DESC
)
return SortKey(
name=self.name,
value_type=ValueType(self.value_type),
default_sort_order=sort_order,
tags=self.tags or {},
description=self.description or "",
)

@classmethod
def from_sort_key(cls, sort_key: SortKey) -> "SortedFeatureViewSortKeyModel":
return cls(
name=sort_key.name,
value_type=str(sort_key.value_type),
default_sort_order=sort_key.default_sort_order.name,
tags=sort_key.tags,
description=sort_key.description,
)


class SortedFeatureViewModel(FeatureViewModel):
"""
Pydantic Model for a Feast SortedFeatureView.
Extends FeatureViewModel by adding the sort_keys field.
"""

sort_keys: List[SortedFeatureViewSortKeyModel]

def to_feature_view(self) -> SortedFeatureView:
"""
Converts this Pydantic model into a SortedFeatureView Python object.
"""
# Convert sources from their pydantic representations.
batch_source = self.batch_source.to_data_source() if self.batch_source else None
stream_source = (
self.stream_source.to_data_source() if self.stream_source else None
)
source = stream_source if stream_source else batch_source
if stream_source and batch_source:
source.batch_source = batch_source

# Build the SortedFeatureView using base fields from FeatureViewModel plus sort_keys.
sorted_fv = SortedFeatureView(
name=self.name,
source=source,
schema=(
[schema.to_field() for schema in self.original_schema]
if self.original_schema
else None
),
entities=[entity.to_entity() for entity in self.original_entities],
ttl=self.ttl,
online=self.online,
description=self.description,
tags=self.tags or None,
owner=self.owner,
sort_keys=[sk.to_sort_key() for sk in self.sort_keys],
)
sorted_fv.materialization_intervals = self.materialization_intervals
sorted_fv.created_timestamp = self.created_timestamp
sorted_fv.last_updated_timestamp = self.last_updated_timestamp
return sorted_fv

@classmethod
def from_feature_view(
cls, sorted_feature_view: SortedFeatureView
) -> "SortedFeatureViewModel":
"""
Converts a SortedFeatureView Python object into its Pydantic model representation.
"""
batch_source = None
if sorted_feature_view.batch_source:
batch_source = KafkaSourceModel.from_data_source(
sorted_feature_view.batch_source
)
stream_source = None
if sorted_feature_view.stream_source:
stream_source = KafkaSourceModel.from_data_source(
sorted_feature_view.stream_source
)

return cls(
name=sorted_feature_view.name,
original_entities=[
EntityModel.from_entity(e)
for e in sorted_feature_view.original_entities
],
ttl=sorted_feature_view.ttl,
original_schema=(
[FieldModel.from_field(f) for f in sorted_feature_view.original_schema]
if sorted_feature_view.original_schema
else None
),
batch_source=batch_source,
stream_source=stream_source,
online=sorted_feature_view.online,
description=sorted_feature_view.description,
tags=sorted_feature_view.tags or None,
owner=sorted_feature_view.owner,
materialization_intervals=sorted_feature_view.materialization_intervals,
created_timestamp=sorted_feature_view.created_timestamp,
last_updated_timestamp=sorted_feature_view.last_updated_timestamp,
sort_keys=[
SortedFeatureViewSortKeyModel.from_sort_key(sk)
for sk in sorted_feature_view.sort_keys
],
)


class OnDemandFeatureViewModel(BaseFeatureViewModel):
"""
Pydantic Model of a Feast OnDemandFeatureView.
Expand Down
5 changes: 5 additions & 0 deletions sdk/python/feast/feast_object.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,17 +17,20 @@
from .protos.feast.core.OnDemandFeatureView_pb2 import OnDemandFeatureViewSpec
from .protos.feast.core.Permission_pb2 import PermissionSpec as PermissionSpec
from .protos.feast.core.SavedDataset_pb2 import SavedDatasetSpec
from .protos.feast.core.SortedFeatureView_pb2 import SortedFeatureViewSpec
from .protos.feast.core.StreamFeatureView_pb2 import StreamFeatureViewSpec
from .protos.feast.core.ValidationProfile_pb2 import (
ValidationReference as ValidationReferenceProto,
)
from .saved_dataset import SavedDataset, ValidationReference
from .sorted_feature_view import SortedFeatureView
from .stream_feature_view import StreamFeatureView

# Convenience type representing all Feast objects
FeastObject = Union[
Project,
FeatureView,
SortedFeatureView,
OnDemandFeatureView,
BatchFeatureView,
StreamFeatureView,
Expand All @@ -42,6 +45,7 @@
FeastObjectSpecProto = Union[
ProjectSpec,
FeatureViewSpec,
SortedFeatureViewSpec,
OnDemandFeatureViewSpec,
StreamFeatureViewSpec,
EntitySpecV2,
Expand All @@ -55,6 +59,7 @@
ALL_RESOURCE_TYPES = list(get_args(FeastObject))
ALL_FEATURE_VIEW_TYPES = [
FeatureView,
SortedFeatureView,
OnDemandFeatureView,
BatchFeatureView,
StreamFeatureView,
Expand Down
Loading

0 comments on commit c5783b6

Please sign in to comment.