You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Writing a custom user DataSource is too difficult in PZ. It is made especially more complicated in the context of validation DataSources which are needed for sentinel (optimized) execution.
Users should only need to override a single get_item (or __iterator__) method which yields outputs from the DataSource. In my opinion, users should not need to know how to construct (or use) PZ's DataRecord class.
Instead, my proposal is the following:
users implement the get_item method, which takes an integer idx and returns a dict
the dict keys are the field names for the record to be returned (which must match those in the DataSource schema); the values are the values for those fields
PZ takes the output dict and converts it into a DataRecord on behalf of the user
One further step may include eliminating the need for users to specify a Schema at all; I will explore this option as part of the work on this issue.
The text was updated successfully, but these errors were encountered:
Writing a custom user
DataSource
is too difficult in PZ. It is made especially more complicated in the context of validationDataSource
s which are needed for sentinel (optimized) execution.Users should only need to override a single
get_item
(or__iterator__
) method which yields outputs from theDataSource
. In my opinion, users should not need to know how to construct (or use) PZ'sDataRecord
class.Instead, my proposal is the following:
get_item
method, which takes an integeridx
and returns adict
dict
keys are the field names for the record to be returned (which must match those in the DataSource schema); the values are the values for those fieldsdict
and converts it into aDataRecord
on behalf of the userOne further step may include eliminating the need for users to specify a
Schema
at all; I will explore this option as part of the work on this issue.The text was updated successfully, but these errors were encountered: