You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
. Thats a fine base default and target developers can override it if their target prefers smaller or larger batches.
The thing thats not supported today is the use case where I want to configure a target, say target-postgres, to output more frequently because my tap records are slow/expensive to get so the possibility of waiting until 10k records are retrieved, the batch is old enough to drain, or the sync completes, is undesirable. I might want to drain after every 50 records or so in some case.
One example that came up a while ago is from the map-gpt-embeddings plugin that requests from openai's api. It takes a while to embed all input data and each request costs money so preferably I'd drain as frequently as possible.
In the base sink class we define a
MAX_SIZE_DEFAULT
thats set to 10,000sdk/singer_sdk/sinks/core.py
Line 133 in e9f2c17
The thing thats not supported today is the use case where I want to configure a target, say target-postgres, to output more frequently because my tap records are slow/expensive to get so the possibility of waiting until 10k records are retrieved, the batch is old enough to drain, or the sync completes, is undesirable. I might want to drain after every 50 records or so in some case.
One example that came up a while ago is from the map-gpt-embeddings plugin that requests from openai's api. It takes a while to embed all input data and each request costs money so preferably I'd drain as frequently as possible.
cc @edgarrmondragon
The text was updated successfully, but these errors were encountered: