Refactoring to support orchestration history chunking #50
Conversation
- Refactor worker options to support orchestration worker-specific settings - Added payload size limit configuraiton - Several debug logging improvements - Improved handling of fatal errors - Major refactoring of OrchestrationRuntimeState to support history chunking - More filtering of duplicate and unnecessary events - Added new parameter to Backend.CompleteOrchestrationWorkItem
|
I will merge this after I validate that it works correctly with the Dapr state stores (like Cosmos DB, which is one of the main motivations for this change). |
| @@ -404,6 +404,8 @@ loop: | |||
| } | |||
|
|
|||
| // mustEmbedUnimplementedTaskHubSidecarServiceServer implements protos.TaskHubSidecarServiceServer | |||
| // | |||
| //lint:ignore U1000 because this is a required gRPC method | |||
| func (grpcExecutor) mustEmbedUnimplementedTaskHubSidecarServiceServer() { | |||
There was a problem hiding this comment.
What gRPC wants us to do here is to change type grpcExecutor struct and add an embed for unimplementedTaskHubSidecarServiceServer. We shouldn't have to implement this method
There was a problem hiding this comment.
I'm not sure what this means, unfortunately. Can you provide an example of the code change I can make that will allow me to delete this method?
There was a problem hiding this comment.
I think removing this method should be enough.
grpcExecutor already embeds unimplementedTaskHubSidecarServiceServer:
durabletask-go/backend/executor.go
Line 45 in 7c6a326
I tried doing it locally but right now this PR isn't building (at least not for me?) with some other really weird issues
Overview
This is a fairly "chunky" PR which refactors how we do orchestration state management. The motivation for this change is to resolve the state store limitation in Dapr Workflow.
With this change, the durable task engine now supports saving orchestration history in multiple chunks rather than all-at-once. This allows it to be compatible with state stores that have:
The changes are designed in a way to ensure that machine failures that occur in between saving "chunks" can be automatically recovered, with no data loss.
Summary of all changes