[WORKFLOW SDK BUG] Workflow Execution Scheduling Timeout

[relusion]

Expected Behavior

Workflows are expected to be scheduled and executed promptly when there is sufficient compute or queue capacity. They should retry automatically in the case of recoverable errors, and eventually complete as long as resources are available.

Actual Behavior

Workflows are being scheduled but are not executed. The Dapr sidecar continuously generates the following error messages:

"scheduling workflow execution with durabletask engine" app_id=consumer-app instance=daec2dca0950 scope=dapr.wfengine.backend.actors type=log ver=1.13.6" "Workflow actor '5db209e4-f5b9-405e-b5f2-25fc1a340bae': execution failed with a recoverable error and will be retried later: 'timed-out trying to schedule a workflow execution - this can happen if there are too many in-flight workflows or if the workflow engine isn't running: context deadline exceeded'" app_id=consumer-app instance=daec2dca0950 scope=dapr.wfengine.backend.actors type=log ver=1.13.6"

Despite allowing the system to wait and retry, the scheduled workflows are never executed, and this issue persists indefinitely. You may need to run the producer application multiple times before the issue surfaces, as sometimes the workflows execute without errors.

Reproduction Steps:

1. Clone the repository: https://github.com/relusion/dapr_pub_sub_workflow_case
2. Start the environment: Run docker-compose up --build to start RabbitMQ, Redis, and the Consumer application.
3. Generate messages:
   • Navigate to the producer directory.
   • Run python producer.py, which generates 100 messages and sends them to RabbitMQ.
4. Observe:
   • The Consumer application should schedule workflows for the incoming messages, but they will not be executed.
   • Check the Dapr logs for repeated errors similar to the one listed above.

[dapr-bot]

This issue has been automatically marked as stale because it has not had activity in the last 60 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[AndriyKalashnykov]

@DeepanshuA and @cgillum - any update on when this issue will be worked on?

[elena-kolevska]

Hey @relusion and @AndriyKalashnykov , sorry for the late response. I can look into this very soon. Since this issue was created we've had many changes in workflows on the dapr side, and we also deprecated the Dapr workflow client. Now you're supposed to use the DaprWorkflowClient to schedule new workflows.
Would be great if you can try that out, but if not I'll check it myself very soon and get back to you.