[Python][Bug] Yielded pipeline does not run on the specified task queue.

[mayankbhagya]

I have a pipeline 'P2' which has overridden the 'queue_name' property and returns 'Q2'.
I have another pipeline 'P1' which runs on the 'default' queue and yields 'P2'.
When yielded, the 'P2' task still runs on the 'default' queue rather than 'Q2'.

Is that a bug? Or is that behavior expected?

@aozarov and I had a related discussion here: http://stackoverflow.com/questions/31854634/set-queue-name-in-yielded-pipelines

[soundofjw]

Not expected but we have the same issue. Definitely worth investigating
further.

On Sun, Aug 9, 2015, 1:25 AM Mayank Bhagya [email protected] wrote:

I have a pipeline 'P2' which has overridden the 'queue_name' property and
returns 'Q2'.
I have another pipeline 'P1' which runs on the 'default' queue and yields
'P2'.
When yielded, the 'P2' task still runs on the 'default' queue rather than
'Q2'.

Is that a bug? Or is that behavior expected?

@aozarov https://github.com/aozarov

—
Reply to this email directly or view it on GitHub
#52.

[mayankbhagya]

I think that happens because the queue name is taken from _PipelineContext rather than the pipeline itself.

I am still not sure because I haven't investigated whether the yielded pipelines inherit the context from their parent pipelines.

[cdman]

Yes, I too would love to have this feature (controlling the task queue on a per-task rather than per-pipeline basis). My usecase is the following: I have some tasks which require a small number of larger (B4) instances while others require a large number of smaller (F2) instances. Thus I have two modules: one with auto-scaling F2s and the other with manual scaling B2s. I would be nice to integrate all this with pipelines (currently it's a Rube-Goldberg machine with tasks and asynchronous pipelines).