Globus Flows wait time for notification

[dylanmcreynolds]

On a job at ALCF that takes 10 minutes before Globus tell us that it is done, roughly 25% of the time is waiting for Globus to poll the worker to notify that the job is complete. This is unfortunate.

We have been in contact with Globus about this and they are looking to reduce the rate to be better, but for a 10 minute job it still might take 60 seconds.

If we need to squeeze out more performance, we'll have to do something to notify our flow that the job is done. Some sort of queueing? Have the last step in the flow be to transfer a done file that we watch? I don't like any of these ideas, but I wanted to get this issue out.

[dylanmcreynolds]

@davramov