Workflow Eviction

[dabercro]

Right now, the console loops through all workflows in its database while updating errors seen. Since workflows with errors add up quickly, this now takes forever.

It would be good to have some way of looping through a subset of workflows. This can either be looking at workflows that are newer than some age, or only looking at workflows that appear with status "manual" in http://cms-unified.web.cern.ch/cms-unified/public/statuses.json (what the old console does).

@vlimant may be able to suggest a table or query that can be directly made to get the workflows that make the statuses.json page.

[dabercro]

@vargasa I would assign this to you, but it looks like you need to be a collaborator first. (Invite was just sent.) Let me know if you have any other questions on this issue.

[vlimant]

the json file on the web is not used, the console reads directly in oracle

https://github.com/CMSCompOps/wtc-console/blob/master/src/unified/models.py

[vlimant]

there seems to be something here

wtc-console/src/workflows/tasks.py

Line 149 in 1179441

# TODO old data cleanup here

as a hook for cleaning

[dabercro]

Ah, I guess I meant, the statuses.json can still be used as a way to determine what the loop should be. How are workflows evicted from that JSON file?

[vlimant]

do not tie to the json file. evicting anything that is not in "manual" anymore is an ok way.

[vlimant]

although, we might want to keep much more, if we are going to use the new console as the new unified report (which was one of the goal here)

[dabercro]

Okay @vargasa I think that means that you'll need to connect to the Oracle database to fill the tasks table. Also, we'll need to add a filter (status like "%manual%" for a first pass) that isn't there right now.

[vlimant]

wtc-console/src/workflows/tasks.py

Line 30 in 1179441

.filter(status__icontains='manual') \

maybe

[sharad1126]

assistance-manual, assistance-manual-recovered, assistance-announced-manual, assistance-announced-manual-recovered are the categories that should have workflows in WTC console because these are the ones that I need to take action on.

[vargasa]

I'll keep this here for reference: https://cms-unified.web.cern.ch/cms-unified/assistance.html#assistance-manual

[vargasa]

Hi @sharad1126 I can confirm that the following status are being fetched from Unified:

<QuerySet [u'assistance-agentfilemismatch-manual', u'assistance-agentfilemismatch-manual-recovered', u'assistance-announced-manual-recovered', u'assistance-manual', u'assistance-manual-recovered']>

This was done by logging:

UnifiedWorkflow.objects.filter(status__icontains='manual').values_list('status',flat=True).distinct()

Which provides a set of unique workflows status from unified that contains the string 'manual'. So maybe the problem is not here (tasks.py) but in the front-end

[vargasa]

@sharad1126 I took a couple of workflows from here and they are shown also on wtc-console as it is in my dev instance. How are you accessing wtc-console?

[sharad1126]

@vargasa I am generally accessing the WTC console through the unified error reports as it directly sends me to the console page of that particular workflow to take an action.

[vargasa]

@sharad Can you send me a link and/or the part of the report to check? Please

[dabercro]

My guess is that the link from Unified still points to the old tool. To make things easier for Sharad, I can try to add a link to the new console from the old one, unless @vlimant wants to make the change or addition in Unified.

[sharad1126]

@dabercro that's right. I'm still using the old tool. @vargasa here is the link of an error report - https://cms-unified.web.cern.ch/cms-unified/report/prebello_Run2018B-v1-JetHT-17Sep2018_pilot_1024p1_180917_170455_3850. It has an option on the top left called console. I use it to directly use the console which points to the default port. on vocms0113.cern.ch/seeworkflow

[vlimant]

in fine, the new wtc-console should have enough information in itself that the unified report would go away. the goal is indeed to have all information of the unified report located in mongodb and have enough visualization for the wtc to perform the operation.
we can add a link on the unified report pointing at the new console using a filtering url to restrict to that particular workflow, for now.
but again, the new console should have enough information in fine, so that the report and the assistance page is not necessary anymore

[sharad1126]

@vlimant As far as I know, the new console isn't giving me enough information regarding the errors. So for now, I would suggest that we use error reports from unified and then go to the console from there.

[vargasa]

It has an option on the top left called console. I use it to directly use the console which points to the default port

I see. This is from the other tool though as I see the cherrypy favicon on the vocms0113 link

the new console isn't giving me enough information regarding the errors.

@vlimant, @dabercro Is all the information in the report that @sharad1126 sent contained in Unified? If it is so as of now is not being retrieved by this console

[dabercro]

It sounds like Sharad wasn't using the displays in the old console either. We can handle the errors in #3 though. For this issue, we should just confirm that the workflows @sharad1126 needs are visible to him.