WIP: Experimental: Add JobList constraint parser and support constraint query strings in flux jobs -f
#5711
Conversation
|
We should decide if adding the constraint query support to |
IMO I think the the advanced query via I think it's more of an open debate if likewise, with the advanced filter/query in place, dunno if we could view some current command line options worthy of retirement (will I guess hidden)? I think |
Agreed. However, we could provide this feature on another command (e.g. |
|
The issue with supporting the query via an option is that quoting can get annoying. Note that in this WIP PR, I've allowed multiple $ flux jobs -f host:pi4 -f completed
JOBID USER NAME ST NTASKS NNODES TIME INFO
ƒ5hsApJ31 grondo hostname CD 1 1 0.205s pi4
$ flux jobs -f host:pi4 -f active
JOBID USER NAME ST NTASKS NNODES TIME INFO
ƒ5YsGD86K grondo sleep R 1 1 56.27s pi4 |
Ahhh I see your point now. Hmmm. I think there is value in One long term thought, what if inactive jobs are stored in a db long term. |
Yes, my hope is hat we can create a constraint->sql converter, but I haven't looked into it so that is a real concern. That might be an argument for keeping this kind of generic query syntax out of flux-jobs.
Yes, I definitely see your point (though I wouldn't use |
Problem: The parse_datetime() utility function is statically configured to assume future dates when a term like "Friday" is given. That is, instead of giving the date for the previous Friday, the function will return then next Friday instead. This behavior should be configurable. Add an assumeFuture parameter to the function which defaults to True. If set to False, then parse_datetime() will assume dates in the past instead.
Problem: Values are always returned as strings by the ConstraintParser parser, but there are times when another type may be more suitable. Additionally, there is currently no way to reduce a set of values to a single element if this is required or beneficial in the result. Add a convert_values mapping to the ConstraintParser class. If an operator is in this dictionary, then the values of a term are passed to the provided callable, which should return a new list of values as a result.
Problem: flux_job_strtostate(3) and flux_job_strtoresult(3) are not exposed in the Python API. Add strtostate() and strtoresult() to flux.job.info.
Problem: There is no user friendly way to create constraint objects that can be used with the job-list service. Add a JobListConstraintParser class which can be used to create RFC 31 constraing objects suitable for sending to the job-list service.
Problem: The flux-jobs(1) -f, --filter option is being updated to take a query string, but there is no "filter" based argparse action to handle this case. Add flux.util.FilterActionConcatenate which concatenates multiple --filter strings together, separated by space.
Problem: There is no way to provide an arbitrary constraint to flux-jobs(1). Update the `-f, --filter` option of flux-jobs(1) to take a query string that will be passed to the JobList constraint parameter instead of the filter paramter.
|
I've rebased this PR now that #6209 went in. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5711 +/- ##
==========================================
- Coverage 83.32% 83.26% -0.07%
==========================================
Files 521 521
Lines 85203 85338 +135
==========================================
+ Hits 70994 71054 +60
- Misses 14209 14284 +75
|
This is an experimental PR that adds a special
JobListConstraintParserclass to support a specialized constraint parser syntax forJobListmeant to be used withflux jobs -f, --filter. This is built on top of @chu11's PR #5656.At this point, this is just a proof-of-concept of how this could work to extend the
-f, --filteroption offlux jobsto support generalized constraint queries while being backwards compatible.The
JobListConstraintParserclass is a two pass parser. The first pass converts tokens without an operator to states and/or results bitmasks, usingorto join these together. The first pass can also be used to do other convenience conversions (likehost:orhosts:tohostlist:, or as is done in this prototype@dtconverted to(not (t_cleanup:<dt or t_run:>dt))to find jobs that were running at a given time). The second pass converts the now modified query string to a JSON constraint object suitable for thejob-listservice.Finally, a
constraintparameter is added to the Python job listing interfaces, andflux job list -f, --filteris converted to pass the option argument as a constraint.e.g.:
after running some sleep jobs: