Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search syntax documentation #443

Merged
merged 5 commits into from
Oct 13, 2023

Conversation

fabiovincenzi
Copy link
Collaborator

@fabiovincenzi fabiovincenzi commented Oct 12, 2023

fixes #327

@fabiovincenzi fabiovincenzi linked an issue Oct 12, 2023 that may be closed by this pull request
suprjinx
suprjinx previously approved these changes Oct 12, 2023
Copy link
Contributor

@suprjinx suprjinx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM


### Example with ```run.duration``` (numeric)

Select only the runs where the duration is exactly 60
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did say 60 in a previous comment, but I should have said 3600 🤦
And the reason I got confused is that we should explicitly mention that it's a duration in seconds.

Suggested change
Select only the runs where the duration is exactly 60
Select only the runs where the duration is exactly 3600 seconds (1 hour)

Comment on lines 161 to 170
### Example with ```run.archived``` (boolean)
Select only the runs where the archived attribute is true
```python
run.archived == True
```

Select only the runs where the archived attribute is not true
```python
run.archived != True
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please update those examples with implicit boolean operations

```

### Run parameters
Run parameters can be accessed via parameters.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Run parameters can be accessed via parameters.
Run parameters can be accessed via attributes.


### Filtering Runs with Unset Parameters

To filter runs based on whether an attribute is not set, you can use the following syntax:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To filter runs based on whether an attribute is not set, you can use the following syntax:
To filter runs based on whether a parameter is not set, you can use the following syntax:

```python
run.attribute is None
```
This expression will return runs for which the specified attribute is not defined.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This expression will return runs for which the specified attribute is not defined.
This expression will return runs for which the specified parameter is not defined.

Comment on lines 205 to 208
- run.archived can be either True or False.
- The duration of run must be greater than 0.
- The run has to contain a metric named 'TestMetric' and its last recorded value must be greater than 2.5.
- The name of run should not end with '4'.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add some formatting to these lines so they become more readable, like:
run.archived can be either True or False

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and same remark for the last example

Showing only the runs where param1 is not set
![FastTrackML Run List of not set param](images/search_runs_none_param.png)

### Filter Runs using Regular Expressions
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

regex is not specific to runs, it works for any string (e.g. metric.name) — it may unclear to users that it can work anywhere, maybe we should add regex ops to the string ops at the top and add an example of filtering metrics using regex?

For the ```string``` attributes you can use the following comparison operator:
- ``` == ```
- ``` != ```
- ``` in ```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in can also be used for checking whether something is in a list, e.g.

run.experiment in ["my-first-experiment", "my-second-experiment"]

Comment on lines 136 to 154
Select only the runs where the duration is not 60
```python
run.duration != 60
```

Select only the runs where the duration is greater than 60
```python
run.duration > 60
```

Select only the runs where the duration is greater or equal to 60
```python
run.duration >= 60
```

Select only the runs where the duration is less than 60
```python
run.duration < 60
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we shouldn't list all the numerical comparison operators here — let's just give 2 or 3 examples, like ==, >=, and <

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same remark for the metric.last example

Comment on lines 218 to 240
```python
metric.name == "TestRun1"
```

Select only the metrics where the name is different from "TestMetric1"
```python
metric.name != "TestRun1"
```

Select only the metrics where "Run1" is contained in the run name
```python
"Metric1" in metric.name
```

Select only the metrics where the name starts with "Test"
```python
metric.name.startswith('Test')
```

Select only the metrics where the name ends with "Metric1"
```python
metric.name.endswith('Metric1')
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should list all the string comparison operators here, especially as we gave them for runs too — maybe we just keep ==, startswith, and re.search

Copy link
Member

@jgiannuzzi jgiannuzzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great stuff, thanks @fabiovincenzi 👍

@fabiovincenzi fabiovincenzi merged commit ade8a52 into G-Research:main Oct 13, 2023
19 checks passed
jgiannuzzi pushed a commit to jgiannuzzi/fasttrackml that referenced this pull request Oct 16, 2023
Add documentation for metrics and runs search
jgiannuzzi pushed a commit to jgiannuzzi/fasttrackml that referenced this pull request Oct 16, 2023
Add documentation for metrics and runs search
suprjinx pushed a commit to suprjinx/fasttrackml that referenced this pull request Oct 23, 2023
Add documentation for metrics and runs search
@fabiovincenzi fabiovincenzi deleted the search-syntax-doc branch November 2, 2023 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Documentation of search syntax
3 participants