diff --git a/docs/images/search_metrics_filter_by_run.png b/docs/images/search_metrics_filter_by_run.png new file mode 100644 index 000000000..cf714d436 Binary files /dev/null and b/docs/images/search_metrics_filter_by_run.png differ diff --git a/docs/images/search_runs.png b/docs/images/search_runs.png new file mode 100644 index 000000000..3eb5fbf99 Binary files /dev/null and b/docs/images/search_runs.png differ diff --git a/docs/images/search_runs_none_param.png b/docs/images/search_runs_none_param.png new file mode 100644 index 000000000..8aa0b981c Binary files /dev/null and b/docs/images/search_runs_none_param.png differ diff --git a/docs/images/search_runs_param_filter.png b/docs/images/search_runs_param_filter.png new file mode 100644 index 000000000..76bce160f Binary files /dev/null and b/docs/images/search_runs_param_filter.png differ diff --git a/docs/images/search_runs_regular_expression_match.png b/docs/images/search_runs_regular_expression_match.png new file mode 100644 index 000000000..85c5e9367 Binary files /dev/null and b/docs/images/search_runs_regular_expression_match.png differ diff --git a/docs/images/search_runs_regular_expression_search.png b/docs/images/search_runs_regular_expression_search.png new file mode 100644 index 000000000..62b570e73 Binary files /dev/null and b/docs/images/search_runs_regular_expression_search.png differ diff --git a/docs/search_syntax.md b/docs/search_syntax.md new file mode 100644 index 000000000..e2b9dcd82 --- /dev/null +++ b/docs/search_syntax.md @@ -0,0 +1,258 @@ +# Search Syntax Documentation +## Introduction +This document serves as a guide to the search syntax that enables users to effectively filter metrics and runs. + +The syntax follows Python conventions, in the sections that follow, we will delve into the specifics of the supported operations and provide examples. + +- [Search Runs](#search-runs) +- [Search Metrics](#search-metrics) +- [Operations](#operations) + - [String operations](#string-operations) + - [Numeric operations](#numeric-operations) + - [Boolean operations](#boolean-operations) + - [Implicit Boolean comparison](#implicit-boolean-comparison) + - [Logical operations](#logical-operations) + - [```in``` operator](#in-operator) +- [Search run examples](#search-run-examples) + - [Example with ```run.name``` (string)](#example-with-runname-string) + - [Example with ```run.duration``` (numeric)](#example-with-runduration-numeric) + - [Example with ```run.archived``` (boolean)](#example-with-runarchived-boolean) + - [Run parameters](#run-parameters) + - [Filtering Runs with Unset Parameters](#filtering-runs-with-unset-parameters) + - [Filter Runs using Regular Expressions](#filter-runs-using-regular-expressions) + - [Complex query for run search](#complex-query-for-run-search) +- [Search metrics examples](#search-metrics-examples) + - [Example with ```metric.name``` (string)](#example-with-metricname-string) + - [Example with ```metric.last``` (numeric)](#example-with-metriclast-numeric) + - [Filter Metrics by run](#filter-metrics-by-run) + - [Complex query for metric search](#complex-query-for-metric-search) + + +## Search Runs +You can filter the runs using the following attributes associated with the ```run``` object: +| Property | Description | Type | +| ---------------------- | --------------------------------------------------- | ---------------- | +| ```run.name``` | Run name | ```string``` | +| ```run.hash``` | Run hash | ```string``` | +| ```run.experiment``` | Experiment name | ```string``` | +| ```run.tags``` | List of run tags | ```dictionary``` | +| ```run.archived``` | True if run is archived, otherwise False | ```boolean``` | +| ```run.active``` | True if run is active(in progress), otherwise False | ```boolean``` | +| ```run.duration``` | Run duration in seconds | ```numeric``` | +| ```run.created_at``` | Run creation datetime | ```numeric``` | +| ```run.finalized_at``` | Run end datetime | ```numeric``` | +| ```run.metrics``` | Set of run metrics | ```dictionary``` | + +## Search Metrics +You can filter the metrics using the following metric attributes associated with the ```metric``` object: +| Property | Type | +| ----------------------- | ------------- | +| ```metric.name``` | ```string``` | +| ```metric.last``` | ```numeric``` | +| ```metric.last_step``` | ```numeric``` | +| ```metric.first_step``` | ```numeric``` | + +## Operations + +### String operations +For the ```string``` attributes you can use the following comparison operator: +- ``` == ``` +- ``` != ``` +- ``` in ``` +- ``` .startswith() ``` +- ``` .endswith() ``` +- ``` re.match() ``` +- ``` re.search() ``` + +### Numeric operations +For the ```numeric``` attributes you can use the following comparison operator: +- ``` == ``` +- ``` != ``` +- ``` > ``` +- ``` >= ``` +- ``` < ``` +- ``` <= ``` + +### Boolean operations +For the ```boolean``` attributes you can use the following comparison operator: +- ``` == ``` +- ``` != ``` + +#### Implicit Boolean comparison +You can also use boolean attributes directly in a conditional statement without any comparison operator. + +Equivalent to comparing the attribute to True. + +```python +run.archived # Checks if run.archived is equal to True +``` +Equivalent to comparing the attribute to False. +```python +not run.archived # Checks if run.archived is equal to False +``` +### Logical operations +You can create complex search queries combining multiple conditions with logical operators. +- ``` and ``` +- ``` or ``` +- ``` not ``` + +### ```in``` operator +The ```in``` operator can also be used for checking whether a value is in a list + +```python +run.experiment in ["my-first-experiment", "my-second-experiment"] +``` + +## Search run examples + +### Example with ```run.name``` (string) +Select only the runs where the name exactly matches "TestRun1" + +```python +run.name == "TestRun1" +``` + +Select only the runs where the name is different from "TestRun1" +```python +run.name != "TestRun1" +``` + +Select only the runs where "Run1" is contained in the run name +```python +"Run1" in run.name +``` + +Select only the runs where the name starts with "Test" +```python +run.name.startswith('Test') +``` + +Select only the runs where the name ends with "Run1" +```python +run.name.endswith('Run1') +``` + +### Example with ```run.duration``` (numeric) + +Select only the runs where the duration is exactly 3600 seconds (1 hour) + +```python +run.duration == 3600 +``` + +Select only the runs where the duration is greater or equal to 3600 seconds (1 hour) +```python +run.duration >= 3600 +``` + +Select only the runs where the duration is less than 3600 seconds (1 hour) +```python +run.duration < 3600 +``` + +### Example with ```run.archived``` (boolean) +Select only the runs where the archived attribute is true +```python +run.archived +``` + +Select only the runs where the archived attribute is not true +```python +not run.archived +``` + +### Run parameters +Run parameters can be accessed via attributes. +![FastTrackML Run List, param filter](images/search_runs_param_filter.png) + +### Filtering Runs with Unset Parameters + +To filter runs based on whether a parameter is not set, you can use the following syntax: + +```python +run.attribute is None +``` +This expression will return runs for which the specified parameter is not defined. + +Showing all the runs +![FastTrackML Run List](images/search_runs.png) + +Showing only the runs where param1 is not set +![FastTrackML Run List of not set param](images/search_runs_none_param.png) + +### Filter Runs using Regular Expressions + +Match finds an exact match at the beginning of a string. +![FastTrackML Run filter using regular expression match](images/search_runs_regular_expression_match.png) + +Search looks for a pattern anywhere in the string. +![FastTrackML Run filter using regular expression match](images/search_runs_regular_expression_search.png) + +### Complex query for run search +The query selects the runs that meet the following conditions: + + +- ```run.archived``` can be either ```True``` or ```False```. +- The duration of ```run``` must be greater than ```0```. +- The ```run``` has to contain a metric named ```'TestMetric'``` and its last recorded value must be greater than ```2.5```. +- The name of ```run``` should not end with ```'4'```. + +```python +(run.archived == True or run.archived == False) and run.duration > 0 and run.metrics['TestMetric'].last > 2.5 and not run.name.endswith('4') +``` + +## Search metrics examples + +### Example with ```metric.name``` (string) +Select only the metrics where the name exactly matches "TestMetric1" +```python +metric.name == "TestRun1" +``` + +Select only the metrics where the name starts with "Test" +```python +metric.name.startswith('Test') +``` + +Select only the metrics where the name contains "Metric1" +```python +re.search("Metric1", metric.name) +``` + +### Example with ```metric.last``` (numeric) + +Select only the metrics where the last value is exactly 1.1 + +```python +metric.last == 1.1 +``` + +Select only the metrics where the duration is greater or equal to 1.1 +```python +metric.last >= 1.1 +``` + +Select only the metrics where the last value is less than 1.1 +```python +metric.last < 1.1 +``` + +### Filter Metrics by run +You can also filter the metrics by combining metric attributes with run attributes. + +Showing the metrics with the last value greater than 6 belonging to a run with the name that starts with marvelous. + +![FastTrackML Metric List filter by metric and run attributes](images/search_metrics_filter_by_run.png) + +### Complex query for metric search +The query selects the metrics that meet the following conditions: + +- The ```metric.name``` field must be ```"TestMetric1"``` or ```"TestMetric2"```. +- The``` metric.last_step``` field must be greater than or equal to ```1```. +- The ```run.name``` field must either end with ```"2"``` or start with ```"TestRun1"```. +- The ```metric.last``` field must be less than ```1.6```. +- The ```run.duration``` must be greater than ```0```. + +```python +((metric.name == "TestMetric1") or (metric.name == "TestMetric2")) and metric.last_step >= 1 and (run.name.endswith("2") or re.match("TestRun1", run.name)) and (metric.last < 1.6) and run.duration > 0 +```