Introduce Template query

[mingshl]

Description

This Pull Request introduces the following key enhancements:

1. New 'Template' Query Type:

Implements a new query type that holds an object of query content.
Supports placeholders within the query content.
Passes the query content to search request processing.

2. Query Rewrite Context Refactoring:

Query Rewrite Context now carries the pipelineContext variables after all search request processors processed.
Template queries resolve placeholders during query rewrites.
Inner queries are constructed dynamically based on resolved placeholders.

Related Issues

#16823
opensearch-project/ml-commons#3054

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[mingshl]

@reta this PR addressed the issue #16823, can you please help move the issue to OpenSearch Project?

[github-actions]

❌ Gradle check result for ff1acc0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[mingshl]

updated git rebase HEAD~3 --signoff

[github-actions]

❌ Gradle check result for cf97416: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[reta]

@reta this PR addressed the issue opensearch-project/OpenSearch#16823, can you please help move the issue to OpenSearch Project?

@mingshl I sadly cannot do it (not a maintainers on ml-commons), @dblock @andrross could you?

[mingshl]

@reta this PR addressed the issue opensearch-project/OpenSearch#16823, can you please help move the issue to OpenSearch Project?

@mingshl I sadly cannot do it (not a maintainers on ml-commons), @dblock @andrross could you?

Thanks @reta. I will reach out.

[mingshl]

resolved changelog conflict.

[github-actions]

❌ Gradle check result for 86a56e1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[rishabhmaurya]

copying my comment from RFC
@mingshl Thanks for adding this support. I do agree on its utility in core, however, the ways it is exposed looks a little complicated to me -

GET my-keyword/_search
{
  "query": {
    "template": {
      "term": {
        "text": {
          "value": "${ext.ml_inference.params.inference_result}", // this is the field generated from ml_inference search request processor 
        }
      }
    }
  },
  "ext": {
    "ml_inference": {
      "params": {
        "text": "sunny"
      }
    }
},

1. is ext some reserved keyword in search pipeline world? If not, we can be more precise say search_pipeline_ext.
2. Assuming a template query can be associated with one or more request processors, for placeholders, we can probably remove redundant information and directly use something like - ${<processor_id>.<output_param>}, so this example will become ${ml_inference.inference_result}.
3. For inputs params, we probably need type information too?

"search_pipeline_ext": {
  "<processor_id>": {
    "params": {
       "<param_name>": {
          "value": <value>,
          "type": <type>
        },
        // OR shorthand notion when type isn't provided explicitly
        "<param_name> : <param_value>"
     }
  }
}

[mingshl]

copying my comment from RFC @mingshl Thanks for adding this support. I do agree on its utility in core, however, the ways it is exposed looks a little complicated to me -

GET my-keyword/_search
{
  "query": {
    "template": {
      "term": {
        "text": {
          "value": "${ext.ml_inference.params.inference_result}", // this is the field generated from ml_inference search request processor 
        }
      }
    }
  },
  "ext": {
    "ml_inference": {
      "params": {
        "text": "sunny"
      }
    }
},

1. is ext some reserved keyword in search pipeline world? If not, we can be more precise say search_pipeline_ext.
2. Assuming a template query can be associated with one or more request processors, for placeholders, we can probably remove redundant information and directly use something like - ${<processor_id>.<output_param>}, so this example will become ${ml_inference.inference_result}.
3. For inputs params, we probably need type information too?

"search_pipeline_ext": {
  "<processor_id>": {
    "params": {
       "<param_name>": {
          "value": <value>,
          "type": <type>
        },
        // OR shorthand notion when type isn't provided explicitly
        "<param_name> : <param_value>"
     }
  }
}

I think the ext is not reserved for search pipeline only. Different plugins have different search extensions. for ml_inference search extension is living in ml-commons repo.

And theoretically the field in search extensions can be used for multiple processors.
for example, ext.ml_inference.params.text field can be used for one ml inference processor to generate embedding, and the second ml inference processor to detect language. It's not linked with one processor.

But I like the idea that in the ml_inference search extension, in the future, we can let user define the interface of variables，but probably the definition of search extension can be part of model interface. Because it's likely that different model use the same variable name but in different types. That can be a future enhancement. tagging @b4sjoo here as he is commiter for model interface in ml-commons. cc @ylwu-amzn

[github-actions]

✅ Gradle check result for e5784fd: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 82.64463% with 21 lines in your changes missing coverage. Please review.

Project coverage is 72.25%. Comparing base (c6dfc65) to head (fcc44d5).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...pensearch/index/query/QueryCoordinatorContext.java	58.82%	7 Missing ⚠️
...g/opensearch/index/query/TemplateQueryBuilder.java	90.16%	3 Missing and 3 partials ⚠️
...pensearch/index/query/BaseQueryRewriteContext.java	88.57%	2 Missing and 2 partials ⚠️
...java/org/opensearch/index/query/QueryBuilders.java	0.00%	1 Missing ⚠️
...rg/opensearch/index/query/QueryRewriteContext.java	0.00%	1 Missing ⚠️
...rch/search/pipeline/PipelineProcessingContext.java	0.00%	1 Missing ⚠️
...g/opensearch/search/pipeline/PipelinedRequest.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #16818   +/-   ##
=========================================
  Coverage     72.24%   72.25%           
+ Complexity    65421    65403   -18     
=========================================
  Files          5306     5309    +3     
  Lines        304215   304300   +85     
  Branches      44118    44124    +6     
=========================================
+ Hits         219788   219866   +78     
+ Misses        66414    66377   -37     
- Partials      18013    18057   +44     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dbwiddis]

Some concerns here:

• breaking API
• synchronization issues
• I'm not sure your substitution is doing what you think it's doing (or it's doing something not expected by a casual reader)

[github-actions]

❌ Gradle check result for fcc44d5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for fcc44d5: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[dbwiddis]

LGTM thanks for your patience!