Fixing create index and use case input parsing bugs

[amitgalitz]

Description

Multiple bug fixes:

1. Address the inconsistency created by this issue: [BUG] Inconsistency for create index between REST and CreateIndex OpenSearch#12775, we want to have the same behavior as users today expect through the regular Create Index REST API. This means that we shouldn't expect _doc in the mapping and we should add it ourselves, this is exactly what core does on the rest layer but have missed doing on transport layer. https://github.com/opensearch-project/OpenSearch/blob/2.x/server/src/main/java/org/opensearch/rest/action/admin/indices/RestCreateIndexAction.java#L106

2. We know correctly expect arrays in the input fields for the default use cases, I make sure to correctly escape the strings weather we are expecting a numerical array or string array. For example:

"[\"text\", \"hello\"]" to "["text", "hello"]"), this is needed for processors that take in string arrays,
This also removes the quotations around the array making the array valid to consume
(e.g. "weights": "[0.7, 0.3]" -> "weights": [0.7, 0.3]) 

this is done through parsing these arrays into strings in RestCreateWorkflowAction (we currently support inputs of string, array, map of strings of strings) and handleing the backlashes in createpipelinestep to correctly send the body to the transport action

3. Added conversational search template without creating a conversation after discussing missing conversation API here: Added Conversation API in MLClient ml-commons#2211

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[owaiskazi19]

Let's wait for opensearch-project/ml-commons#2211 to be merged before merging this

You went with the default way of creating the memory id. I see

[dbwiddis]

LGTM with a few suggestions.

[amitgalitz]

Let's wait for opensearch-project/ml-commons#2211 to be merged before merging this

Yes had a conversation with Yaliang on this, for now users will have to create there own conversation and put into the rag processor, conversation as user created isn't too bad as these are basically per user conversation and should be created for each new one. If we created a conversation here and user got it from state index it's fairly similar amount of calls for the users, Yaliang suggested we auto create this conversation idea in RAG processor if processor doesn't get one in the search request. I overall agree with that, it is probably too tight for 2.13 though.

[owaiskazi19]

We are doing so much of regex. I am worried it might fail if user has some complex mappings or payload. Is there any parsing library we can use to achieve the same?

[owaiskazi19]

I overall agree with that, it is probably too tight for 2.13 though.

Can we have a follow up issue to handle it later?

[codecov]

Codecov Report

Attention: Patch coverage is 59.52381% with 17 lines in your changes are missing coverage. Please review.

Project coverage is 72.54%. Comparing base (149e22a) to head (cad55fa).

Files	Patch %	Lines
...search/flowframework/workflow/CreateIndexStep.java	50.00%	6 Missing and 3 partials ⚠️
.../org/opensearch/flowframework/util/ParseUtils.java	56.25%	6 Missing and 1 partial ⚠️
...h/flowframework/rest/RestCreateWorkflowAction.java	75.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x     #600      +/-   ##
============================================
- Coverage     72.75%   72.54%   -0.22%     
- Complexity      681      684       +3     
============================================
  Files            82       82              
  Lines          3491     3526      +35     
  Branches        279      285       +6     
============================================
+ Hits           2540     2558      +18     
- Misses          832      845      +13     
- Partials        119      123       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[amitgalitz]

@owaiskazi19 I changed method to what you suggested for ParseUtils.removingBackslashesAndQuotesInArrayInJsonString, I also made slight change to ParseUtils.parseStringToObjectMap() in latest commit

[owaiskazi19]

@amitgalitz we need to forward port this to main