Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement sematic/lexcial search feature in datagateway search #1527

Draft
wants to merge 17 commits into
base: develop
Choose a base branch
from

Conversation

kennethnym
Copy link
Member

@kennethnym kennethnym commented May 18, 2023

Description

This PR adds a dropdown that lets the user choose to search with either Lucene or the new ML API, which provides endpoints for performing semantic/lexical search. The ML API utilizes OpenSearch and LLM to perform search operations.

DataGateway.Search.and.8.more.pages.-.Work.-.Microsoft.Edge.2023-08-31.17-47-19.mp4
  • Semantic/lexical search results are sorted based on their relevance to the query, from the most relevant to the least. They are displayed in a table just like other types of search results.
  • When semantic/lexical search is enabled, other search result types are disabled and the tabs are not shown
  • Only 50 results are shown at the moment. This is arbitrary (n_top_results)
  • The backend powering the feature has a temporary URL, so I just hardcoded it in the code, similar to Suggest similar investigations based on the currently displayed investigation on investigation landing page #1522

Please feel free to suggest changes to the UI.

cc @agbeltran

Testing instructions

Add a set up instructions describing how the reviewer should test the code

  • Review code
  • Check Actions build
  • Review changes to test coverage

Agile board tracking

Closes #1526

@kennethnym kennethnym added enhancement New feature or request datagateway-search Issues relating to the search plugin labels May 18, 2023
@agbeltran
Copy link
Member

cc @saifulkhan

@codecov
Copy link

codecov bot commented Jun 1, 2023

Codecov Report

Patch coverage: 71.77% and project coverage change: -0.43% ⚠️

Comparison is base (ec7d542) 96.20% compared to head (159eee0) 95.78%.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1527      +/-   ##
===========================================
- Coverage    96.20%   95.78%   -0.43%     
===========================================
  Files          161      164       +3     
  Lines         6885     6975      +90     
  Branches      2139     2154      +15     
===========================================
+ Hits          6624     6681      +57     
- Misses         240      273      +33     
  Partials        21       21              
Flag Coverage Δ
common 94.74% <47.82%> (-0.80%) ⬇️
dataview 96.85% <ø> (ø)
download 96.32% <ø> (ø)
search 95.75% <85.89%> (-0.76%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
packages/datagateway-common/src/app.types.tsx 100.00% <ø> (ø)
packages/datagateway-common/src/api/index.tsx 94.88% <33.33%> (-1.83%) ⬇️
packages/datagateway-common/src/api/ml.ts 47.36% <47.36%> (ø)
packages/datagateway-common/src/api/lucene.tsx 83.67% <55.55%> (-16.33%) ⬇️
...teway-search/src/searchPageContainer.component.tsx 92.41% <82.22%> (-2.36%) ⬇️
...search/src/search/searchTypeDropdown.component.tsx 85.71% <85.71%> (ø)
...teway-search/src/table/mlSearchTable.component.tsx 92.00% <92.00%> (ø)
...ateway-search/src/searchBoxContainer.component.tsx 100.00% <100.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

# Conflicts:
#	packages/datagateway-dataview/src/views/card/isis/isisDataPublicationsCardView.component.test.tsx
#	packages/datagateway-dataview/src/views/table/isis/isisStudiesTable.component.test.tsx
The new v2 ML API now supports lexical support. A new dropdown is added to let user choose between 3 search modes: Lucene, semantic, and lexical.
@kennethnym kennethnym changed the title Implement sematic search feature in datagateway search Implement sematic/lexcial search feature in datagateway search Aug 25, 2023
- Fix search not working when Lucene search mode selected
- Update tests to reflect URL params chhanges
New searchType param is added, so the expected value needs to be updated
@kennethnym
Copy link
Member Author

Updated demo video to reflect the new changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datagateway-search Issues relating to the search plugin enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add semantic search for testing
2 participants