[EPIC] Use Solr for search in DGU Find

[kentsanggds]

Why

To simplify DGU's architecture by removing OpenSearch and Publish, which will reduce maintenance burden.

What

Add Solr Gem to find as a feature so can switch between Opensearch and Solr

Search results page

Preview Give feedback

1. Add results page using Solr data #1318
   5 of 5
   +0
   

   
2. Update search term query and add filters on results page #1337
   +0
3. Topic filter should only include topics for datasets that are included in the search results #1335
   +0
   
4. Update Solr topic filter, fix removing filters and persist keywords #1344
   +0
   
5. Don’t clear search keywords when filters are applied
6. Fix removing filters
7. Format filter should only include valid formats for datafiles of datasets that are included in the search results #1336
   +0
   
8. OGL Licence filter doesn't return all results #1354
   +0
   
9. Ensure solr search results only include active datasets #1365
   +0
   
10. Prevent XSS: sanitise input (done by rails), use strong params (not needed)
11. Paginate solr search results #1368
    +0
    
12. Search results should be paginated #1339
    +0
    
13. Search separately for each individual whitespace-separated term #1340
    +0
    
14. Add handling for RSolr::Error::Http - 400 Bad Request error when invalid term is entered e.g. -
15. Scope Publisher filter to returned datasets (TIMEBOX) - this only works when searching by keywords
16. Show Availability for unreleased datasets (TIMEBOX, post launch)
17. (WON'T DO) Escape special Solr characters in query
18. Add search tips for no results found page #1378
    +0
    
19. Explicitly set Solr query operator to OR #1388
    +0
    

Individual dataset page

Preview Give feedback

1. Add dataset page to display individual search results from Solr #1316
   22 of 24
   +0
   
2. Update Licence details on dataset page #1359
   +0
   
3. Incorrect licence type sometimes shown in metadata box on the dataset page #1327
   +0
   
4. Add dataset's datafile preview page using Solr #1317
   0 of 4
   +0
   
5. Handle 'Not found' datasets e.g. /3c0d0074-05a3-491a-a144-89ea54654d47/aberdeen-city
6. Issue rendering /dataset/solr/41600666-30dc-4243-9b64-4be3e8f5c65f/spend-over-f25-000-in-2gether-nhs-foundation-trust (not in integration)
7. Legacy datasets - they will work when we switch over, it's a redirect
8. Only return active datasets #1387
   +0
   
9. Review OpenSearch tests for any edge cases
10. (NOT NOW) Formats should be derived e.g. /562eb38e-387a-440d-bc61-c7cb7e927fba/coastal-physiographic-features-barrier-beach (Resource format not guessed automatically ckan/ckan#1350)
11. (NOT NOW) Tests are slow and difficult to extend. WIP: https://github.com/alphagov/datagovuk_find/compare/main-solr...solr-use-factories-for-testing, basic dataset factory added since then for unit tests

Testing, pre-launch tasks

Preview Give feedback

1. Update the Changelog to inform users we're switching to Solr #1348
   +0
   
2. Errors appear in logs (Kibana)
3. Check Solr performance and carry out load testing
4. Manually test popular URLs https://lookerstudio.google.com/u/0/reporting/1dLIysu5Ie131ZL3gGT7zyWBhffNgTSX2/page/cdu
5. Errors appear in Sentry
6. Configure Sentry #1386
   +0
   
7. Double check that all indexes are available in production (for orgs)
8. Update routes, controllers and views to switch it over to Solr
9. Comms, scheduled maintenance?
10. Add a note to DGU documentation + /support page
11. Remove unused code and dependencies #1404
    +0
    
12. Remove /dashboard page #1398
    +0
    

Post launch fixes

Preview Give feedback

1. Only include datasets in the results (exclude harvesters) #1407
   +0
   
2. Cache organisations for 10 minutes #1405
   +0
   
3. Handle Solr connection errors #1402
   +0
   
4. Handle missing organisations #1401
   +0
   
5. Increase the limit on organization facet #1400
   +0
   
6. Fix previews when some data rows are nil #1397
   +0
   
7. Handle missing params #1396
   +0
   
8. Allow users to filter by other valid formats #1395
   +0
   
9. Force HTML request format #1394
   +0
   
10. Use strong parameters #1413
    +0
    

Post launch

Preview Give feedback

1. Remove OpenSearch from Find codebase
2. Retire DGU Publish govuk-infrastructure#1579
   DGU +1
   
3. Remove OpenSearch from infrastructure
4. Update architecture diagrams and docs

[kentsanggds]

We had a chat about how to proceed with progressing and reviewing the work in smaller chunks.

Proposal is to create another branch main-solr from which other solr update branches will be fed into. This will ensure that smaller PRs can be reviewed and merged into it without affecting any deployments to production.

The integration cluster can be updated to point to commits on this branch specifically when there is a need to deploy things for testing on EKS.

As there is generally very little development on the Find application I don't forsee any issues around the branch drifting from the main branch.