Solr Resiliency Issues

[nickumia-reisys]

Component: Solr
Version: 8.11
Environment: cloud.gov

This issue is primarily for historical reference. The specific issues may be no-ops in reality..

How to reproduce

1. Specify solr configuration
2. Create solr service
3. Load data into solr
4. Load test (high volume or deep data)

or

1. Specify solr configuration
2. Create solr service
3. Load some data into solr
4. Wait a while
5. Try to reindex the rest of the data

Expected behavior

Reliable Sub-200ms Response times
Resilient Operations

Actual behavior

• Varaible (up to 2000ms) Response times

• Shard lost. If the leader shard is lost before another replica can take over, it causes ckan to completely fail.
  

• Search-Reindex fails... (reason unknown, suspect: Zookeeper lost)
  
  
  
  

Sketch

[TO DO]

References

• Here's a short review of someone optimizing solr performance in their own system. A few of the concepts are helpful.

[nickumia-reisys]

I think we've proven that solr is not resilient across changes. If we define a specification and let it run, it's stable. But if we update-service, solr has a high probability of breaking..

More References for the future:

• Solr Performance Data, https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceData
• https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceFactors
• https://stackoverflow.com/questions/55081729/how-many-shards-are-proper-for-each-collections-in-solr
• https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems

[jbrown-xentity]

After discussion, we want to be able to raise the RAM limit that is hardcoded in the code.

• Update solr brokerpak to allow for variable node size (RAM)
• Merge and tag the release
• Utilize this tag in the ssb broker to create more nodes and specify RAM (~32G)
  Focus on prod, but do in parallel on staging.

[nickumia-reisys]

datagov-brokerpak-solr change:

• Expose solrMem and solrCpu GSA-TTS/datagov-brokerpak-solr#30

datagov-ssb change:

• Update solrcloud brokerpak datagov-ssb#113

catalog.data.gov change:

• Load test improvements catalog.data.gov#405
  • CKAN Collection parameters
  • Solr Service parameters

[nickumia-reisys]

This is more of a historical issue now. It is specifically related to Solrcloud which we have abandoned in favor of the stability of Solr on ECS. Investigation into this would be covered by the following ticket. Although the specific investigation of SolrCloud would need to be a deliberate effort,

• Dissect Solr Performance through New Relic #3956

As more context for the future, per @jbrown-xentity,

Due to load/compatability issues, we do not plan to use solr cloud without significant research and development.