SPIKE:Set default index shards to 1

[roborourke]

The default number of shards in ElasticPress matches the old Elasticsearch default of 5 shards. It's rare our shards reach the kind of size that they really need to be split, and they can still have at least one replica for resilience.

Fewer shards generally improves search time, but lowers data ingestion speed as bulk indexing requests cannot be parallelised.

This seems like an appropriate trade off for site search.

Acceptance criteria:
Answer this questions before going further with the original acceptance criteria

• What happens to existing clusters
• Ho can we make sure this will work correctly with 3+ nodes
• How to measure and should we the performance

Original acceptance criteria

• Default shard count is 1
• Default shard replica count is 1
• Default shard count can be configured
• Shard count config option is documented with guidance on changing it
• Shard count is a number divisible by the number of ES data nodes
• Simulate with 3+ nodes

[roborourke]

Needs some infra work to expose number of nodes ideally. Should be a constant called ES_NODE_COUNT

[veselala]

Hey team! Please add your planning poker estimate with Zenhub @ferschubert-hm @jerico @kovshenin @mikelittle @sivanovhm @wisyhambolu

[veselala]

@mikelittle we converted this issue to a spike and estimated it to 5 SP.

[roborourke]

What happens to existing clusters

The changes outlined here would only take effect the next time content is reindexed. If indexing is failing due to reaching the shard limit then it should hopefully solve or improve that on reindex, as a reindex is needed anyway.

How can we make sure this will work correctly with 3+ nodes

May need a dedicated primary node but do we have any instances with more than 2 nodes? Will we? I forgot I'd noted this: "Shard count is a number divisible by the number of ES data nodes" - I had meant to also suggest adding an environment variable for the number of nodes in a stack. Or it could be derived and stored by a background task or request run prior to reindexing to set the default and round the target value to the nearest divisible number.

GET _cat/nodes will list available nodes.

How to measure and should we the performance

The main trade-off will be ingestion versus read speed. More shards allows for parallel ingestion, fewer shards allows for faster searching, so I would propose the following approach to testing:

• using a substantial dataset e.g. 100k posts over 100 subsites
• benchmark indexing speed with 1, 2 and current default of 5 shards
• benchmark 1000 search requests with 1, 2 and current default of 5 shards

The approach can be run and worked out locally but will need to be run on a multi-node test stack in the cloud too.

Parallel ingestion may not be that beneficial as we hit memory usage as a limiting factor sometimes, and apart from reindexing ingestion doesn't need to be quick. There are other ways to manage ingestion memory issues too, by adding a short sleep or wait in between requests. Will be interesting to see what the results are.