ElasticSearch7 create unlimited amount of indices

[astelmashenko]

Describe the bug
After certain amount of time shards problem raised. We checked ES and found out around 2300 shards, we names like conductor_task_log_20221004.

Details
Conductor version: 3.12.2
Persistence implementation: Postgres
Queue implementation: Postgres
Lock: Redis

To Reproduce
Steps to reproduce the behavior:

1. Just use elastic for a while

Expected behavior
Data is cleaned up periodically.

[astelmashenko]

Investigating code I see

    private void createIndexesTemplates() {
        try {
            initIndexesTemplates();
            updateIndexesNames();
            Executors.newScheduledThreadPool(1).scheduleAtFixedRate(this::updateIndexesNames, 0, 1, TimeUnit.HOURS);
        } catch (Exception e) {
            logger.error("Error creating index templates!", e);
        }
    }

See this line
Executors.newScheduledThreadPool(1).scheduleAtFixedRate(this::updateIndexesNames, 0, 1, TimeUnit.HOURS);

Where updateIndexesNames creates new index every week.
Can someone explain why does it change names for indices?

I'd like to change this behavior, because we reaching limits on shards.

[astelmashenko]

I also checked below methods are not used anywhere:

IndexDAO.getEventExecutions
IndexDAO.getMessages

so it is probably safe to stop indexing them:

conductor.app.eventMessageIndexingEnabled=false
conductor.app.eventExecutionIndexingEnabled=false

However to stop creating new indices we need to change this config:
conductor.elasticsearch.autoIndexManagementEnabled=false
And provision indices for conductor manually.

Can anybody comment on this?

[manan164]

Hi @astelmashenko , Thanks for reporting this. We will add support for archival from ES also.
Let me know if this works, or we can chat here for more realtime collaboration.

[astelmashenko]

@manan164 ,

We will add support for archival from ES

Yes, it will help.
Additionally, I'd add reaction to indexing flag, e.g. if conductor.app.eventMessageIndexingEnabled=false is set then do not create new indices once a week. Maybe think also about changing period of creating new indices, e.g. WEEK, MONTH, YEAR
Thanks

[astelmashenko]

@manan164 , are there any work done on archival of indicies?