Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
* [#53] Update replication.factor inplace * Update docker, ssl, and tests Since we'll be updating a topic's replication.factor in-place, we need a multi-broker cluster to test against. This is a pretty big diff, but the changes are essentially: * create new secrets to support multiple brokers * update the docker/docker-compose files to support a multi-broker cluster. This cluster can still be run locally with docker-compose up, and CI systems can use the docker-compose.test.yaml to execute a well-defined environment * update the tests to play nicely with the new secrets and docker setup (brokers aren't hard-coded to localhost anymore) * when a topic creation is applied, the resource will be considered 'Pending' until its existence is confirmed via client.ReadTopic. This was less of an issue with one broker, but once we introduced others, the acc tests would sometimes fail because a topic would get created but subsequent steps that ran immediately afterwards wouldn't see it * Update RF before adding partitions If a plan both: * increases the partitions * changes the replication.factor Then there's no point in creating new partitions with the old replication.factor, just to go through and change it immediately afterwards. * Add IsReplicationFactorUpdating method to Client Before we introduced the in-place replication.factor changes, the replication.factor of a topic was updated atomically across all of its partitions: * when the topic was first created, all of its initial partitions had the same rf * when partitions were added to a topic, they would be replicated according to the topic's rf * when a topic's rf was changed, it would be deleted and created again (bullet point 1) Now that a topic's rf may be updated in-place, the rf is not updated atomically; it's instead updated in a rolling/eventually-consistent manner. For sometime after the apply, some partitions will have the new rf, and some will still have the old rf. Eventually, however, they'll all have the new rf. This commit introduces a new method to determine whether or not an rf update has finished propagating through all of a topic's partitions, and changes the kafka_topic resource to be aware of this propagating/updating state. * Add acc tests for in-place rf update Overall flow: * produce a well-known set of messages to a new topic * change that topic's rf and partition count * verify the changes went through and that no messages were lost * Wait for rf update before adding partitions We can't add partitions while a replication.factor update is ongoing, so this commit waits for an rf update to finish before adding any new partitions. Because we're now waiting for the rf to update separately, the IsReplicationFactorUpdating logic has been removed from topicRefreshFunc (which only waits for partition and config updates to finish). * Add CanAlterReplicationFactor method This new method determines if the cluster supports altering a topic's replication.factor in-place. If so, then the provider will go ahead and make alterations in-place; otherwise, it will replace the topic like it did before. populateAPIVersions was changed to ensure that the max api versions are supported by every broker in the cluster. * Send apiVersions request asynchronously to brokers
- Loading branch information
