Spring Boot, Kafka, Zookeeper, Kafdrop, Spring Data/JPA, MySQL, Rest API

Sample kafka-project

• Single topic per individual platform.
  • Kafka as a traditional message broker.
• Single topic to all platforms.
  • Different clientType value attribute. To be processed separately using recordFilterStrategy.

Use Cases(Address creation)

• Improve the speed of synchronizing customer information in a typical Insurance domain.
• In the Insurance domain, multiple platforms are able to update customer information, like addresses:
  • Website application
  • Salesforce
  • Life400 - mainframe
• Use Kafka to receive and process all ADDRESS creation from multiple platforms and process to the downstream services.
• Introduced address_transaction table which will be referred by the consumers.

Use Case #1

• Kafka as traditional message broker.
  • One topic per platform
    • ADDRESS_CREATE_SALESFORCE
    • ADDRESS_CREATE_WEBSITE
    • ADDRESS_CREATE_LIFE400
  • One producer endpoint per platform/topic.
  • One consumer listener per platform/topic.

Sample High-level architecture diagram

Use Case #2

• Kafka single topic with consumer using recordFilterStrategy.
  • One topic for all platforms with different clientType.
    • ADDRESS_CREATE_ALL_PLATFORM
  • One producer endpoint to all platforms.
  • One consumer listener per platform clientType by using recordFilterStrategy.
    • containerFactory = "allPlatformSalesforceFactory"
    • containerFactory = "allPlatformWebsiteFactory"
    • containerFactory = "allPlatformLife400Factory"

Sample High-level architecture diagram

• 

Prerequisites on running this project:

• Install Java 11(Needed by Kafkdrop - kafka message viewer).
• Install Docker Desktop.
• JMeter - for testing.

Tech stack and services in kafka-project.

• kafka
• zookeeper
• Kafkdrop - kafka message viewer
• kafka-producer service
• kafka-consumer service
• address-service - services to process consume messages
• MySql - to save all transactions

Steps in running kafka-project.

• Checkout the project in Github.
• Go the project directory.

cd /<path>/kafka-project

• Install kafka-project services.

mvn clean install

• Start kafka-project services.

docker-compose up

• View Kafdrop on the browser using the below URL.

http://localhost:9000

Testing kafka-project locally.

• Copy the JMeter script to JMeter bin directory.

JMeter script - kafka-project/_external_files/kafka-producer-v1.jmx
JMeter bin - <JMeter path>/bin
cp <JMeter script> <JMeter bin>

• Open JMeter interface. No need to run in non-GUI mode.

cd <JMeter bin>
./jmeter.sh

• Execute JMeter script.
• 

View Kafka messages using Kafdrop.

• 
• 
• 
• 

View the address records.

• Setup MySQL client.
• 
• Verify the data.
•