Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wip][feat][io] Debezium DB2 source connector for Pulsar #19821

Open
wants to merge 45 commits into
base: master
Choose a base branch
from
+845 −15
Open

[wip][feat][io] Debezium DB2 source connector for Pulsar #19821

Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
061b4df
Added Debezium DB2 connector for Pulsar
devinbost Mar 15, 2023
9972036
Initial commit for DB2 Debezium connector testing
devinbost Mar 17, 2023
4ea5216
Updated db2 source config
devinbost Mar 17, 2023
f38a5c9
Removed CI file
devinbost Mar 17, 2023
5f281bb
Updating docs pom
devinbost Mar 22, 2023
d3c5bbd
Added missing hooks to execute DB2 tests and updated some test code.
devinbost Mar 24, 2023
0fa7509
Updated source config for doc purposes
devinbost Mar 28, 2023
92dc034
Update pulsar-io/debezium/db2/src/main/java/org/apache/pulsar/io/debe…
devinbost Mar 29, 2023
3e97cc1
Separated db2 connection with subsequent commands
devinbost Mar 30, 2023
817d7dc
Added more startup commands
devinbost Mar 30, 2023
58a4ca0
Updated image for Debezium test container to use the new DB2 image
devinbost Mar 30, 2023
c482f3b
Fixed image reference
devinbost Mar 31, 2023
e6b5599
Fix license header
devinbost Mar 31, 2023
262d7c6
Fixed format of license header
devinbost Mar 31, 2023
d29db07
Fixed license format by running mvn initialize license:format
devinbost Mar 31, 2023
e493cf8
Added more license details and updated CI to build docker image for DB2
devinbost Mar 31, 2023
fa9188e
Added DB2 image to Pulsar IO step in CI
devinbost Mar 31, 2023
4f05272
Added check to ensure that commands aren't run on DB2 until DB finish…
devinbost Mar 31, 2023
b04c23e
Added wait checks to ensure DB2 has initialized before connecting
devinbost Mar 31, 2023
5ab2163
Added more logging to troubleshoot container startup
devinbost Mar 31, 2023
c6812ef
Added more logging to troubleshoot container startup if there are errors
devinbost Mar 31, 2023
77557d1
Add more checks since something is timing out in the test
devinbost Mar 31, 2023
dd7515e
Increasing timeout since DB2 takes a while to start
devinbost Mar 31, 2023
bf0d3e1
Increasing timeout since DB2 takes a while to start.
devinbost Mar 31, 2023
414738c
Increasing wait time between subsequent commands
devinbost Mar 31, 2023
a9673de
Catching exception and printing more debug info
devinbost Mar 31, 2023
49c342f
Refactored DB2 testing to use JDBC driver
devinbost Apr 3, 2023
4cdd64b
Added JDBC logging for DB2
devinbost Apr 4, 2023
5ffe1e4
Added test wait strategy based on testcontainers DB2 container
devinbost Apr 4, 2023
ab96ad1
fixed typo
devinbost Apr 4, 2023
189ef39
Fixed issue with DB2 calls in test
devinbost Apr 4, 2023
e400261
Removed CDC logic from test
devinbost Apr 4, 2023
2b2de0d
Added step to setup CDC
devinbost Apr 4, 2023
970fd53
Refactored cdcsetup.sh execution to be synchronous to prevent race on DB
devinbost Apr 4, 2023
f7bcb0d
Fixed style checks
devinbost Apr 4, 2023
992aa52
Fixed case sensitive values
devinbost Apr 4, 2023
1544db3
Fixed case
devinbost Apr 4, 2023
4ce089a
Added more properties to DB2 source to try to fix connector restarts
devinbost Apr 5, 2023
faf2eab
Removing additional insert to avoid duplicate write from failing test
devinbost Apr 5, 2023
16170a8
Incremented number of expected messages.
devinbost Apr 5, 2023
972c349
Added db2 bind command
devinbost Apr 5, 2023
b437fd5
Fixed bind command
devinbost Apr 5, 2023
2dbd84a
Added more logging
devinbost Apr 5, 2023
ce6e395
Added retention
devinbost Apr 14, 2023
45f2fde
Fixed retention limit
devinbost Apr 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Removed CDC logic from test
@devinbost
devinbost committed Apr 4, 2023
commit e4002619fd16dd0835c07b344181a104ebb3b06c
8 changes: 4 additions & 4 deletions ...va/org/apache/pulsar/tests/integration/io/sources/debezium/DebeziumDB2DbSourceTester.java
View file
Original file line number Diff line number Diff line change
@@ -77,12 +77,12 @@ public void setServiceContainer(DebeziumDB2DbContainer container) {
public void prepareSource() {
log.info("Starting DB2");

debeziumDB2DbContainer.getPreparedStatement(debeziumDB2DbContainer.enableCdcStatement())
.execute();
//debeziumDB2DbContainer.getPreparedStatement(debeziumDB2DbContainer.enableCdcStatement())
// .execute();
debeziumDB2DbContainer.getPreparedStatement(debeziumDB2DbContainer.createTableStatement())
.execute();
debeziumDB2DbContainer.getPreparedStatement(debeziumDB2DbContainer.addCdcTableStatement())
.execute();
//debeziumDB2DbContainer.getPreparedStatement(debeziumDB2DbContainer.addCdcTableStatement())
// .execute();
debeziumDB2DbContainer.getPreparedStatement(debeziumDB2DbContainer.insertStatement())
.execute();
var result = debeziumDB2DbContainer.getStoreNameResult();