Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source versioning: Postgres, MySQL and Load generator #647

Merged
merged 45 commits into from
Jan 3, 2025
Merged
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
e008303
Source versioning initial implementation
bobbyiliev Sep 3, 2024
9bb3a55
Use source table instead of table from source
bobbyiliev Sep 3, 2024
467b05e
MySQL source: separate for tables and all tables
bobbyiliev Sep 3, 2024
2fcb9a1
Loadgen source: add all tables bool attr
bobbyiliev Sep 3, 2024
95e7eeb
Add tests
bobbyiliev Sep 4, 2024
876caf9
Add more tests for mysql and loadgen
bobbyiliev Sep 4, 2024
3acaa24
Add ignore columns for MySQL
bobbyiliev Sep 4, 2024
9e512e1
Add source_id logic
bobbyiliev Sep 9, 2024
37b51a7
Add source table migration guide
bobbyiliev Sep 9, 2024
bc0db2d
Add deprecated message
bobbyiliev Sep 9, 2024
9b97e58
Ignore text columns for load gen source tables
bobbyiliev Sep 11, 2024
d9626c0
Refactor source table for individual sources
bobbyiliev Sep 13, 2024
1b6ccc2
Add datasource
bobbyiliev Sep 13, 2024
a8a21e4
Format examples
bobbyiliev Sep 13, 2024
18900b2
Add Kafka source table resource
bobbyiliev Sep 16, 2024
fe382fa
Review updates
bobbyiliev Sep 16, 2024
3026124
Update guide migration guide
bobbyiliev Sep 16, 2024
ca3c823
Update guide migration guide
bobbyiliev Sep 16, 2024
f81f05b
Add import examples for Kafka source tables
bobbyiliev Sep 17, 2024
5f43535
Add upstream mysql and postgres table names
bobbyiliev Sep 20, 2024
d4b31b6
Fix unit tests
bobbyiliev Sep 20, 2024
47bb160
Add Kafka upstream references
bobbyiliev Sep 20, 2024
3c39694
Add integration tests
bobbyiliev Sep 24, 2024
76e22c2
Fix failing test
bobbyiliev Sep 24, 2024
14995ed
Extend data source to include upstream names
bobbyiliev Sep 24, 2024
acf1190
Small updates
bobbyiliev Sep 27, 2024
82c0304
Switch back to latest image
bobbyiliev Sep 30, 2024
c87e984
FromAsCasing: 'as' and 'FROM' keywords' casing do not match
bobbyiliev Oct 6, 2024
48a6d31
Add source reference data source
bobbyiliev Oct 6, 2024
7d05413
Add source reference data source example
bobbyiliev Oct 6, 2024
c94efd0
First round of the initial PR change requests
bobbyiliev Oct 14, 2024
811d160
Fix failing tests
bobbyiliev Oct 14, 2024
0f7a921
Fix failing tests
bobbyiliev Oct 14, 2024
009b3c2
Second round of the initial PR change requests
bobbyiliev Oct 15, 2024
ea4f8dd
Add unit tests to data source source reference
bobbyiliev Oct 15, 2024
41eb593
PR change requests
bobbyiliev Oct 21, 2024
9ba091e
Fix failing tests
bobbyiliev Oct 21, 2024
3b62b79
Fix failing tests
bobbyiliev Oct 21, 2024
30b0db5
Add alter source refresh to data source
bobbyiliev Oct 28, 2024
e9573b3
Add new kafk acolumns from mz_kafka_source_tables
bobbyiliev Oct 28, 2024
74410c0
Fix failing MockSourceTableKafkaScan test
bobbyiliev Oct 28, 2024
6b3af39
Remove confusing line from migration guide
bobbyiliev Oct 29, 2024
001988a
Remove a left behind comment
bobbyiliev Nov 18, 2024
285933f
explicitly enable create table from source as --all-features is not w…
bobbyiliev Nov 18, 2024
ef60874
Generate docs
bobbyiliev Jan 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix failing tests
bobbyiliev committed Jan 3, 2025
commit 811d160471a88c9f7d05ba2ad55f82dafef293b1
2 changes: 1 addition & 1 deletion pkg/materialize/source_table_kafka.go
Original file line number Diff line number Diff line change
@@ -367,7 +367,7 @@ func (b *SourceTableKafkaBuilder) Create() error {
if inlineOptions.Enabled {
upsertOption += " (VALUE DECODING ERRORS = (INLINE"
if inlineOptions.Alias != "" {
upsertOption += fmt.Sprintf(" AS %s", inlineOptions.Alias)
upsertOption += fmt.Sprintf(" AS %s", QuoteIdentifier(inlineOptions.Alias))
}
upsertOption += "))"
}
2 changes: 1 addition & 1 deletion pkg/materialize/source_table_kafka_test.go
Original file line number Diff line number Diff line change
@@ -81,7 +81,7 @@ func TestResourceSourceTableKafkaCreateWithUpsertOptions(t *testing.T) {
\(REFERENCE "topic"\)
FORMAT JSON
INCLUDE KEY, HEADERS, PARTITION, OFFSET, TIMESTAMP
ENVELOPE UPSERT \(VALUE DECODING ERRORS = \(INLINE AS my_error_col\)\)
ENVELOPE UPSERT \(VALUE DECODING ERRORS = \(INLINE AS "my_error_col"\)\)
EXPOSE PROGRESS AS "database"."schema"."progress";`,
).WillReturnResult(sqlmock.NewResult(1, 1))

2 changes: 1 addition & 1 deletion pkg/resources/resource_source_table_kafka_test.go
Original file line number Diff line number Diff line change
@@ -264,7 +264,7 @@ func TestResourceSourceTableKafkaCreateIncludeFalseWithAlias(t *testing.T) {
FROM SOURCE "materialize"."public"."kafka_source"
\(REFERENCE "topic"\)
FORMAT JSON
ENVELOPE UPSERT \(VALUE DECODING ERRORS = \(INLINE AS decoding_error\)\);`,
ENVELOPE UPSERT \(VALUE DECODING ERRORS = \(INLINE AS "decoding_error"\)\);`,
).WillReturnResult(sqlmock.NewResult(1, 1))

// Query Id