encoding default "null" failed in avro protocol

[wk989898]

What did you do?

1. run a changefeed with Avro protocol
2. run sql

create table t31(
    id int primary key,
    col varchar(255) DEFAULT "null"
);

insert into t31(id) values(2);

What did you expect to see?

No response

What did you see instead?

[ERROR] [schema_registry.go:335] ["GetCachedOrRegister: Could not make goavro codec"] [error="Record \"default.test.t31\" field \"col\": default value ought to encode using field schema: cannot encode binary bytes: expected: string; received: <nil>"]
[ERROR] [avro.go:102] ["AppendRowChangedEvent: avro encoding failed"] [error="[CDC:ErrAvroSchemaAPIError]schema manager API error: Record \"default.test.t31\" field \"col\": default value ought to encode using field schema: cannot encode binary bytes: expected: string; received: <nil>"] [errorVerbose="[CDC:ErrAvroSchemaAPIError]schema manager API error: Record \"default.test.t31\" field \"col\": default value ought to encode using field schema: cannot encode binary bytes: expected: string; received: <nil>\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/[email protected]/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/pkg/sink/codec/avro.(*schemaManager).GetCachedOrRegister\n\tgithub.com/pingcap/tiflow/pkg/sink/codec/avro/schema_registry.go:336\ngithub.com/pingcap/tiflow/pkg/sink/codec/avro.(*BatchEncoder).avroEncode\n\tgithub.com/pingcap/tiflow/pkg/sink/codec/avro/avro.go:227\ngithub.com/pingcap/tiflow/pkg/sink/codec/avro.(*BatchEncoder).AppendRowChangedEvent\n\tgithub.com/pingcap/tiflow/pkg/sink/codec/avro/avro.go:100\ngithub.com/pingcap/tiflow/pkg/sink/codec.(*encoderGroup).runEncoder\n\tgithub.com/pingcap/tiflow/pkg/sink/codec/encoder_group.go:116\ngithub.com/pingcap/tiflow/pkg/sink/codec.(*encoderGroup).Run.func2\n\tgithub.com/pingcap/tiflow/pkg/sink/codec/encoder_group.go:95\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1598"]

Versions of the cluster

TiCDC version (execute cdc version):

v7.1.3

[wk989898]

The root reason is goavro treats string literal "null" as null, but TiCDC doesn't .

tiflow/pkg/sink/codec/avro/avro.go

Lines 451 to 457 in 725d938

	// https://stackoverflow.com/questions/22938124/avro-field-default-values
	if defaultValue == nil {
	field["type"] = []interface{}{"null", avroType}
	} else {
	field["type"] = []interface{}{avroType, "null"}
	}
	field["default"] = defaultValue

goavro code:
https://github.com/linkedin/goavro/blob/5ec5a5ee7ec82e16e6e2b438d610e1cab2588393/record.go#L109-L114