storage: malformed key terminator byte (columnar blocks)

[jbowens]

On 8666fa6
https://mesolite.cluster.engflow.com/invocations/default/e399298e-413b-4a3d-be32-9a493fb4ef8f

=== RUN   TestTenantLogic_udf_privileges/show_grants
panic: malformed key terminator byte: 8988
goroutine 231115 [running]:
github.com/cockroachdb/cockroach/pkg/storage.checkEngineKey({0xc00b6499b0?, 0x0?, 0x0?})
	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:463 +0xb9
github.com/cockroachdb/cockroach/pkg/storage.EngineKeySplit({0xc00b6499b0, 0x2, 0x0?})
	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:481 +0x30
github.com/cockroachdb/cockroach/pkg/storage.(*cockroachKeyWriter).ComparePrev(0xc005630960, {0xc00b6499b0, 0x2, 0x8})
	github.com/cockroachdb/cockroach/pkg/storage/pebble_key_schema.go:75 +0x37
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).evaluatePoint(0xc00ac98c08, {{0xc00b6499b0?, 0x47007b?, 0x0?}, 0xc0000f3008?}, 0x3e)
	github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:471 +0x5d
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).AddWithForceObsolete(0xc00ac98c08, {{0xc00b6499b0?, 0x2?, 0x8?}, 0x35?}, {0xc00500f5c0, 0x3e, 0x40}, 0x0)
	github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:345 +0x11b
github.com/cockroachdb/pebble/internal/compact.(*Runner).writeKeysToTable(0xc00f1612d8, {0x88a4da0, 0xc00ac98c08})
	github.com/cockroachdb/pebble/internal/compact/external/com_github_cockroachdb_pebble/internal/compact/run.go:203 +0x72c
github.com/cockroachdb/pebble/internal/compact.(*Runner).WriteTable(0xc00f1612d8, {0x4, 0x2, {0x0, 0x0, {0x0, 0x0}, 0x0, {0x0, 0x0}, ...}}, ...)
	github.com/cockroachdb/pebble/internal/compact/external/com_github_cockroachdb_pebble/internal/compact/run.go:143 +0x1ca
github.com/cockroachdb/pebble.(*DB).compactAndWrite(0xc00a1d9b08, 0x3, 0xc00667e608, {0x0, 0x0, 0x0}, 0x7)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2913 +0x706
github.com/cockroachdb/pebble.(*DB).runCompaction(0xc00a1d9b08, 0x3, 0xc00667e608)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2808 +0x5cd
github.com/cockroachdb/pebble.(*DB).flush1(0xc00a1d9b08)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1448 +0x3b4
github.com/cockroachdb/pebble.(*DB).flush.func1({0x8830478?, 0xc0142577d0?})
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1183 +0xee
runtime/pprof.Do({0x8830318?, 0xd1f5b80?}, {{0xc000941ce0?, 0xc005fd8f70?, 0xc01aa41200?}}, 0xc005fd8fa8)
	GOROOT/src/runtime/pprof/runtime.go:51 +0x9d
github.com/cockroachdb/pebble.(*DB).flush(0xc005fd8fb8?)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1176 +0x52
created by github.com/cockroachdb/pebble.(*DB).maybeScheduleFlush in goroutine 218298
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1092 +0xdb
goroutine 1 [chan receive]:
testing.(*T).Run(0xc00511d1e0, {0x6fedbcf?, 0x0?}, 0x73e0d68)
	GOROOT/src/testing/testing.go:1750 +0x3ab
testing.runTests.func1(0xc00511d1e0)
	GOROOT/src/testing/testing.go:2161 +0x37
testing.tRunner(0xc00511d1e0, 0xc003d4faa8)
	GOROOT/src/testing/testing.go:1689 +0xfb
testing.runTests(0xc005139f68, {0xc001f0e540, 0x8, 0x8}, {0xcf50c40?, 0x0?, 0xcf50c40?})
	GOROOT/src/testing/testing.go:2159 +0x445
testing.(*M).Run(0xc001d47720)
	GOROOT/src/testing/testing.go:2027 +0x68b
pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test.TestMain(0xc001d47720)
	pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test/pkg/ccl/logictestccl/tests/3node-tenant/generated_test.go:77 +0x1bb

Jira issue: CRDB-43448

[blathers-crl]

Hi @jbowens, please add branch-* labels to identify which branch(es) this C-bug affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[jbowens]

Hrm this \x89\x88 key is being written during a flush, so we're reading from the memtable. Either we already mangled it in a roundtrip to a sstable long before it was committed to this Engine, or we're mangling something in-memory in the columnar writer as we're building the block.

[jbowens]

cc @RaduBerinde

[RaduBerinde]

If we had written it to an sstable, we would have crashed here:

cockroach/pkg/storage/pebble_key_schema.go

Line 111 in 8666fa6

if (len(key)-versionLen) != int(keyPrefixLen) || key[keyPrefixLen-1] != 0x00 {