feat(dataobj): cardinality estimation

[owen-d]

• Introduces value cardinality estimation, initially for metadata fields
• Backed by hyperloglog, initially using sparse representations for space conservation
• Only writes a uint64 cardinality count estimation on flush
• Refactors statistics writing to centralize logic, using a new ColumnStatsBuilder

[rfratto]

LGTM! Just a few tiny comments

[rfratto]

This is only used internally by the dataset.ColumnBuilder, so we don't need to export anything here as part of the API of dataset :)

(I like limiting APIs whenever possible because I find it makes it easier to navigate code completion and godoc pages)

[rfratto]

Would it be simpler to do this inside ColumnBuilder.Append instead? That already has a check to see if the append was successful

[owen-d]

That is what's happening. It monitors the success of the page builder to see if it should append stats, primarily because the page builder returns false when full. In turn, this check ensures we only add the stats once, rather than twice when a new page is cut.

It makes sense to add this as a code comment -- I'll update it.

[rfratto]

Sorry, I'm still not following. I was asking if this would be simpler:

// Append adds a new value into cb with the given zero-indexed row number. If
// the row number is higher than the current number of rows in cb, null values
// are added up to the new row.
//
// Append returns an error if the row number is out-of-order.
func (cb *ColumnBuilder) Append(row int, value Value) error {
	if row < cb.rows {
		return fmt.Errorf("row %d is older than current row %d", row, cb.rows)
	}

	// We give two attempts to append the data to the buffer; if the buffer is
	// full, we cut a page and then append to the newly reset buffer.
	//
	// The second iteration should never fail, as the buffer will always be empty
	// then.
	for range 2 {
		if cb.append(row, value) {
			cb.rows = row + 1
                        cb.statsBuilder.Append(value) 
			return nil
		}

		cb.flushPage()
	}

	panic("ColumnBuilder.Append: failed to append value to fresh buffer")
}

Since there's already a condition for success there, it would avoid a second check for success on the append

[owen-d]

Ah yeah I see 👍

[rfratto]

🚀