Group semantics regression on FLORES 200

[mkuchnik]

FLORES 200 was working on 0c66893 but it is different on main (487ec5c).

Repro from recipes directory:

import mlcroissant as mlc
dataset = mlc.Dataset(file="../../../datasets/flores-200/metadata.json", debug=False)
print(dataset)
records = dataset.records(record_set="language_translations_train_data_with_metadata")
print(sum(1 for r in records))

Before the test set was added (causing the record_set to be renamed), the call would be:

import mlcroissant as mlc
dataset = mlc.Dataset(file="../../../datasets/flores-200/metadata.json", debug=False)
print(dataset)
records = dataset.records(record_set="language_translations_with_metadata")
print(sum(1 for r in records))

Note: this can block for a long time in the correct case, so it may be better to print iteratively to see if count goes over 997.

The length should be 204 * 997, but it is only 997. This is because the data from each of the 204 .dev files have been collapsed at some point. Instead of each of the files yielding 997 rows, the effect is 997 samples and some of the columns are None.

[marcenacp]

@mkuchnik Thanks for opening the bug. I expected https://github.com/mlcommons/croissant/blob/main/datasets/recipes/output/translations_from_zip.jsonl to cover this test case (extract the filename for each line), but apparently there is a difference. I can come up with a better implementation for tomorrow.

[marcenacp]

@mkuchnik Do you confirm that PR #309 fixes this issue?

translation and language used to be computed separately, so it was hard to reconstruct the matching. Hence the na values. Now, all fields are computed together within the same pd.DataFrame. The operation is called ReadFields. It simplifies the code (1 operation instead of 3 operations) and it should fix the bug.

[mkuchnik]

@marcenacp Looks good to me! Seems a bit faster, too.