CSV problems with locale different than western

[arturbac]

I just wanted to tell You that I found problematic to use csv in glaze.
In my country locale (and in Germany and France too ) decimal prefix is comma.
So You will in general encounter only semicolon separated CSV.
I taught it would be nice to have Opts variable for specifying the separator to use instead of hardcoded ','
And second thing csv's usually dont have column names, it would be nice to be able to specify them before parsing.
Currently I have to load file fix it so replace ; with , add columns to be able to use csv parsing in glaze

[stephenberry]

Thanks for pointing this out. A compile time option to set the separator to ; makes sense. And, reading a CSV without column names should definitely be supported. Will add an issue from this.

[stephenberry]

That makes sense, I think the default for CSV should just accept commas (that's what the C means), but I do think having an option that supports both at runtime would be good.

[stephenberry]

So, just looking at the frequency of semi-colons versus commas doesn't work. Because if we have three decimal numbers in a row, we would expect three commas and only two semi-colons when the delimiter is semi-colons and the locale is European.

The issue becomes more complex with the fact that numbers could be decimal or not, because we write them out as efficiently as possible, so if they are zeros then we'd have no commas.

There is no straightforward way to determine the format in a fully robust manner, and the more advanced methods we use the more runtime cost we incur.

I also think it is best to write CSV numbers in compliant JSON format, with the . for decimals. This allows us to make the two formats more compatible, it reduces binary in code by not needing to have different number serialization/deserialization functions, and it also makes it follow the standard C++ library more, with to_chars/from_chars being locale independent.

CSV that delimits JSON is extremely useful, and this becomes invalid if using European decimal format. I don't really want a lesser subset of features in Glaze based on the delimiter used. It becomes confusing to users, and adds complexity to the code base. I don't like the idea of various CSV standards in Glaze not being able to round-trip with each other.

I think it is best to keep CSV locale independent and JSON compliant in Glaze and users can transform their CSV documents beforehand if they want to load them with Glaze.

[arturbac]

Ok so for working with external csv I will have to switch other library as I use csv I didn't produced and they were not produced with glaze.

[arturbac]

'There is no straightforward way to determine the format in a fully robust manner'
If You can count column labels, and for that when You can assume column names must contain alphanumeric and can't contain comma and semicolon in name, You can determine separator that way.

[stephenberry]

We want to support CSV files without column names, so we'd need a way to determine the delimiter in these cases. And, like you pointed out, even when we have column names we need to make a number of assumptions. I'm trying to move towards more rigid standards because these decisions on flexibility affect performance, code size, test size, test compilation times, documentation bloat, user comprehension etc.

Pardon the inconvenience, but I would use another library for CSV files that use semi-colons and locale specific numbers. Handling multiple delimiters isn't the main issue, the bigger issue is needing to bifurcate the number parsing/serializing algorithms. That's something that could be done in the future, but I don't want to jump into supporting locale dependent numbers with Glaze right now.

It might seem like an annoyance to need to use another library, but Glaze is particularly useful because it uses formats that synthesize well with JSON, making interfaces work well with each other.

I'm just explaining my philosophy for Glaze, not that your ideas are invalid, just that they aren't within the current scope of the project.