Fix export to JSON when dataset larger than batch size

[albertvillanova]

Fix export to JSON (lines=False) when dataset larger than batch size.

Fix #7037.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[albertvillanova]

The test before confirms the bug.

There are different possible solutions to this issue:

• the easiest would be to write multiple JSON files, one for each batch; this solution can be done in parallel if num_proc is passed
• alternatively, we could tweak the writing and remove the extra [ and ] characters; this solution will only be valid if orient="records"
• others?

[varadhbhatnagar]

@albertvillanova I was planning to approach it in the second way for orient="records" , orient="values" and orient="index". For orient="split", the columns and index can be written in one go and the data can be written in streaming manner. For orient="columns", each column can be written in a streaming way. LMK if I should go ahead with this.

The test before confirms the bug.

There are different possible solutions to this issue:

* the easiest would be to write multiple JSON files, one for each batch; this solution can be done in parallel if `num_proc` is passed

* alternatively, we could tweak the writing and remove the extra `[` and `]` characters; this solution will only be valid if `orient="records"`

* others?