arrow files

[ebruevcen]

A technical question:

How do we convert .csv files (or a wrangled R data frame) into .arrow files?

[yjunechoe]

Piggy-backing on this question, is it possible to transfer Arrow table objects directly between R/Julia sessions via RCall (since R also has the arrow package)?

For example, what I'd hoped would work:

using RCall
using SMLP2023
R"library(arrow)"
R"$(SMLP2023.dataset(:kb07))"
ERROR: MethodError: no method matching sexpclass(::Arrow.Table)

[palday]

I'm not sure why you would do this, but the way you tried didn't work because R reads Arrow directly into a dataframe. To make your example work, you need to convert things to a dataframe:

using RCall
using SMLP2023
R"$(DataFrame(SMLP2023.dataset(:kb07)))"

[palday]

An example conversion from CSV to Arrow using Julia would look like:

using Arrow
using CSV
using DataFrames

csv_path = "some_file.csv"

# read in the data and convert it to a dataframe
df = CSV.read(csv_path, DataFrame)

# do stuff ....

# write out the dataframe as an Arrow table
arrow_path = "some_file.arrow"
Arrow.write(arrow_path, df)

[palday]

As @dmbates mentioned, it can also be worthwhile to use the compress keyword argument to specify compression: https://arrow.apache.org/julia/dev/reference/#Arrow.write