Add dataset loading capabilities

[davidmrdavid]

From what I gather, there's no go-to utility to load a matrix / dataset from disk without resorting to the fromJSON function of DenseMatrix and SparseMatrix. It would be great if we had a clean dataset loading interface supporting .csvs, the MatrixMarket format (very popular in R), and others alike.

Thanks! I would be happy to discuss this further and/or lend y'all a hand

[josdejong]

Makes a lot of sense. We should think through what API we want to offer. Mathjs is a library that works in both browser and node.js, so ideally we offer an API that can be used in both environments and/or is as much environment agnostic as possible. We could create a function fromReadableStream to create a DensMatrix or SparseMatrix maybe, building on the Streams API? And similarly a function fromCSV, or would that overlap too much with simply utilizing an existing CSV parser?

Note that there are some unit tests that use MatrixMarket, see
https://github.com/josdejong/mathjs/blob/develop/tools/matrixmarket.js
https://github.com/josdejong/mathjs/tree/develop/tools/matrices

[davidmrdavid]

Great! I'd love to take a stab at this, though I won't be freeing-up for a couple of weeks, so I shouldn't commit just yet.

Anyhow, I'm not fully understanding the use case of your fromReadableStream suggestion. Do you mean to say that users should have a different API for reading from disk v.s downloading a dataset from the web? I would argue that we could simply have a fromCSV(src) function that, with the help of some lightweight input validation on src, would determine whether or not to download from the web or read from disk.

[josdejong]

Thanks for your offer 👍

Before implementing anything I think it will be good to write out the various use cases we see. That way we can see what kind of API makes sense. What pops in my mind:

• fromCSV and toCSV functionality.
• Is CSV format the same as MatrixMarket data type or do we need a separate fromMTX and toMTX?
• the functions can import/export data as string and as a ReadableStream (that's great for .

We should be careful not to reinvent the wheel here, and embrace API's and libraries that are already out there. The idea for a "simple" fromCSV(src) may turn out not to be so simple: you need to built in support for nodejs and the browser, people will need to be able to send authentication headers and CORS headers etc etc.

It would be nice if we can allow people to use their rest client of choice and embrace that instead of creating a full blown rest client ourselves. So usage can look like:

// in the browser (user can add credentials, tokens, headers, whatever):
const dataString1 = await fetch(url).text() // <-- people use their loading mechanism of choice
const matrix1 = fromCSV(dataString1)

// in the browser, streaming for large amounts of data:
const dataStream2 = fetch(url).body // <-- people use their loading mechanism of choice
const matrix2 = fromCSV(dataStream2)

// in nodejs:
const dataStream3 = fs.createReadStream(file) // <-- people use their loading mechanism of choice
const matrix3 = fromCSV(dataStream3)

[davidmrdavid]

Good point, let's write down a handful of use cases.

Here's what I would suggest
As a minimum:

• fromCSV and toCSV
• fromMTX and toMTX because the two formats are quite different
  and then, as a stretch goal:
• fromCRS and toCRS (Compressed Row Storage)
• fromCCS and toCCS (Compressed Column Storage)

I think that would cover enough ground to accept most datasets you'd find online.

With respect to not reinventing the wheel, I agree that it would be most convenient to let people use their loading mechanism of choice. So long as we give a couple of examples on the website on how to do that, then it would probably be enough :)

Any other thoughts?

[josdejong]

Agree, sounds good 👍 . A number of examples is a good idea.

[danielruss]

How is this coming along. I could really use the ability to read matrix market files from R/python. So, I am willing to help. I have a little toy that takes when given an array of lines (the file) it can create a sparse matrix. It possibly can become a first iteration of fromMTX. However, I don't handle complex or double matrices.

[davidmrdavid]

@danielruss , I actually never got around it due to my year getting a little crazy. If you want to give it a stab, I'd say go ahead :)

[gwhitney]

Moving to Discussions.