Support dict-like database migration

[mrocklin]

Objects like HDF5 files, SQL databases, Mongo Databases, and dicts of chunks of things might all be convertible. One could also imagine adding a very thin Directory type that expressed a filesystem hierarchy as a nested record type.

A natural common interchange format here might include dicts of either chunks(DataFrame) or chunks(ndarray) depending on the underlying dshape. This would also be a good place for chunks(dynd.nd.array).

[cpcloud]

what exactly do you mean here by dict-like database?

do you mean this?

d = {'db1': MongoDB, 'db2': HDF5File}

[mrocklin]

We represent database-like things with Record dshapes. e.g.

>>>  discover('sqlite:///mydb')
{
 table1: var * {... },
 table2: var * {... },
}

>>> discover('myfile.hdf5')
{
 data: {x: 10 * 10 * int, y: 10 * 10 * int},
 date: datetime
}

Perhaps when I said dict-like I meant Record dshape.

I can imagine doing something like the following:

>>> into('myfile.hdf5', 'sqlite://my.db')

Probably there are no nice and clean direct mechanisms for this, but probably something like Dict(Chunks(pd.DataFrame)) would be a sufficient intermediary. This is all low-priority though. We don't have an immediate use for this.