[Swift] Adds a way to store external data and use them directly within the ByteBuffer

[mustiikhalil]

Addresses an issue where we keep copying the data instead of using the data from the already created objects. This would fix the issue of using too much memory since it would copy a big object while already having that object in memory.

The solution for this was retaining a reference to the object within the ByteBuffer, and then using a pointer to the underlying data.

[mustiikhalil]

@liuliu we are changing how one of the API's for the ByteBuffer is working, would be nice to validate it on your end too since you were the one who requested it. And if this is merged, before you reply and something breaks you can just open a PR/ticket

[liuliu]

This seems to be good to me? My request (as implemented) is the assumingMemoryBound API that avoids the copy. And this change (seems to me) extends that to other types (Data, etc). Our original intention is to wrap SQLite returned blob pointer, which has a guaranteed lifetime for us (hence no need to wrap into Data and then use). As long as that API is kept, it seems great?

[mustiikhalil]

This seems to be good to me? My request (as implemented) is the assumingMemoryBound API that avoids the copy. And this change (seems to me) extends that to other types (Data, etc). Our original intention is to wrap SQLite returned blob pointer, which has a guaranteed lifetime for us (hence no need to wrap into Data and then use). As long as that API is kept, it seems great?

If that's the case all good! Now in theory it should not copy any of the data structures and simply retain them within the buffer & point towards their underlying memory.

Data, and arrays should be deallocated when the ByteBuffer is deallocated, but with assumingMemoryBound we will have to invoke the deallocate manually from the origin point of the pointer (which is the case currently too).

[liuliu]

Yeah, makes a lot of sense. In our case, the lifetime of that pointer is managed by SQLite. The only gotcha of this implementation might be holding Data for arbitrarily long time, especially if Data is a memory-mapped of big file (a few GiB) and only part of these are flatbuffers encoded (you could imagine a zip file with some metadata in flatbuffers, and the whole file is loaded in Data), that could be troublesome. But for people have that problem, they can always explicitly make a copy of the subdata at higher level.