Use big files without heap memory overflow

[truthz03]

Hi,

I'm currently developing a Blazor WebAssembly app which should use SQlite with EF Core.

I found out that Emscripten was used under the hood and because of the default system "MEMFS" my whole DB was loaded into the heap memory.
This is a problem because my DB can reach some 100MB and therefore I have fear to get out of memory problems.

I already tried IDBFS but I think this will not solve my problem because all data will still be inside the heap memory and will only be synced with the IndexedDB.
I already tried to use "autoPersist" but it doesn't work because it seems that .Net8 currently uses 3.1.34 and "autoPersist" was added in version 3.1.61: https://github.com/emscripten-core/emscripten/blob/main/ChangeLog.md#3161---053124
Will "autoPersist" solve my heap memory problem or doesn't this change anything?

I also found out that a new system called "WasmFS" is in progress. Will this solve my heap memory problem? Can this be used to directly write and read from a persistent file system without loading the whole data to the heap memory?

Thanks
Thomas

[sbc100]

100Mb doesn't seem to big by today's standard. Are you actually seeing OOM issues trying to using 100mb+ of memory?

[truthz03]

Yes I already got some problems.
On my tablet I get an emscripten_realloc_buffer error increasing from 352452608 -> 387710976
Because my Blazor app also needs about 250MB and my DB needs about 100MB I reach the limit.

My current woraround is to set <EmccMaximumHeapSize>1610612736</EmccMaximumHeapSize> inside my Blazor project.
This fixes the problem I if I understand correctly, setting the max to 1.6GB means that my DB can reach about 1.35GB bevore I get the problem again.
I don't know if this is a good practice but it works for now ^^

[sbc100]

That sounds like a reasonable solution. Perhaps re-open this issue if you start running into cases where your users browser don't have enough memory.

[sbc100]

Looking at IDBFS it looks like it supports persistence and it looks like the contents can potentially be stored outside of the heap area but it also looks like the entire file contents will still need to be stored in JS memory once of the FS is loaded:

emscripten/src/library_idbfs.js

Lines 250 to 264 in 35632d6

	storeLocalEntry: (path, entry, callback) => {
	try {
	if (FS.isDir(entry['mode'])) {
	FS.mkdirTree(path, entry['mode']);
	} else if (FS.isFile(entry['mode'])) {
	FS.writeFile(path, entry['contents'], { canOwn: true });
	} else {
	return callback(new Error('node type not supported'));
	}
	FS.chmod(path, entry['mode']);
	FS.utime(path, entry['timestamp'], entry['timestamp']);
	} catch (e) {
	return callback(e);
	}

So I don't think its will make much different to the overall memory used and your 100mb database will still appear in the overall memory used by the tab.