long-lived IndexedDB transactions #29
rhashimoto
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I modified the sample IndexedDB VFS to try extending the life of IDB transactions to reduce the transaction overhead. This is inspired by some game-changing work from @jlongster on IDB with SQL.js.
Keeping an IDB transaction alive in an asynchronous context is tricky because they automatically close at the end of an event loop when no outstanding requests remain. You have to determine when the transaction might be idle and issue an extra request to bridge to the next VFS call. And not only that, a transaction has an "inactive" state when it won't accept requests - you can only issue requests in an event loop iteration in which it is created or it returns results.
I'm using two strategies for these bridging requests. When a file is locked, an indefinite number of extra requests will be made to keep the transaction alive, so all I/O made under a lock will be in a single IDB transaction. When a file is not locked, a small configurable number (currently 2) of extra requests will be made, and if exhausted then a new transaction will be opened the next time one is needed. In general, SQLite requests a lock for the main database file and not for journals.
This change makes a ballpark 2x to 4x improvement for most tests on the benchmark page on Chrome on my desktop machine with the default journaling, which is also written to IndexedDB. The improvements are even more dramatic with memory journaling, where many tests improve by 15x.
Further improvements are still possible, especially when journaling to IndexedDB (though this should probably be avoided for all but the most RAM-needy applications).
My VFS implementation was modified to use a separate IDB database for each SQLite file, which now seems like a mistake because creation/deletion, which happens a lot for journal files, is slow. My motivation for putting files in separate IDB databases was so that testing for file existence (which also happens a lot for journal files) could be done using(fixed).IDBFactory.databases
which doesn't require a transaction, but I didn't realize until later that Firefox doesn't have it yet and the fallback method of opening the database, checking if it was created, and deleting if so is agonizingly slowI also removed the LRU cache from the implementation so now there is no VFS cache at all. There are likely gains possible from adding a more specialized cache back in. Without a cache, a partial or unaligned write requires a block read first so a 1-block cache could help. This occurs a lot with journal files(1-block write cache added).Beta Was this translation helpful? Give feedback.
All reactions