Revisiting knowledgebase persistence

[guregu]

It'd be cool if we could automatically persist modules somehow. I don't think this needs to be built-in necessarily, instead it might be better to provide a hook for assert/retract and allow users to implement the persistence themselves.

db_log in predicates.c and the experimental persistence directive Trealla used to have is pretty close to what I'm looking for.

Some prior art would be prolog_listen/2,3 from SWI. I would be totally fine with less functionality and a different API though. For example, I don't need the ability to prevent modifications from happening, or be notified of retractall separately from retract, etc. Just hooks that already exist in db_log would be fine, as long as there is a way to grab clauses by the ID that db_log uses, so I know what data to persist.

My use case is a magical persistent interpreter that lives in the cloud™. I'd like to experiment with using Prolog knowledgebases to bang out CRUD APIs, like how auth.pl works. With WASM, we could call into the host in the assert/retract hooks and have the host persist the changes in whatever cloud database thing is convenient (or even browser local storage and such in the case of trealla-js).

[infradig]

Thanks for reminding me I should remove auth.pl file. Hooks or goal_expansion/2 might indeed work.

…

On Fri, 4 Nov 2022, 23:22 guregu, ***@***.***> wrote: It'd be cool if we could automatically persist modules somehow. I don't think this needs to be built-in necessarily, instead it might be better to provide a hook for assert/retract and allow users to implement the persistence themselves. db_log <https://github.com/trealla-prolog/trealla/blob/c91212a3b62f9d4d40668304326b66b3f27f14d2/src/predicates.c#L1661> in predicates.c and the experimental persistence directive Trealla used to have <https://github.com/guregu/trealla-old#persistenceexperimental> is pretty close to what I'm looking for. Some prior art would be prolog_listen/2,3 <https://www.swi-prolog.org/pldoc/man?predicate=prolog_listen/2> from SWI. I would be totally fine with less functionality and a different API though. For example, I don't need the ability to prevent modifications from happening, or be notified of retractall separately from retract, etc. Just hooks that already exist in db_log would be fine, as long as there is a way to grab clauses by the ID that db_log uses, so I know what data to persist. My use case is a magical persistent interpreter that lives in the cloud™. I'd like to experiment with using Prolog knowledgebases to bang out CRUD APIs, like how auth.pl <https://github.com/trealla-prolog/trealla/blob/main/library/auth.pl> works. With WASM, we could call into the host in the assert/retract hooks and have the host persist the changes in whatever cloud database thing is convenient. — Reply to this email directly, view it on GitHub <#74>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFNKSERYIM2RZQGBDF4WVTDWGUEZZANCNFSM6AAAAAARXFR4VY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[infradig]

The persist directive was really a hangover from a previous Prolog project. I used it to create a demo micro-blogging platform, basically a clone of HN. The thought of managing a web-server & database instance etc was too off-putting even for a toy. But maybe a cloud-based one with persistence to AWS or similar would be doable.

[guregu]

Hooks or goal_expansion/2 might indeed work.

Whoops, totally forgot about goal_expansion/2. That might do it.

The persist directive was really a hangover from a previous Prolog project. I used it to create a demo micro-blogging platform, basically a clone of HN.

Interesting! Sounds very cool. I agree that maintaining servers is a pain, especially for fun side projects.

One of Trealla's strengths is that it's very compact. This puts it in a unique position to fit inside the restrictions of serverless/FaaS/PaaS/buzzword runtimes, which have tough size restrictions (Cloudflare Workers is 1MB, for example). My eventual goal for the WASM stuff is to get Trealla running in one of them with persistence. Hopefully this will result in a zero-effort hosting situation for Prolog on the web.

[infradig]

I don't know much about file systems in the browser. Does it give a virtual file system to do usual C-style I/O? Is it persistent or get wiped per load. What about if running as worker thread in the cloud? Sorry for my woeful ignorance.

…

On Sat, Nov 5, 2022 at 8:25 PM guregu ***@***.***> wrote: Hooks or goal_expansion/2 might indeed work. Whoops, totally forgot about goal_expansion/2. That might do it. The persist directive was really a hangover from a previous Prolog project. I used it to create a demo micro-blogging platform, basically a clone of HN. Interesting! Sounds very cool. I agree that maintaining servers is a pain, especially for fun side projects. One of Trealla's strengths is that it's very compact. This puts it in a unique position to fit inside the restrictions of serverless/FaaS/PaaS/buzzword runtimes, which have tough size restrictions (Cloudflare Workers is 1MB, for example). My eventual goal for the WASM stuff is to get Trealla running in one of them with persistence. Hopefully this will result in a zero-effort hosting situation for Prolog on the web. — Reply to this email directly, view it on GitHub <#74 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFNKSEVTYYTIKPGN6FTL6M3WGYYYZANCNFSM6AAAAAARXFR4VY> . You are receiving this because you commented.Message ID: ***@***.***>

[guregu]

No need to apologize! This stuff is very new and unstable and there's like 10 different ways to do everything.

The Trealla WASM port targets a specification called WASI (WebAssembly System Interface). This is basically "POSIX for WASM" and gives us most of the standard C APIs we know and love. The tricky part is that each WASI runtime/library implements these APIs differently. WASI isn't (yet?) built into browsers but it is built into some serverless platforms.

Currently, trealla-js uses wasmer-js which has a virtual in-memory POSIX-ish filesystem. This is ephemeral and disappears on refresh, but it's easy to add persistence. The simplest way would be with localStorage, but there's also the Cache API to store arbitrary cache blobs, IndexedDB which is kind of like an embedded NoSQL DB, the FileSystem API that can access your actual OS filesystem but only works in Chrome, etc.
However, platforms like Cloudflare Workers use their own WASI library that I need to add support for (shouldn't be too hard). They have ephemeral filesystems, but cloud services to store blobs (S3 clones) and key-value DBs and whatnot.
Some runtimes, like wasmer-go which the Go port currently uses, don't have virtual filesystems but chroot-style ones instead.
To complicate things even further, there are runtimes that have their own nonstandard APIs for message passing and stuff but I'll ignore them for now.
And of course, anything that can run container images can run regular Trealla without needing WASM at all.

TL;DR: we have access to a POSIX filesystem but it's usually ephemeral, but it's generally straightforward to add a bit of glue to persist it