Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Task]: Improve DB insert logic #1085

Open
aponcedeleonch opened this issue Feb 18, 2025 · 0 comments
Open

[Task]: Improve DB insert logic #1085

aponcedeleonch opened this issue Feb 18, 2025 · 0 comments

Comments

@aponcedeleonch
Copy link
Contributor

aponcedeleonch commented Feb 18, 2025

Description

These lines fire a concurrent db insert task for each new alert. This is not a problem in and of itself, since version 3+ of SQLite implements a multi-read/single-write concurrency model, but there's a catch.

The process of writing to the database requires the following logical steps

  1. obtain a SHARED lock -> still readable, new readers allowed
  2. obtain a RESERVED lock -> still readable, new readers allowed
  3. write some stuff durably to the transaction log -> still readable, new readers allowed
  4. obtain a PENDING lock -> still readable, no new readers allowed
  5. obtain an EXCLUSIVE lock -> not readable anymore

Here's SQLite docs page.

Each of the statements goes through this pipe, which means we have a bunch of (green) threads who likely all acquired a PENDING lock, so no new readers are allowed.

Thanks to @blkt for the above investigation

To avoid issues as the ones seen in #924 we should implement batch inserts into alerts table.

Handling all of the recording logic in a single transaction would also be ideal. Right now, we're inserting independently prompts, outputs and alerts. We could write into the 3 tables acquiring a lock for the transaction. Docs

BEGIN IMMEDIATE
-- write to prompts, output, and alerts
COMMIT;

Additional Context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants