Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Merged by Bors] - sql: add copy-based migrations and VACUUM INTO #6085

Closed
wants to merge 79 commits into from
Closed

[Merged by Bors] - sql: add copy-based migrations and VACUUM INTO #6085

wants to merge 79 commits into from

Commits on Jun 1, 2024

  1. sql: improve database schema handling 

    Use consistent package/folder structure for local.sql and state.sql
databases.

When a new database is created, use schema script instead of running
all the migrations. Also, check that there's no schema drift by
diffing database schema against the schema script after running
migrations.
    @ivan4th
    ivan4th committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    7a97f7a View commit details
    Browse the repository at this point in the history

  2. sql: fixup: malsync and rewards

    @ivan4th
    ivan4th committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    6a53628 View commit details
    Browse the repository at this point in the history

  3. sql: add migration / schema drift related tests

    @ivan4th
    ivan4th committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    5db7e18 View commit details
    Browse the repository at this point in the history

  4. sql: make schema drift table ignore regexp configurable

    @ivan4th
    ivan4th committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    c2938b2 View commit details
    Browse the repository at this point in the history

  5. sql: refactor localsql / statesql tests

    @ivan4th
    ivan4th committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    ec3397e View commit details
    Browse the repository at this point in the history

  6. Update CHANGELOG.md

    @ivan4th
    ivan4th committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    1af47bb View commit details
    Browse the repository at this point in the history

  7. config: fix db presets for db schema drift detection

    @ivan4th
    ivan4th committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    7bb2b1b View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. sql: fix review comments

    @ivan4th
    ivan4th committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    bc98240 View commit details
    Browse the repository at this point in the history

  2. sql: add database schema handling docs

    @ivan4th
    ivan4th committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    895f50f View commit details
    Browse the repository at this point in the history

  3. Merge branch 'develop' into feature/schema-snapshot

    @ivan4th
    ivan4th committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    6f33999 View commit details
    Browse the repository at this point in the history

  4. config: fix build errors

    @ivan4th
    ivan4th committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    6257174 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. Fix README.md

    @ivan4th
    ivan4th committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    aada7c6 View commit details
    Browse the repository at this point in the history

  2. Fix go.mod / go.sum

    @ivan4th
    ivan4th committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    dbea047 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. sql: use go:generate for database schema files

    @ivan4th
    ivan4th committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    6bf114a View commit details
    Browse the repository at this point in the history

  2. sql: make schema drift fatal by default

    @ivan4th
    ivan4th committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    bd62a8e View commit details
    Browse the repository at this point in the history

  3. sql: remove db-ignore-schema-rx config option 

    Also, add a migration that removes stray tables from quicksynced DBs
    @ivan4th
    ivan4th committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    c9d1a6c View commit details
    Browse the repository at this point in the history

  4. sql: avoid cyclic dependencies in future coded migrations

    @ivan4th
    ivan4th committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    beccddd View commit details
    Browse the repository at this point in the history

  5. Merge branch 'develop' into feature/schema-snapshot

    @ivan4th
    ivan4th committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    8552a96 View commit details
    Browse the repository at this point in the history

  6. activation: fix test

    @ivan4th
    ivan4th committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    2e41e1a View commit details
    Browse the repository at this point in the history

  7. sql: fix tests

    @ivan4th
    ivan4th committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    dd2ed3e View commit details
    Browse the repository at this point in the history

  8. sql: fix query cache handling

    @ivan4th
    ivan4th committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    13e8c14 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. sql: update mocks

    @ivan4th
    ivan4th committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    fbf2880 View commit details
    Browse the repository at this point in the history

  2. sql: fix QueryCache related mocks

    @ivan4th
    ivan4th committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    6dbc902 View commit details
    Browse the repository at this point in the history

  3. node: don't print error twice upon failure

    @ivan4th
    ivan4th committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    09e36bb View commit details
    Browse the repository at this point in the history

  4. sql: fix schema drift on Windows

    @ivan4th
    ivan4th committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    9dc3f0c View commit details
    Browse the repository at this point in the history

  5. sql: update docs on schema handling

    @ivan4th
    ivan4th committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    8bbeeb5 View commit details
    Browse the repository at this point in the history

  6. sql, malsync: fix handling of context cancelation

    @ivan4th
    ivan4th committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    30dcf71 View commit details
    Browse the repository at this point in the history

  7. sql: split Schema.Migrate() method

    @ivan4th
    ivan4th committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    8506114 View commit details
    Browse the repository at this point in the history

  8. sql: another fix for Windows newlines in the schema

    @ivan4th
    ivan4th committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    feb39fe View commit details
    Browse the repository at this point in the history

  9. merge-nodes: fix test naming

    @ivan4th
    ivan4th committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    77b46d1 View commit details
    Browse the repository at this point in the history

  10. sql: close db on schema errors

    @ivan4th
    ivan4th committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    ee36ee4 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2024

  1. sql, datastore: remove unneeded mocks

    @ivan4th
    ivan4th committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    da6ab28 View commit details
    Browse the repository at this point in the history

  2. sql: fix naming

    @ivan4th
    ivan4th committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    cbf4195 View commit details
    Browse the repository at this point in the history

  3. sql: remove unneeded assertions from tests

    @ivan4th
    ivan4th committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    b56dc59 View commit details
    Browse the repository at this point in the history

  4. sql: schemagen: fix help

    @ivan4th
    ivan4th committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    f1f11fe View commit details
    Browse the repository at this point in the history

  5. Merge branch 'develop' into feature/schema-snapshot

    @ivan4th
    ivan4th committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    a172a94 View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. Merge branch 'develop' into feature/schema-snapshot

    @ivan4th
    ivan4th committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    b149e23 View commit details
    Browse the repository at this point in the history

  2. Moved database schema handling docs to CODING.md

    @ivan4th
    ivan4th committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    c2c7e6f View commit details
    Browse the repository at this point in the history

  3. api: fix database handling in the test

    @ivan4th
    ivan4th committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    0400d33 View commit details
    Browse the repository at this point in the history

  4. sql: fix identities test

    @ivan4th
    ivan4th committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    a2679a5 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. sql: simplify StateDatabase and LocalDatabase interfaces

    @ivan4th
    ivan4th committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    33b9501 View commit details
    Browse the repository at this point in the history

  2. node: make it possibe to allow localsql schema drift

    @ivan4th
    ivan4th committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    ff35334 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. sql: add copy-based migrations and VACUUM INTO 

    When vacuuming is required, copy the source database using VACUUM INTO
to a temporary database, perform migrations on the temporary database
with journal_mode=OFF and synchronous=OFF, and then copy back the
migrated temporary database over the original one using VACUUM INTO.
Fixes #6069
    @ivan4th
    ivan4th committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    3d38b8f View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2024

  1. sql: fix issues found by linter

    @ivan4th
    ivan4th committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    1df9c30 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'develop' into feature/schema-snapshot

    @ivan4th
    ivan4th committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    93a5087 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'develop' into feature/schema-snapshot

    @ivan4th
    ivan4th committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    b568b25 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2024

  1. sql: fix review comments

    @ivan4th
    ivan4th committed Jun 29, 2024
    Configuration menu
    Copy the full SHA
    2c29103 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2024

  1. Merge branch 'develop' into feature/schema-snapshot

    @ivan4th
    ivan4th committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    bd28a9f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'develop' into feature/schema-snapshot

    @ivan4th
    ivan4th committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    507aca6 View commit details
    Browse the repository at this point in the history

  3. sql: fix failing tests

    @ivan4th
    ivan4th committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    5b47e76 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'develop' into feature/schema-snapshot

    @ivan4th
    ivan4th committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    4090f64 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2024

  1. Merge branch 'develop' into feature/schema-snapshot

    @ivan4th
    ivan4th committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    945f2ec View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. Merge branch 'develop' into feature/schema-snapshot

    @ivan4th
    ivan4th committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    ce9647b View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. Merge branch 'develop' into feature/schema-snapshot

    @ivan4th
    ivan4th committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    1220037 View commit details
    Browse the repository at this point in the history

  2. statesql: update schema

    @ivan4th
    ivan4th committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    dcab056 View commit details
    Browse the repository at this point in the history

  3. tmp: fix lint errors (will need to revert)

    @ivan4th
    ivan4th committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    1d8d310 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. Revert "tmp: fix lint errors (will need to revert)" 

    This reverts commit 1d8d310.
    @ivan4th
    ivan4th committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    a942ccc View commit details
    Browse the repository at this point in the history

  2. Move statesql migrations to a separate package to avoid cyclic deps

    @ivan4th
    ivan4th committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    fe95d18 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'develop' into feature/schema-snapshot

    @ivan4th
    ivan4th committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    354dc56 View commit details
    Browse the repository at this point in the history

  4. sql: fix schemagen

    @ivan4th
    ivan4th committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    22847df View commit details
    Browse the repository at this point in the history

  5. Merge branch 'feature/schema-snapshot' into feature/vacuum-into

    @ivan4th
    ivan4th committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    5278bf4 View commit details
    Browse the repository at this point in the history

  6. Address comments

    @ivan4th
    ivan4th committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    25185be View commit details
    Browse the repository at this point in the history

  7. fetch: fix tests

    @ivan4th
    ivan4th committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    9dcdd59 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2024

  1. sql: fix error handling

    @ivan4th
    ivan4th committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    8bf64c7 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2024

  1. statesql, localsql: fix InMemoryTest

    @ivan4th
    ivan4th committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    e7b563b View commit details
    Browse the repository at this point in the history

  2. sql: fix in-memory db handling 

    Don't check for incomplete migrations in case of an in-memory DB
    @ivan4th
    ivan4th committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    5247124 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. Merge branch 'develop' into feature/schema-snapshot

    @ivan4th
    ivan4th committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    a59cc3c View commit details
    Browse the repository at this point in the history

  2. Addressed comments

    @ivan4th
    ivan4th committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    47fbc84 View commit details
    Browse the repository at this point in the history

  3. sql: ignore whitespace during schema drift checks

    @ivan4th
    ivan4th committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    2e7e150 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'feature/schema-snapshot' into feature/vacuum-into

    @ivan4th
    ivan4th committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    01c3686 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'develop' into feature/vacuum-into

    @ivan4th
    ivan4th committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    bd81eb5 View commit details
    Browse the repository at this point in the history

  6. sql: fix failing migration tests on Windows

    @ivan4th
    ivan4th committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    670df1b View commit details
    Browse the repository at this point in the history

  7. Address comments

    @ivan4th
    ivan4th committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    d69eab4 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2024

  1. Address more comments

    @ivan4th
    ivan4th committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    26aa21c View commit details
    Browse the repository at this point in the history

  2. sql: improve error handling during migrations

    @ivan4th
    ivan4th committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    3a3bd94 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. sql: hold exclusive database locks during copy-based migration

    @ivan4th
    ivan4th committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    f462433 View commit details
    Browse the repository at this point in the history

  2. sql: further simplify error handling

    @ivan4th
    ivan4th committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    3119a63 View commit details
    Browse the repository at this point in the history

  3. Remove unneeded errors.Join

    @ivan4th
    ivan4th committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    11527ab View commit details
    Browse the repository at this point in the history

  4. Address comments

    @ivan4th
    ivan4th committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    64f4d38 View commit details
    Browse the repository at this point in the history