Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fulltext refactor, including dupes checking #196

Open
wants to merge 78 commits into
base: main
Choose a base branch
from
Open

Fulltext refactor, including dupes checking #196

wants to merge 78 commits into from

Commits on Aug 18, 2023

  1. feat(test): add fixtures and docs to facilitate adding Fulltext

    @spool
    spool committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    eb4d977 View commit details
    Browse the repository at this point in the history

  2. feat(test): add fixtures and docs to facilitate adding Fulltext

    @spool
    spool committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    c298557 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2023

  1. feat(fulltext): update Fulltext model for alto2txt2fixtures changes

    @spool
    spool committed Sep 23, 2023
    Configuration menu
    Copy the full SHA
    07b97ef View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2023

  1. [pre-commit.ci] pre-commit autoupdate 

    updates:
- [github.com/pre-commit/mirrors-prettier: v3.0.2 → v3.0.3](pre-commit/mirrors-prettier@v3.0.2...v3.0.3)
- [github.com/adamchainz/django-upgrade: 1.14.1 → 1.15.0](adamchainz/django-upgrade@1.14.1...1.15.0)
- [github.com/asottile/pyupgrade: v3.10.1 → v3.13.0](asottile/pyupgrade@v3.10.1...v3.13.0)
- [github.com/psf/black: 23.7.0 → 23.9.1](psf/black@23.7.0...23.9.1)
- [github.com/Riverside-Healthcare/djLint: v1.32.1 → v1.34.0](djlint/djLint@v1.32.1...v1.34.0)
    @pre-commit-ci
    pre-commit-ci[bot] authored Sep 26, 2023
    Configuration menu
    Copy the full SHA
    05662dd View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2023

  1. fix(test): correct Fulltext.plaintext_path -> Fulltext.path

    @spool
    spool committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    00d8141 View commit details
    Browse the repository at this point in the history

  2. merge: main with fulltext-hmd

    @spool
    spool committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    890851e View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2023

  1. [pre-commit.ci] pre-commit autoupdate 

    updates:
- [github.com/pre-commit/mirrors-prettier: v3.0.2 → v3.0.3](pre-commit/mirrors-prettier@v3.0.2...v3.0.3)
- [github.com/adamchainz/django-upgrade: 1.14.1 → 1.15.0](adamchainz/django-upgrade@1.14.1...1.15.0)
- [github.com/asottile/pyupgrade: v3.10.1 → v3.13.0](asottile/pyupgrade@v3.10.1...v3.13.0)
- [github.com/psf/black: 23.7.0 → 23.9.1](psf/black@23.7.0...23.9.1)
- [github.com/Riverside-Healthcare/djLint: v1.32.1 → v1.34.0](djlint/djLint@v1.32.1...v1.34.0)
    @pre-commit-ci @spool
    pre-commit-ci[bot] authored and spool committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    887b066 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/pre-commit-ci-update-config' int… 

    …o pre-commit-ci-update-config
    @spool
    spool committed Sep 29, 2023
    Configuration menu
    Copy the full SHA
    d8d9aa6 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2023

  1. merge: poetry.lock fix

    @spool
    spool committed Oct 1, 2023
    Configuration menu
    Copy the full SHA
    dd42c7f View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2023

  1. fix: update deps for linux server build

    @griff-rees
    griff-rees committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    c3a7986 View commit details
    Browse the repository at this point in the history

  2. fix: bootstrap.js in vendors.js 5.3.1 -> 5.3.2

    @griff-rees
    griff-rees committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    ff7d89b View commit details
    Browse the repository at this point in the history

  3. chore: squash gazetteer, mitchells and newspaper database migra… 

    …tions
    @griff-rees
    griff-rees committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    e621e7e View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2023

  1. merge: fetch from GitHub

    @spool
    spool committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    3f4d3d6 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/fulltext-hmd' into fulltext-hmd-… 

    …dedupe
    @griff-rees
    griff-rees committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    257c618 View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2023

  1. feat: add lwmdb.utils::DupeRemoveConfig and helper functions

    @griff-rees
    griff-rees committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    2d0bcfc View commit details
    Browse the repository at this point in the history

  2. fix(ci): apply pre-commit to last commit

    @griff-rees
    griff-rees committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    9adc487 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2023

  1. chore: squash newspaper database migrations and fix docstrings

    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    55abc50 View commit details
    Browse the repository at this point in the history

  2. chore: remove backup newspaper and mitchells migrations

    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    ee44fc2 View commit details
    Browse the repository at this point in the history

  3. chore: remove residual mitchells migration squash

    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    3a9ed15 View commit details
    Browse the repository at this point in the history

  4. chore: clean up comments in lwmdb/utils.py

    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    ee8e81f View commit details
    Browse the repository at this point in the history

  5. fix(DedupeConfig): add some tests and fixes around DedupeConfig

    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    a660141 View commit details
    Browse the repository at this point in the history

  6. fix: clean up Mitchells migration

    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    83d2b06 View commit details
    Browse the repository at this point in the history

  7. feat(utils): improve info dupes_to_rm returns and logs

    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    defdade View commit details
    Browse the repository at this point in the history

  8. fix(convert_similar_qs_to_records): aggregate multiple dupe cases

    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    637b4d4 View commit details
    Browse the repository at this point in the history

  9. feat(test): parameterize test_dupes_to_rm for dupes > 1

    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    6cae85d View commit details
    Browse the repository at this point in the history

  10. feat(test): ensure full Newspaper aggregation tested in `test_dupes… 

    …_to_rm`
    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    c9a70be View commit details
    Browse the repository at this point in the history

  11. fix(fulltext): mv previous 0001_initial.py -> 0001_initial.py.bak

    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    fe34277 View commit details
    Browse the repository at this point in the history

  12. fix(fulltext): fix migrations squash to 0001_initial.py

    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    7430c3d View commit details
    Browse the repository at this point in the history

  13. fix(DedupeConfig): attempt to ensure records_to_keep are filterable… 

    … `QuerySets`
    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    eb76ee9 View commit details
    Browse the repository at this point in the history

  14. fix(filter_by_not_all_null_fk): qs.Model -> qs.model

    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    96ac85e View commit details
    Browse the repository at this point in the history

  15. fix(filter_by_not_all_null_fk): qs.model.filter -> `qs.model.object… 

    …s.filter`
    @spool
    spool committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    ebcb0c3 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2023

  1. fix(doc): replace mount config in local.yml to fix live reload

    @spool
    spool committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    8a4d5bf View commit details
    Browse the repository at this point in the history

  2. fix(utils): fix and rename filter_by_not_all_null_fk -> `qs_to_full… 

    …_record_qs`
    @griff-rees
    griff-rees committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    b1b0efa View commit details
    Browse the repository at this point in the history

  3. merge: resolve conflict

    @griff-rees
    griff-rees committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    6e5562e View commit details
    Browse the repository at this point in the history

  4. (doc): add manage_duplication.md to document dedupe options

    @griff-rees
    griff-rees committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    937450a View commit details
    Browse the repository at this point in the history

  5. feat(doc): expand examples demonstrating managing duplicate records

    @griff-rees
    griff-rees committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    c348b66 View commit details
    Browse the repository at this point in the history

  6. feat(ci): remove fulltext-hmd-dedupe and add fulltext-hmd branche… 

    …s to `GitHub` `CI`
    @griff-rees
    griff-rees committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    07af889 View commit details
    Browse the repository at this point in the history

  7. doc: update lwmdb-doc.svg database diagram schema

    @spool
    spool committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    4cbfb80 View commit details
    Browse the repository at this point in the history

  8. feat(doc): update lwmd-doc.svg -> lwmdb-doc-fulltext.svg

    @spool
    spool committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    2c28b88 View commit details
    Browse the repository at this point in the history

  9. feat(doc): rename lwmdb-doc-fulltext.svg -> `docs/assets/lwmdb-sche… 

    …ma.svg`
    @spool
    spool committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    97d9a02 View commit details
    Browse the repository at this point in the history

  10. feat: add required errors field to Fulltext and fix mkdocs `loc… 

    …al.yml`
    @spool
    spool committed Oct 8, 2023
    Configuration menu
    Copy the full SHA
    ae6959f View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2023

  1. fix(fulltext): rm fulltext fixture_path field

    @spool
    spool committed Oct 14, 2023
    Configuration menu
    Copy the full SHA
    dbfdd8c View commit details
    Browse the repository at this point in the history

  2. fix(fulltext): squash fulltext merges.

    @spool
    spool committed Oct 14, 2023
    Configuration menu
    Copy the full SHA
    f57156e View commit details
    Browse the repository at this point in the history

  3. fix(fulltext): rm squashed migrations

    @spool
    spool committed Oct 14, 2023
    Configuration menu
    Copy the full SHA
    afe8421 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2023

  1. feat(fulltext): finish fulltext/migrations squash

    @spool
    spool committed Oct 15, 2023
    Configuration menu
    Copy the full SHA
    d7962d2 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2023

  1. fix(ci): fix static whitenoise production deployment

    @griff-rees
    griff-rees committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    518d6b1 View commit details
    Browse the repository at this point in the history

  2. fix(chore): apply pre-commit lenting etc.

    @griff-rees
    griff-rees committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    f629fd6 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2023

  1. feat(refactor): move Fulltext model to newspapers and add `produc… 

    …tion` config gen
    @spool
    spool committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    b03af4b View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2023

  1. feat(FullText): rename Fulltext to FullText and tidy references/tests

    @spool
    spool committed Oct 29, 2023
    Configuration menu
    Copy the full SHA
    3040f5c View commit details
    Browse the repository at this point in the history

  2. feat(FullText): replace fulltext references with full_text

    @spool
    spool committed Oct 29, 2023
    Configuration menu
    Copy the full SHA
    58eda6e View commit details
    Browse the repository at this point in the history

  3. feat(FullText): add item_code and tests for matching Item records

    @spool
    spool committed Oct 29, 2023
    Configuration menu
    Copy the full SHA
    ba52244 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2023

  1. feat(test): add test_plaintext.json.zip and fixture loading test

    @griff-rees
    griff-rees committed Oct 30, 2023
    Configuration menu
    Copy the full SHA
    0a835fa View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2023

  1. fix(DataProvider): set 'code' fiel to use '-' like 'bl_hmd' -> 'bl-hmd'

    @spool
    spool committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    26651fa View commit details
    Browse the repository at this point in the history

  2. fix(test): add newspapers/tests/__init__.py

    @spool
    spool committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    9f22e87 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2023

  1. fix(doc): typo in lwmdb/utils.py docstring

    @spool
    spool committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    dc6cd41 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2023

  1. feat(FullText): add means of logging edge cases for Item -> FullText

    @spool
    spool committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    b59b4aa View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2023

  1. [pre-commit.ci] pre-commit autoupdate 

    updates:
- [github.com/pre-commit/mirrors-mypy: v1.7.0 → v1.7.1](pre-commit/mirrors-mypy@v1.7.0...v1.7.1)
    @pre-commit-ci
    pre-commit-ci[bot] authored Nov 27, 2023
    Configuration menu
    Copy the full SHA
    4210faf View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2023

  1. fix(chore): merge main to fulltext-refactor

    @spool
    spool committed Nov 30, 2023
    Configuration menu
    Copy the full SHA
    a97f4f8 View commit details
    Browse the repository at this point in the history

  2. feat(ci): update fulltext-refactor from 'origin/pre-commit-ci-updat… 

    …e-config'
    @spool
    spool committed Nov 30, 2023
    Configuration menu
    Copy the full SHA
    4b79b40 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2023

  1. chore(deps): bump jupyter-server from 2.9.1 to 2.11.2 

    Bumps [jupyter-server](https://github.com/jupyter-server/jupyter_server) from 2.9.1 to 2.11.2.
- [Release notes](https://github.com/jupyter-server/jupyter_server/releases)
- [Changelog](https://github.com/jupyter-server/jupyter_server/blob/main/CHANGELOG.md)
- [Commits](jupyter-server/jupyter_server@v2.9.1...v2.11.2)

---
updated-dependencies:
- dependency-name: jupyter-server
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
    @dependabot
    dependabot[bot] authored Dec 5, 2023
    Configuration menu
    Copy the full SHA
    a93292f View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2023

  1. [pre-commit.ci] pre-commit autoupdate 

    updates:
- [github.com/pre-commit/mirrors-prettier: v3.1.0 → v4.0.0-alpha.4](pre-commit/mirrors-prettier@v3.1.0...v4.0.0-alpha.4)
- [github.com/PyCQA/isort: 5.12.0 → 5.13.0](PyCQA/isort@5.12.0...5.13.0)
- [github.com/PyCQA/bandit: 1.7.5 → 1.7.6](PyCQA/bandit@1.7.5...1.7.6)
    @pre-commit-ci
    pre-commit-ci[bot] authored Dec 11, 2023
    Configuration menu
    Copy the full SHA
    51c6817 View commit details
    Browse the repository at this point in the history

Commits on Dec 15, 2023

  1. refactor: rename DupeRemov -> DupeFix

    @spool
    spool committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    41ccffe View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/dependabot/pip/jupyter-server-2.… 

    …11.2' into fulltext-refactor
    @spool
    spool committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    b0f49e2 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/pre-commit-ci-update-config' int… 

    …o fulltext-refactor
    @spool
    spool committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    e9cf7be View commit details
    Browse the repository at this point in the history

  4. fix: rename DupeFixConfig -> DupeConfig and fix `mitchells_data_p… 

    …ath` test `fixture`
    @spool
    spool committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    8fd2fb1 View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2024

  1. chore(deps-dev): bump jinja2 from 3.1.2 to 3.1.3 

    Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.3.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](pallets/jinja@3.1.2...3.1.3)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
    @dependabot
    dependabot[bot] authored Jan 11, 2024
    Configuration menu
    Copy the full SHA
    4ddfee7 View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2024

  1. fix(doc): docstring typos, improve func names, mailhog -> mailpit

    @spool
    spool committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    b2fc6e8 View commit details
    Browse the repository at this point in the history

  2. fix(chore): update jinja2 -> 3.1.3

    @spool
    spool committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    382d419 View commit details
    Browse the repository at this point in the history

  3. fix(chore): merge pre-commit updates

    @spool
    spool committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    4bcc27a View commit details
    Browse the repository at this point in the history

  4. fix(ci): add pre-fulltext-backups to .dockerignore

    @spool
    spool committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    607672c View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2024

  1. feat(ui): front end layout and branding

    @spool
    spool committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    3cbe379 View commit details
    Browse the repository at this point in the history

  2. fix(ci): add admin_cursor_navigator to mypy pre-commit config

    @spool
    spool committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    1dc1c02 View commit details
    Browse the repository at this point in the history

  3. fix: rm file

    @spool
    spool committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    0fa1aa2 View commit details
    Browse the repository at this point in the history

  4. fix(ci): add migrations used in server config

    @griff-rees
    griff-rees committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    7199134 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/fulltext-refactor' into fulltext… 

    …-refactor
    @griff-rees
    griff-rees committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    44bc2c6 View commit details
    Browse the repository at this point in the history

Commits on Jan 20, 2024

  1. fix: fulltext.__str__, mkdocs docker, svg formatting

    @griff-rees
    griff-rees committed Jan 20, 2024
    Configuration menu
    Copy the full SHA
    d63b056 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2024

  1. fix: rm mitchells and newspaper migrations

    @griff-rees
    griff-rees committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    1a3d189 View commit details
    Browse the repository at this point in the history

  2. fix(test): add of VITURAL_HOST to ProductionENVGenConfig write test

    @spool
    spool committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    204f3c9 View commit details
    Browse the repository at this point in the history