Merge branch 'datajoint:master' into populate_success_count

.codespellrc

@@ -0,0 +1,5 @@
+[codespell]
+skip = .git,*.pdf,*.svg,*.csv,*.ipynb,*.drawio
+# Rever -- nobody knows
+# numer -- numerator variable
+ignore-words-list = rever,numer

.github/workflows/development.yaml

@@ -93,6 +93,16 @@ jobs:
           black datajoint --check -v
           black tests --check -v
           black tests_old --check -v
+  codespell:
+    name: Check for spelling errors
+    permissions:
+      contents: read
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+      - name: Codespell
+        uses: codespell-project/actions-codespell@v2
   publish-docs:
     if: |
       github.event_name == 'push' &&

.github/workflows/docs.yaml

@@ -0,0 +1,19 @@
+name: Manual docs release
+on:
+  workflow_dispatch:
+jobs:
+  publish-docs:
+    runs-on: ubuntu-latest
+    env:
+      DOCKER_CLIENT_TIMEOUT: "120"
+      COMPOSE_HTTP_TIMEOUT: "120"
+    steps:
+      - uses: actions/checkout@v3
+      - name: Deploy docs
+        run: |
+          export MODE=BUILD
+          export PACKAGE=datajoint
+          export UPSTREAM_REPO=https://github.com/${GITHUB_REPOSITORY}.git
+          export HOST_UID=$(id -u)
+          docker compose -f docs/docker-compose.yaml up --exit-code-from docs --build
+          git push origin gh-pages

CHANGELOG.md

@@ -1,5 +1,11 @@
 ## Release notes
 
+### Upcoming
+- Added - Codespell GitHub Actions workflow
+- Added - GitHub Actions workflow to manually release docs
+- Changed - Update `datajoint/nginx` to `v0.2.6`
+- Changed - Migrate docs from `https://docs.datajoint.org/python` to `https://datajoint.com/docs/core/datajoint-python`
+
 ### 0.14.1 -- Jun 02, 2023
 - Fixed - Fix altering a part table that uses the "master" keyword - PR [#991](https://github.com/datajoint/datajoint-python/pull/991)
 - Fixed - `.ipynb` output in tutorials is not visible in dark mode ([#1078](https://github.com/datajoint/datajoint-python/issues/1078)) PR [#1080](https://github.com/datajoint/datajoint-python/pull/1080)
@@ -31,7 +37,7 @@
 - Fixed - Fix queries with backslashes ([#999](https://github.com/datajoint/datajoint-python/issues/999)) PR [#1052](https://github.com/datajoint/datajoint-python/pull/1052)
 
 ### 0.13.7 -- Jul 13, 2022
-- Fixed - Fix networkx incompatable change by version pinning to 2.6.3 (#1035) PR #1036
+- Fixed - Fix networkx incompatible change by version pinning to 2.6.3 (#1035) PR #1036
 - Added - Support for serializing numpy datetime64 types (#1022) PR #1036
 - Changed - Add traceback to default logging PR #1036
 
@@ -83,7 +89,7 @@
 - Fixed - `schema.list_tables()` is not topologically sorted (#838) PR #893
 - Fixed - Diagram part tables do not show proper class name (#882) PR #893
 - Fixed - Error in complex restrictions (#892) PR #893
-- Fixed - WHERE and GROUP BY clases are dropped on joins with aggregation (#898, #899) PR #893
+- Fixed - WHERE and GROUP BY classes are dropped on joins with aggregation (#898, #899) PR #893
 
 ### 0.13.0 -- Mar 24, 2021
 - Re-implement query transpilation into SQL, fixing issues (#386, #449, #450, #484, #558). PR #754

LNX-docker-compose.yml

@@ -44,7 +44,7 @@ services:
       interval: 15s
   fakeservices.datajoint.io:
     <<: *net
-    image: datajoint/nginx:v0.2.5
+    image: datajoint/nginx:v0.2.6
     environment:
       - ADD_db_TYPE=DATABASE
       - ADD_db_ENDPOINT=db:3306

README.md

@@ -5,10 +5,18 @@
 
 # Welcome to DataJoint for Python!
 
-DataJoint for Python is a framework for scientific workflow management based on relational principles. DataJoint is built on the foundation of the relational data model and prescribes a consistent method for organizing, populating, computing, and querying data.
-
-DataJoint was initially developed in 2009 by Dimitri Yatsenko in Andreas Tolias' Lab at Baylor College of Medicine for the distributed processing and management of large volumes of data streaming from regular experiments. Starting in 2011, DataJoint has been available as an open-source project adopted by other labs and improved through contributions from several developers.
-Presently, the primary developer of DataJoint open-source software is the company DataJoint (https://datajoint.com).
+DataJoint for Python is a framework for scientific workflow management based on 
+relational principles. DataJoint is built on the foundation of the relational data 
+model and prescribes a consistent method for organizing, populating, computing, and 
+querying data.
+
+DataJoint was initially developed in 2009 by Dimitri Yatsenko in Andreas Tolias' Lab at 
+Baylor College of Medicine for the distributed processing and management of large 
+volumes of data streaming from regular experiments. Starting in 2011, DataJoint has 
+been available as an open-source project adopted by other labs and improved through 
+contributions from several developers.
+Presently, the primary developer of DataJoint open-source software is the company 
+DataJoint (https://datajoint.com).
 
 ## Data Pipeline Example
 
@@ -18,7 +26,13 @@ Presently, the primary developer of DataJoint open-source software is the compan
 
 ## Getting Started
 
-- Install from PyPI
+- Install with Conda
+
+     ```bash
+     conda install -c conda-forge datajoint
+     ```
+
+- Install with pip
 
      ```bash
      pip install datajoint
@@ -33,9 +47,4 @@ Presently, the primary developer of DataJoint open-source software is the compan
 - Contribute
   - [Development Environment](https://datajoint.com/docs/core/datajoint-python/latest/develop/)
 
-  - [Guidelines](https://datajoint.com/docs/community/contribute/)
-
-- Legacy Resources (To be replaced by above)
-  - [Documentation](https://docs.datajoint.org)
-
-  - [Tutorials](https://tutorials.datajoint.org)
+  - [Guidelines](https://datajoint.com/docs/about/contribute/)

datajoint/__init__.py

@@ -1,5 +1,5 @@
 """
-DataJoint for Python is a framework for building data piplines using MySQL databases
+DataJoint for Python is a framework for building data pipelines using MySQL databases
 to represent pipeline structure and bulk storage systems for large objects.
 DataJoint is built on the foundation of the relational data model and prescribes a
 consistent method for organizing, populating, and querying data.

datajoint/autopopulate.py

@@ -118,7 +118,7 @@ def _job_key(self, key):
 
     def _jobs_to_do(self, restrictions):
         """
-        :return: the query yeilding the keys to be computed (derived from self.key_source)
+        :return: the query yielding the keys to be computed (derived from self.key_source)
         """
         if self.restriction:
             raise DataJointError(

datajoint/blob.py

@@ -449,7 +449,7 @@ def pack_dict(self, d):
         )
 
     def read_struct(self):
-        """deserialize matlab stuct"""
+        """deserialize matlab struct"""
         n_dims = self.read_value()
         shape = self.read_value(count=n_dims)
         n_elem = np.prod(shape, dtype=int)

datajoint/connection.py

@@ -113,7 +113,7 @@ def conn(
     :param init_fun: initialization function
     :param reset: whether the connection should be reset or not
     :param use_tls: TLS encryption option. Valid options are: True (required), False
-        (required no TLS), None (TLS prefered, default), dict (Manually specify values per
+        (required no TLS), None (TLS preferred, default), dict (Manually specify values per
         https://dev.mysql.com/doc/refman/5.7/en/connection-options.html#encrypted-connection-options).
     """
     if not hasattr(conn, "connection") or reset:

datajoint/schemas.py

@@ -21,7 +21,7 @@
 
 def ordered_dir(class_):
     """
-    List (most) attributes of the class including inherited ones, similar to `dir` build-in function,
+    List (most) attributes of the class including inherited ones, similar to `dir` built-in function,
     but respects order of attribute declaration as much as possible.
 
     :param class_: class to list members for

datajoint/table.py

@@ -230,7 +230,7 @@ def ancestors(self, as_objects=False):
 
     def parts(self, as_objects=False):
         """
-        return part tables either as entries in a dict with foreign key informaiton or a list of objects
+        return part tables either as entries in a dict with foreign key information or a list of objects
 
         :param as_objects: if False (default), the output is a dict describing the foreign keys. If True, return table objects.
         """

docs/.docker/pip_requirements.txt

@@ -8,3 +8,4 @@ mkdocs-gen-files
 mkdocs-literate-nav
 mkdocs-exclude-search
 mkdocs-jupyter
+mkdocs-section-index

docs/docker-compose.yaml

@@ -16,7 +16,7 @@ services:
       - ..:/main
     user: ${HOST_UID}:anaconda
     ports:
-      - 8080:80
+      - 80:80
     command:
       - sh
       - -c

docs/mkdocs.yaml

@@ -5,16 +5,20 @@ repo_url: https://github.com/datajoint/datajoint-python
 repo_name: datajoint/datajoint-python
 nav:
   - DataJoint Python: index.md
-  - Getting Started: getting-started/index.md
+  - Quick Start Guide: quick-start.md
   - Concepts: 
     - Principles: concepts/principles.md
-    - Glossary: concepts/glossary.md
+    - Data Model: concepts/data-model.md
+    - Data Pipelines: concepts/data-pipelines.md
+    - Teamwork: concepts/teamwork.md
+    - Terminology: concepts/terminology.md
   - System Administration: 
-    - Database Administration: sysadmin/dba.md
-    - File Storage: sysadmin/filestore.md
+    - Database Administration: sysadmin/database-admin.md
+    - Bulk Storage Systems: sysadmin/bulk-storage.md
+    - External Store: sysadmin/external-store.md
   - Client Configuration: 
     - Install: client/install.md
-    - Credentials: client/creds.md
+    - Credentials: client/credentials.md
     - Settings: client/settings.md
     - File Stores: client/stores.md
   - Schema Design:
@@ -25,6 +29,7 @@ nav:
       - Primary Key: design/tables/primary.md
       - Attributes: design/tables/attributes.md
       - Lookup Tables: design/tables/lookup.md
+      - Manual Tables: design/tables/manual.md
       - Blobs: design/tables/blobs.md
       - Attachments: design/tables/attach.md
       - Filepaths: design/tables/filepath.md
@@ -39,12 +44,14 @@ nav:
     - Schema Drop: design/drop.md
     - Schema Modification: design/alter.md
   - Data Manipulations:
+    - manipulation/index.md
     - Insert: manipulation/insert.md
     - Delete: manipulation/delete.md
     - Update: manipulation/update.md
     - Transactions: manipulation/transactions.md
   - Data Queries:
-    - Common Commands: query/common-commands.md
+    - Principles: query/principles.md
+    - Example Schema: query/example-schema.md
     - Fetch: query/fetch.md
     - Iteration: query/iteration.md
     - Operators: query/operators.md
@@ -60,16 +67,13 @@ nav:
     - Populate: compute/populate.md
     - Key Source: compute/key-source.md
     - Distributed Computing: compute/distributed.md
+  - Publish Data: publish-data.md
   - Internals:
     - SQL Transpilation: internal/transpilation.md
-  - Reproducibility:
-      - Table Tiers: reproduce/table-tiers.md
-      - Make Method: reproduce/make-method.md
-  - Existing Pipelines: existing-pipelines.md
   - Tutorials:
-      - tutorials/json.ipynb
+      - JSON Datatype: tutorials/json.ipynb
   - FAQ: faq.md
-  - Develop: develop.md
+  - Developer Guide: develop.md
   - Citation: citation.md
   - Changelog: changelog.md
   - API: api/ # defer to gen-files + literate-nav
@@ -125,6 +129,7 @@ plugins:
         - "*/archive/*md"
   - mkdocs-jupyter:
       include: ["*.ipynb"]
+  - section-index
 markdown_extensions:
   - attr_list
   - toc:
@@ -148,6 +153,7 @@ markdown_extensions:
   - pymdownx.magiclink # Displays bare URLs as links
   - pymdownx.tasklist: # Renders check boxes in tasks lists
       custom_checkbox: true
+  - md_in_html
 extra:
   generator: false # Disable watermark
   version:

docs/src/.overrides/partials/nav.html

@@ -14,7 +14,7 @@
         {#-
         Add DataJoint home link to navigation header, otherwise unchanged
         -#}
-        <a href="https://datajoint.com/docs/core/" title="DataJoint">
+        <a href="https://datajoint.com/docs/" title="DataJoint">
             ⬅ Home
         </a>
     </label>

docs/src/citation.md

@@ -1,7 +1,7 @@
 # Citation
 
-If your work uses the DataJoint API for Python, please cite the following manuscript and Research Resource Identifier (RRID):
+If your work uses the DataJoint for Python, please cite the following manuscript and Research Resource Identifier (RRID):
 
 - Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
 
-- DataJoint API for Python - [RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543) - Version `Enter version here`
+- DataJoint for Python - [RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543) - Version `Enter version here`

docs/src/client/credentials.md

@@ -0,0 +1,46 @@
+# Credentials
+
+Configure the connection through DataJoint's `config` object:
+
+```python
+> import datajoint as dj
+DataJoint 0.4.9 (February 1, 2017)
+No configuration found. Use `dj.config` to configure and save the configuration.
+```
+
+You may now set the database credentials:
+
+```python
+dj.config['database.host'] = "alicelab.datajoint.io"
+dj.config['database.user'] = "alice"
+dj.config['database.password'] = "haha not my real password"
+```
+
+Skip setting the password to make DataJoint prompt to enter the password every time.
+
+You may save the configuration in the local work directory with 
+`dj.config.save_local()` or for all your projects in `dj.config.save_global()`.
+Configuration changes should be made through the `dj.config` interface; the config file 
+should not be modified directly by the user.
+
+You may leave the user or the password as `None`, in which case you will be prompted to 
+enter them manually for every session.
+Setting the password as an empty string allows access without a password.
+
+Note that the system environment variables `DJ_HOST`, `DJ_USER`, and `DJ_PASS` will 
+overwrite the settings in the config file.
+You can use them to set the connection credentials instead of config files.
+
+To change the password, the `dj.set_password` function will walk you through the 
+process:
+
+```python
+dj.set_password()
+```
+
+After that, update the password in the configuration and save it as described above:
+
+```python
+dj.config['database.password'] = 'my#cool!new*psswrd'
+dj.config.save_local()   # or dj.config.save_global()
+```