DermaML

---

Table of Contents

1. Overview

   1.1. Project Organization

   1.2. References

2. Getting Started

   2.1. Important Notes

   2.2. Setup Steps

3. Project Conventions

4. Documentation

---

1. Overview

A description of the project.

1.1. Project Organization

├── README.md       <- README for the project
├── LICENSE         <- license for contents of project
├── Makefile        <- Makefile containing useful shortcuts (`make` rules).
│                      Use `make help` to show the list of available rules.
├── pyproject.toml  <- Python project configuration file (e.g., Python package
│                      dependencies)
│
├── bin             <- project scripts and programs
│
├── data            <- project data. See "Project Conventions" section for
│   │                  data organization and processing conventions.
│   │
│   ├── final       <- data directly used to generate key results (e.g., data
│   │                  for figures in reports)
│   ├── processed   <- processed data that is ready for use
│   └── raw         <- source data in its original format
│
├── docs            <- project documentation
│   │
│   ├── api         <- source code documentation (generated by `make docs`)
│   ├── references  <- reference materials (e.g., research articles)
│   └── reports     <- research reports
│
├── extras          <- optional environment configuration files, etc.
│
├── notebooks       <- research notes and Jupyter notebooks. See "Project
│                      Conventions" section for notebook conventions.
│
├── src             <- source code for the project
│
├── tests           <- test code for the project
│
└── tox.ini         <- configuration file for tox testing framework

1.2. References

• List of key project references

---

2. Getting Started

2.1. Important Notes

• This project uses poetry to manage Python package dependencies.

• As of 2022/08/06, this project does not currently work with Python versions greater than 3.8.

  • Recommendation. If Python 3.8 is not your default Python version, use pyenv to install Python 3.8 and configure it as the Python version for the project.

2.2. Setup Steps

• Prerequisites

  • Install Python 3.8.x.

  • Install Poetry.

• Set up a dedicated virtual environment for the project. Any of the common virtual environment options (e.g., venv, direnv, conda) should work. Below are instructions for setting up a direnv or poetry environment. Note: to avoid conflicts between virtual environments, only one method should be used to manage the virtual environment.

  • direnv Environment. Note: direnv manages the environment for both the Python and shell.

    • Prerequisite. Install direnv.

    • Copy extras/dot-envrc to the project root directory, and rename it to .envrc.

      $ cd $PROJECT_ROOT_DIR
      $ cp extras/dot-envrc .envrc

    • Grant permission to direnv to execute the .envrc file.

      $ direnv allow

  • poetry Environment. Note: poetry only manages the Python environment (it does not manage the shell environment).

    • Prerequisite. Install Poetry.

    • Create a poetry environment that uses a specific Python executable. For instance, if python3.8 is on your PATH, the following command creates (or activates if it already exists) a Python virtual environment that uses python3.8.

      $ poetry env use python3.8

      For commands to use other Python executables for the virtual environment, see the Poetry Quick Reference.

• Upgrade pip to the latest released version.

  $ pip install --upgrade pip

• Install the Python packages required for the project.

  $ poetry install

  Note. For virtual environments not created with poetry (e.g., direnv), a system- or user-level installation of poetry might fail (e.g., if paths to Python packages required by poetry are missing from the PYTHONPATH environment variable in the virtual environment). To avoid having to manually modify PYTHONPATH, install poetry within the virtual environment before running poetry install:

  $ pip install poetry

• Download project data from remote storage.

  $ dvc pull

• Do research! Make discoveries! Advance knowledge!

---

3. Project Conventions

Guidelines for Organizing Data

• All data should be placed in the data directory.

• Data should be organized into the following subdirectories.

  • raw: source data in its original format. Data in this directiory data should never be modified.

  • processed: processed data that is ready for use. All data in this directory should be generated by a deterministic, automateable process (possibly multi-step) that uses only raw data as input.

  • final: data directly used to generate key results (e.g., data for figures in reports). All data in this directory should be generated by a deterministic, automateable process (possibly multi-step) that uses processed and/or raw data as input.

Guidelines for Organizing notebooks Directory

• Depending on the nature of the project, it may be useful to organize the notebooks directory into sub-directories (e.g., by team member, by sub-project).

Research Notes

• Research notes should be placed in the notebooks directory and should be named using the following conventions:

  YYYY-MM-AUTHOR_INITIALS-Notes.md

  where the year and month indicate the month that the notes were written.

  • The time period covered by each set of research notes should be adjusted to match the pace of the project (which may change over time). For instance, if updates are made only a few times a year, it is reasonable to omit the month from the file name: YYYY-AUTHOR_INITIALS-Notes.md.

• When a non-trivial modification is made to an existing entry, the modification date should be indicated in a "last updated" line that immediately follows the entry header. For instance:

  _Last Updated_: 2022-05-31

Jupyter Notebooks

• Jupyter notebooks should be placed in the notebooks directory and should be named using the following convention:

  YYYY-MM-DD-AUTHOR_INITIALS-BRIEF_DESCRIPTION.ipynb

  where the date used for the notebook is approximately the date the original experiment was performed.

  • Example: 2019-01-17-KC-information_theory_analysis.ipynb

• Notebook Modifications

  • When minor modifications are made to a notebook (e.g., code updates that do not materially change the results, addition of a few of related experiments), use a "History" block (in Markdown format) to document the changes. Example:

    ### History
    
    #### 2022-05-31
    - Replaced `seaborn.distplot()` with `seaborn.histplot()`
      because `distplot()` has been deprecated.

  • When signification changes are made to a notebook (e.g., major modifications to algorithms, addition of experiments to explore a new direction), the modified notebook should saved to a new file with a name constructed from the modification date and the initials of the person who made the modifications.

---

4. Documentation

• Automated Testing

• FastDS Quick Reference

• Julia Quick Reference

• Jupyter Quick Reference

• MLflow Quick Reference

• pdoc Quick Reference

• pyenv Quick Reference

• Poetry Quick Reference

• Project Environment

---