Reorganizing repo

.github/workflows/e2etest.yml

@@ -13,7 +13,8 @@ jobs:
         tar -xzf geocache.tgz
     - name: Run geocoder
       run: |
-        poetry run ./generate-geocodes.py --ids_filter test/random200-ids.txt --images_ndjson data/images.ndjson --output_format id-location.txt --geocode > test/random200-geocoded.txt 2> test/random200.logs.txt
+        PYTHONPATH=. poetry run oldnyc/geocode/geocode.py --ids_filter test/random200-ids.txt --images_ndjson data/images.ndjson --output_format id-location.txt --geocode > >(tee test/random200-geocoded.txt) 2> >(tee test/random200.logs.txt >&2)
+        # See https://stackoverflow.com/a/692407/388951 for the stdout/stderr redirection
     - name: Check for diffs
       run: |
         git diff --exit-code test/
@@ -25,11 +26,11 @@ jobs:
     - uses: ./.github/actions/setup
     - name: Run cropper
       run: |
-        poetry run ocr/crop_morphology.py --beta 2 --overwrite test/721675b.jpg
-        poetry run ocr/crop_morphology.py --beta 2 --overwrite --border_only --output_pattern '%s.border.jpg' test/721675b.jpg
+        poetry run oldnyc/crop/crop_to_text.py --beta 2 --overwrite test/721675b.jpg
+        poetry run oldnyc/crop/crop_to_text.py --beta 2 --overwrite --border_only --output_pattern '%s.border.jpg' test/721675b.jpg
     - name: Run photo detector
       run: |
-        poetry run nyc/find_pictures.py test/*f.jpg > test/detected-photos.ndjson
+        poetry run oldnyc/crop/find_pictures.py test/*f.jpg > test/detected-photos.ndjson
     - name: Check for diffs
       run: |
         git diff --exit-code test/
@@ -45,17 +46,16 @@ jobs:
         git clone https://github.com/oldnyc/oldnyc.github.io.git
     - name: Run ingestion
       run: |
-        PYTHONPATH=. poetry run data/ingest.py
+        PYTHONPATH=. poetry run oldnyc/ingest/ingest.py
     - name: Check for diffs
       run: |
         git diff --exit-code data/
     - name: Generate static site
       run: |
         export PYTHONPATH=.
-        poetry run ./nyc/crops-to-json.py nyc/crops.txt > /tmp/crops.json
-        poetry run ./nyc/records_to_photos.py data/images.ndjson /tmp/crops.json data/photos.ndjson
-        echo '{"fixes": {}}' > ocr/feedback/fixes.json
-        poetry run ./generate_static_site.py --leave-timestamps-unchanged
+        poetry run oldnyc/crop/records_to_photos.py data/images.ndjson data/crops.ndjson data/photos.ndjson
+        echo '{"fixes": {}}' > data/feedback/fixes.json
+        poetry run oldnyc/site/generate_static_site.py --leave-timestamps-unchanged
     - name: Check for diffs
       run: |
         cd ../oldnyc.github.io

.github/workflows/pytest.yml

@@ -8,7 +8,7 @@ jobs:
     steps:
     - uses: actions/checkout@v4
     - uses: ./.github/actions/setup
-    - run: poetry run pytest
+    - run: PYTHONPATH=. poetry run pytest
     - run: poetry run ruff check
     - run: poetry run ruff format --check
     - run: poetry run pyright

.vscode/settings.json

@@ -11,5 +11,8 @@
             "source.organizeImports.ruff": "always",
         },
         // "editor.wordBasedSuggestions": "off"
+    },
+    "markdownlint.config": {
+        "code-block-style": false
     }
 }

README.md

@@ -14,12 +14,10 @@ To get going on development:
 ```bash
 git clone git://github.com/danvk/oldnyc.git
 cd oldnyc
-virtualenv env
-source env/bin/activate
-pip install -r requirements.txt
+poetry install
 ```
 
-See [nyc/howto.md](nyc/howto.md) for more details on how to perform specific tasks.
+See [howto.md](howto.md) for more details on how to perform specific tasks.
 
 If you're interested in building your own "Old" site using this code, check out [this great writeup][3] on Old Ravenna.
 

data/README.md

@@ -1,19 +1,20 @@
-# ETL (Extract, Transform, Load)
+# OldNYC Ddata
 
-Goal is to pull together disparate data sources into a single `images.ndjson` file.
+All data comes together into a single `images.ndjson` file (see oldnyc/ingest).
 
-Format should be similar to the one used in [OldTO].
+Input data (sources of truth) live in `data/originals`. Files in the top-level
+`data` directory are derived from those and other sources.
 
 Inputs:
 
-- `nyc/milstein.csv`: the vintage 2013 CSV file from the NYPL that started it all
+- `data/originals/milstein.csv`: the vintage 2013 CSV file from the NYPL that started it all
   - Contains image ID (typically starts with "7" and ends with "f")
   - Contains title, alt_title
   - Contains dates
   - Contains creator
   - Contains source (corresponds to subcollection)
   - Contains Address and Full Address (unclear provenance)
-- `Milstein_data_for_DV.csv`: the 2024 update to the CSV
+- `data/originals/Milstein_data_for_DV.csv`: the 2024 update to the CSV
   - Contains image ID (with a capital "F" this time)
   - Contains title (which may have changed since 2013)
   - Contains two UUIDs, which can be used to construct a Digital Collections (DC) URL
@@ -22,3 +23,6 @@ Inputs:
 - `data.json`: contains OCR text from 2015 (Ocropy) plus manual fixes
 - `gpt-text.json`: contains OCR text from 2024 via OpenAI
 
+TODO:
+
+- Document provenance for all files.

feedback/README.md → data/feedback/README.md

@@ -2,8 +2,8 @@
 
 OldNYC incorporates user feedback in a variety of ways, most notably:
 
-  * Detection of rotated images
-  * OCR correction
+- Detection of rotated images
+- OCR correction
 
 This document describes how to pull in new user feedback and push the
 changes to the site.
@@ -15,19 +15,18 @@ side-by-side on the file system.
 
 Usage:
 
-    curl "https://brilliant-heat-1088.firebaseio.com/.json?print=pretty&auth=..." -o feedback/user-feedback.json
-    cp feedback/user-feedback.json feedback/user-feedback.$(date +%Y-%m-%dT%H:%M:%S).json
+    curl "https://brilliant-heat-1088.firebaseio.com/.json?print=pretty&auth=..." -o data/feedback/user-feedback.json
+    cp data/feedback/user-feedback.json data/feedback/user-feedback.$(date +%Y-%m-%dT%H:%M:%S).json
 
 This will update `feedback/user-feedback.json`.
 
 ## Step 2: Update rotations
 
 Run:
 
-    cd analysis/rotations
-    ./extract_rotations.py
+    poetry run oldnyc/feedback/extract_rotations.py
 
-This will update `analysis/rotations/corrections.json`
+This will update `data/feedback/corrections.json`
 
 Commit all the rotations and you'll be able to review them via `git webdiff` when you run `generate_rotated_images.py` in the oldnyc.github.io repo.
 
@@ -37,7 +36,7 @@ To review the changes before committing, use this [localturk] template:
 
 ```bash
 (echo 'photo_id,rotation'; git diff rotations.json | grep '^\+' | grep -v 'last_date' | sed 1d | sed 's/\+ *"//' | sed 's/,//' | sed 's/": /,/') > /tmp/new-rotations.txt
-localturk template.html /tmp/new-rotations.txt checked-rotations.txt
+localturk oldnyc/feedback/rotation-review.html /tmp/new-rotations.txt checked-rotations.txt
 ```
 
 [localturk]: https://github.com/danvk/localturk
@@ -46,24 +45,23 @@ localturk template.html /tmp/new-rotations.txt checked-rotations.txt
 
 Run:
 
-    cd ocr/feedback
-    ./extract_user_ocr.py
-    ./ocr_corrector.py
+    poetry run oldnyc/feedback/extract_user_ocr.py
+    poetry run oldnyc/feedback/ocr_corrector.py
 
-This will update `ocr/feedback/{corrections,fixes}.json`.
+This will update `data/feedback/{corrections,fixes}.json`.
 `corrections.json` is an exhaustive list of new OCR corrections, while
 `fixes.json` includes just one corrected version of the text for each
 image.
 
-To manually review updates, open review/index.html in a browser.
+To manually review updates, open `data/feedback/review/index.html` in a browser.
 
 To reject some changes, re-run `ocr_corrector.py` as it suggests.
 
 ## Step 4: Update the static site
 
 Run:
 
-    ./generate_static_site.py
+    poetry run oldnyc/site/generate_static_site.py
     cd ../oldnyc.github.io
     git diff