Merge pull request #20 from oislen/dev

Update Repo Documentation
oislen · Feb 10, 2025 · 4f47761 · 4f47761
2 parents 36e00cc + 19fa5a6
commit 4f47761
Show file tree

Hide file tree

Showing 11 changed files with 741 additions and 540 deletions.
diff --git a/README.md b/README.md
@@ -2,39 +2,68 @@
 
 ## Overview
 
-This git repository contains code and configurations for implementing a Convolutional Neural Network to classify images containing cats or dogs. The data was sourced from the [dogs-vs-cats](https://www.kaggle.com/competitions/dogs-vs-cats/overview) Kaggle competition, and also from [freeimages.com](https://www.freeimages.com/) using a web scraper. Docker containers were used to deploy the application on an EC2 spot instances in order to scale up hardware and computation power. 
+This git repository contains code and configurations for implementing a Convolutional Neural Network to classify images containing cats or dogs. The data was sourced from the [dogs-vs-cats](https://www.kaggle.com/competitions/dogs-vs-cats/overview) Kaggle competition, and also from [freeimages.com](https://www.freeimages.com/) using a web scraper.
 
-## Repo Contents
+Two models were trained to classify the images; an AlexNet8 model via Keras and a VGG16 model via Torch.
 
-* The __aws__ subdirectory contains batch and shell scripts for configuring ec2 spot instances and the deploying docker container remotely. 
-* The __conda__ subdirectory contains batch and shell scripts for creating a local conda environment for the project. 
-* The __data_prep__ subdirectory contains python utility scripts to data cleansing and processing for modelling.
-* The __kaggle__ subdirectory contains python scripts for downloading and unzipping competition data from Kaggle.
-* The __model__ subdirectory contains python scripts for initiating and training CNN models.
-* The __ref__ subdirectory contains previous analysis and kernals on dogs vs cats classification from Kaggle community members.
-* The __report__ subdirectory contains reportable images and plots generated by the application.
-* The __webscrapers__ subdirectory contains webscraping tools for downloading cats and dogs images from [freeimages.com](https://www.freeimages.com/).
+Docker containers were used to deploy the application on an EC2 spot instances in order to scale up hardware and computation power. 
 
-## Application Scripts
+![Workflow](doc/catclassifier.jpg)
 
-The main dog and cat image classification application is contained within the root scripts:
+## Analysis Results
 
-* The __01_prg_kaggle_data.py__ script downloads / unzips the cat vs dogs competition data.
-* The __02_prg_scrape_imgs.py__ script scrapes additional cat and dog images from [freeimages.com](https://www.freeimages.com/).
-* The __03_prg_keras_model.py__ script trains, fits and makes image predictions of the cat and dog images using a CNN model.
-* The __analysis_results.ipynb__ file contains a high level summary aof the analysis results.
-* The __cons.py__ script contains programme constants and configurations.
-* The __Dockerfile__ builds the application container for deployment on ec2.
-* The __exeDocker.bat__ executes the Docker build process locally on windows.
-* The __requirements.txt__ file contains the python package dependencies for the application.
+The images were further normalised using rotations, scaling, zooming, flipping and shearing prior to the modelling training phase.
 
-## Analysis Results
+![Generator Plot](report/torch/generator_plot.jpg)
+
+Models were trained across 10 to 25 epochs using stochastic gradient descent and cross entropy loss. Learning rate reduction on plateau and early stopping were implemented as part of training procedure.
+
+![Predicted Images](report/torch/pred_images.jpg)
+
+See the analysis results notebook for a further details on the analysis; including CNN architecture and model performance.
+
+* https://nbviewer.org/github/oislen/CatClassifier/blob/main/report/torch_analysis_results.ipynb
+
+## Running the Application (Windows)
+
+### Anaconda
+
+Create a local conda environment for the Cat Classifier app using [anaconda](https://www.anaconda.com/):
+
+```
+conda create --name CatClassifier python=3.12 --yes
+conda activate CatClassifier
+pip install -r requirements.txt
+```
+
+Execute the webscrapers and model training pipeline using the following commands and the local conda environment:
+
+```
+:: run webscrapers
+python webscrapers/prg_scrape_imgs.py --run_download_comp_data --run_webscraper
+:: run model training pipeline
+python model/prg_torch_model.py --run_model_training --run_testset_prediction
+```
+
+The model training and evaluation report can be opened with:
+
+```
+jupyter lab --ip=0.0.0.0 --allow-root "report/torch_analysis_results.ipynb"
+```
+### Docker
+
+The latest version of the Cat Classifier app can be found as a [docker](https://www.docker.com/) image on dockerhub here:
+
+* https://hub.docker.com/repository/docker/oislen/cat-classifier
 
-See the analysis results notebook for a summary of the project; including image processing, CNN architecture and model performance.
-* https://nbviewer.org/github/oislen/CatClassifier/blob/main/notebooks/torch_analysis_results.ipynb
+The image can be pulled from dockerhub using the following command:
 
-## Docker Container
+```
+docker pull oislen/cat-classifier:latest
+```
 
-The application docker container is available on dockerhub here:
+The Cat Classifier app can then be started within a jupyter lab session using the following command and the docker image:
 
-https://hub.docker.com/repository/docker/oislen/cat-classifier
+```
+docker run --name cc --shm-size=512m --publish 8888:8888 -it oislen/cat-classifier:latest
+```
diff --git a/doc/catclassifier.drawio b/doc/catclassifier.drawio
@@ -0,0 +1,94 @@
+<mxfile host="app.diagrams.net" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36" version="25.0.3">
+  <diagram name="Page-1" id="bBaojYlcXimtXHP4Su1T">
+    <mxGraphModel dx="1434" dy="746" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="850" pageHeight="1100" math="0" shadow="0">
+      <root>
+        <mxCell id="0" />
+        <mxCell id="1" parent="0" />
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-1" value="freeimages.com" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#ffe6cc;strokeColor=#000000;" vertex="1" parent="1">
+          <mxGeometry x="30" y="110" width="100" height="80" as="geometry" />
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-2" value="Kaggle" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#ffe6cc;strokeColor=#000000;" vertex="1" parent="1">
+          <mxGeometry x="30" y="270" width="100" height="80" as="geometry" />
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-3" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;exitPerimeter=0;entryX=0;entryY=0;entryDx=0;entryDy=15;entryPerimeter=0;" edge="1" parent="1" source="kfa_lcFOGdhym_HThWfJ-1" target="kfa_lcFOGdhym_HThWfJ-23">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="350" y="320" as="sourcePoint" />
+            <mxPoint x="230" y="200" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-4" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;exitPerimeter=0;entryX=0;entryY=1;entryDx=0;entryDy=-15;entryPerimeter=0;" edge="1" parent="1" source="kfa_lcFOGdhym_HThWfJ-2" target="kfa_lcFOGdhym_HThWfJ-23">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="350" y="320" as="sourcePoint" />
+            <mxPoint x="230" y="220" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-5" value="Data Collection" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;strokeColor=default;fillColor=default;" vertex="1" parent="1">
+          <mxGeometry x="110" y="20" width="90" height="30" as="geometry" />
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-7" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0;exitDx=0;exitDy=52.5;exitPerimeter=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="1" source="kfa_lcFOGdhym_HThWfJ-2" target="kfa_lcFOGdhym_HThWfJ-24">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="400" y="310" as="sourcePoint" />
+            <mxPoint x="520" y="313" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-9" value="" style="endArrow=none;dashed=1;html=1;dashPattern=1 3;strokeWidth=2;rounded=0;" edge="1" parent="1">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="310" y="450" as="sourcePoint" />
+            <mxPoint x="310" y="40" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-11" value="" style="endArrow=none;dashed=1;html=1;dashPattern=1 3;strokeWidth=2;rounded=0;" edge="1" parent="1">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="630" y="450" as="sourcePoint" />
+            <mxPoint x="630" y="50" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-12" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;exitPerimeter=0;" edge="1" parent="1" source="kfa_lcFOGdhym_HThWfJ-23" target="kfa_lcFOGdhym_HThWfJ-13">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="310" y="200" as="sourcePoint" />
+            <mxPoint x="410" y="230" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-13" value="&lt;div&gt;AlexNet8 Model&lt;/div&gt;&lt;div&gt;VGG16 Model&lt;/div&gt;&lt;div&gt;&lt;hr&gt;&lt;/div&gt;&lt;div&gt;37K Images&lt;/div&gt;&lt;div&gt;Image Transformations&lt;/div&gt;&lt;div&gt;Pretrained Model Weights&lt;/div&gt;&lt;div&gt;Early Stopping&lt;/div&gt;&lt;div&gt;Stochastic Gradient Descent&lt;/div&gt;&lt;div&gt;Learning Rate Deduction&lt;/div&gt;&lt;div&gt;Cross Entropy Loss Criterion&lt;/div&gt;" style="whiteSpace=wrap;html=1;aspect=fixed;rounded=1;align=center;fillColor=#d5e8d4;strokeColor=#000000;" vertex="1" parent="1">
+          <mxGeometry x="380" y="135" width="170" height="170" as="geometry" />
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-14" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.75;exitDx=0;exitDy=0;entryX=0;entryY=0;entryDx=0;entryDy=15;entryPerimeter=0;" edge="1" parent="1" source="kfa_lcFOGdhym_HThWfJ-13" target="kfa_lcFOGdhym_HThWfJ-25">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="400" y="310" as="sourcePoint" />
+            <mxPoint x="712" y="294" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-15" value="Output" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;strokeColor=default;fillColor=default;" vertex="1" parent="1">
+          <mxGeometry x="710" y="20" width="60" height="30" as="geometry" />
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-16" value="Modelling" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;strokeColor=default;fillColor=default;" vertex="1" parent="1">
+          <mxGeometry x="435" y="20" width="60" height="30" as="geometry" />
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-17" value="" style="endArrow=classic;html=1;rounded=0;exitX=1.024;exitY=0.394;exitDx=0;exitDy=0;exitPerimeter=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" edge="1" parent="1" source="kfa_lcFOGdhym_HThWfJ-13" target="kfa_lcFOGdhym_HThWfJ-18">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="400" y="310" as="sourcePoint" />
+            <mxPoint x="690" y="170" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-18" value="Jupyter Lab Report&lt;div&gt;&lt;hr&gt;Data Processing&lt;/div&gt;&lt;div&gt;Model Architecture&lt;/div&gt;&lt;div&gt;Model Performance&lt;/div&gt;&lt;div&gt;Image Prediction&lt;/div&gt;&lt;div&gt;&lt;br&gt;&lt;/div&gt;" style="whiteSpace=wrap;html=1;aspect=fixed;rounded=1;fillColor=#e1d5e7;strokeColor=#000000;" vertex="1" parent="1">
+          <mxGeometry x="680" y="140" width="120" height="120" as="geometry" />
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-23" value="Train and Validation Data" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#ffe6cc;strokeColor=#000000;" vertex="1" parent="1">
+          <mxGeometry x="180" y="180" width="100" height="80" as="geometry" />
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-24" value="Test Data" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#ffe6cc;strokeColor=#000000;" vertex="1" parent="1">
+          <mxGeometry x="180" y="350" width="100" height="80" as="geometry" />
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-25" value="Submission&lt;div&gt;Data&lt;/div&gt;" style="shape=cylinder3;whiteSpace=wrap;html=1;boundedLbl=1;backgroundOutline=1;size=15;fillColor=#ffe6cc;strokeColor=#000000;" vertex="1" parent="1">
+          <mxGeometry x="690" y="360" width="100" height="80" as="geometry" />
+        </mxCell>
+        <mxCell id="kfa_lcFOGdhym_HThWfJ-26" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;exitPerimeter=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="1" source="kfa_lcFOGdhym_HThWfJ-24" target="kfa_lcFOGdhym_HThWfJ-25">
+          <mxGeometry width="50" height="50" relative="1" as="geometry">
+            <mxPoint x="400" y="310" as="sourcePoint" />
+            <mxPoint x="450" y="260" as="targetPoint" />
+          </mxGeometry>
+        </mxCell>
+      </root>
+    </mxGraphModel>
+  </diagram>
+</mxfile>
diff --git a/doc/catclassifier.jpg b/doc/catclassifier.jpg
diff --git a/report/keras/AlexNet8_archecture.png → report/keras/AlexNet8_architecture.png b/report/keras/AlexNet8_archecture.png → report/keras/AlexNet8_architecture.png