capstone-project-3

MScA Robotics Capstone for Group 3 staring Fall 2019

Setup

Prerequisites

This setup requires a Raspberry Pi3 or Pi4 with the OS Raspberrian for Robots by DexterOS installed. It also requires Python3, OpenCV, and tf.lite.Interpreter (via tflite_runtime).

Instructions for Installing:

• Raspberrian for Robots
• OpenCV (Pi3, Pi4)
• Python3 (needs to be 3.6 or greater)
• tf.lite.interpreter (Needs to be the Linux ARM version that corresponds to your Pi's processor architecture and Python version)

NOTE: Run "uname -m" to confirm. If it says aarch64 then it is 64 bit. If it says armv7l then it is 32 bit.

• EDGE TPU Runtime

Next, for the audio portion there are additional needs: the low-level virual machine and pyaudio. Both are better installed through apt-get then they are through pip.

sudo apt-get install llvm
sudo apt-get install python3-pyaudio

Directory Setup

Clone the repository and install required packages.

git clone [email protected]:MScA-Robotics/capstone-project-3.git
cd capstone-project-3
pip3 install -r requirements.txt

There are required directories and models that are not committed to the repo. These are needed for the models to run and save data.

Within the scav-hunt directory, create a models directory and a noise directory. Within models create an audio and visual directory.

scav-hunt/
|-- models/
|   |-- audio/
|   |-- visual/
|-- noise/    

Models

There are two models needed for the scavenger hunt. An audio model and a visual model.

• The audio model needs to be a tflite model with model.tflite file and labels.txt in the sca-hunt/models/audio directory. More information and UChicago download links can be found in the folder's README

• The visual model is a tensorflow-lite model and needs to be a .tflite file with an associated labelmap.txt in the same directory.

Download both and put them in the associtated directory structure.

Also needed are the four numpy array files that make up the noise masking for the audio file.

scav-hunt/
|-- models/
|   |-- audio/
|   |   |-- model.tflite
|   |   |-- labels.txt
|   |   |-- README.md
|   |-- visual/
|   |   |-- visual_model/
|   |   |   |-- model.tflite
|   |   |   |-- labels.txt
|-- noise/
|   |-- mean_freq.npy
|   |-- noise_db.npy
|   |-- noise_thresh.npy
|   |-- std_freq.npy

Finally, copy the example boundaries file into it's own boundaries file.

cd scav-hunt/coneutils
cp boundaries.json.example boundaries.json    

Setup is complete and the bot should run. Test by running the scavbot.py and scavear.py scripts. A microphone needs to be plugged in for scavear and a camera plugged in for scavbot.

python3 scavbot.py
python3 scavear.py