This class is called computer vision, which I suppose means that I’m supposed to tell you something about how exactly we can use computers to do this thing we call vision.
As we’ll see, doing this turns out to be significantly harder than people first expected.
Part of the challenge here is that vision, like other processes that involve the brain, can be a bit hard to pin down.
What exactly does it mean to see? To have vision?
You’re probably using your own vision system right now to read these words, but if I asked you to break down piece by piece how exactly your brain is processing the light that hits your retina into meaningful information, you would have a really tough time.
The vision researchers Peter Hart and Richard Duda had a really nice way of putting this when they wrote one of the first computer vision books: "Paradoxically, we are all expert at perception, but none of us knows much about it."
Now, as you may know, it’s taken longer than we thought, but we have made some good progress in computer vision. Today, the computer vision systems we’ve built are even better than humans at certain tasks.
So, how did we get here, how do these systems actually work, and what’s next? What does it mean that we can build really good systems for certain tasks? What problems have these systems solved and what new problems are new advances enabling?
| Order | Module | Topics | Libraries Used |
|---|---|---|---|
| 1 | The Original Problem | Image Processing, Edge Detection, Filtering, Hough Transform | Numpy, Matplotlib |
| 2 | Learning to See | Machine Learning Fundamentals, Bias and Variance, Overfitting, Decision Trees | Numpy, Matplotlib, Sklearn |
| 3 | Neural Networks Demystified | Neural Network Fundamentals, Backpropogation, Gradient Descent | Numpy, Matplotlib, Scipy |
| 4 | Autonomous Driving | Autonomous Driving History, Applications of Neural Networks, Computer Vision for Autonomous Driving | Numpy, Matplotlib, Scipy |
| 5 | Deep Learning | Training Your Own Models, Classification, Detection, GANs | Pytorch, fastai, Tensorflow, Keras |
| 6 | The 3d World | Keypoint Detection and Matching, Projective Geometry, Camera Calibration, 3d Reconstruction | Opencv, Opensfm |
Detailed course schedule is here. The course is broadly broken into 6 modules and 1 semester-long group project.
Upon successful completion of this course, you should be able to:
- Setup and manage a scientific python computing environment
- Implement key computer vision algorithms in python to solve real computer vision problems
- Train + Validate machine learning models to find data-based solutions to computer vision problems
- Interpret algorithm performance through measurement techniques and by creating clarifying visualizations
- Debug issues in computer vision algorithms in python
- Understand the difference between analytical and empirical techniques, and the role each plays in the field
- Discuss historical and current trends in computer vision and key research developments
- Collaborate with peers and advisors to find and implement solutions to real computer vision problems
Most modules will be accompanied with a coding challenge. These challenges will be automatically graded using an evaluation server. The evaluation server will also run anti-plagiarism checks to ensure each student's code is their own. Most coding challenges will be individual, some may be in groups. Students have a limited number of submissions to the evaluation server before the submission deadline. If possible, we will provide an automatically updating leaderboard. Coding challenges will be graded according to performance on an evaluation metric (accuracy, precision, recall..). Students should keep their solutions to challenges private, even after the course is over.
Each module will end with a short in-class quiz. These quizzes will test basic knowledge of the techniques covered in each section, and probe the understanding gained in the coding challenges.
Participation counts for 10% of your final grade, and will be monitored on Piazza. It provides useful statistics of a user's engagement on Piazza posts, discussion and other activities and these statistics will be used for grading. Click here for FAQs.
The course will include a semester-long group project. This project represents a significant portion of a student's grade, and will be broken up into graded checkpoints, including a final presentation.
We will make heavy use of python in this course, spend lots of time in jupyter notebooks, and frequently use numpy, matplotlib, scipy, sklearn, opencv, tensorflow, and pytorch.
Familiarity with the key techniques of linear algebra, univariate calculus will be useful. We will hit the ground running with python and numpy, so a little experience here would also helpful. The fastai courses are a really nice starting point.
We will draw from 3 texts in this course, each available online:
- Szeliski, Computer Vision: Algorithms and Applications
- Prince, Computer Vision: Models, Learning, and Inference
- Goodfellow, Deep Learning
- fastai book
Students are encouraged to discuss coding challenge strategies and techniques, but are not permitted to share code. We run auto-plagarism checks on all your submissions that compare your code to your peer's code. We also check against submissions from previous years. If we find that your code overlaps with another student's code, we are required to report you to the office of academic affairs. Students should keep their solutions to challenges private, even after the course is over. Quizzes are conducted on canvas, no collaboration is permitted on quizzes. The Course project will be completed in groups - students are of course encouraged to collaborate within their teams - and final projects will be posted on public GitHub pages.