Contents
Lesson 13/Classic machine learning/~1 week

Classic CV and augmentation

An image as a tensor, filters you write by hand, and which augmentation is valid for which problem.

All lessonsLesson 13 · Classic CV and augmentation▾

An image is just a tensor of numbers. Before neural nets, it's worth understanding what you do with it: how you represent it, what a convolution filter is, and which augmentation helps without breaking the label. This module is the bridge to CNNs: if you understand the image as a tensor and convolution by hand, the convolutional layers later aren't magic anymore.

01

The image as a tensor

A tensor is a grid of numbers with several dimensions, the generalization of a matrix. A grayscale image is a 2D matrix: each number is the intensity of a pixel. A color image is a 3D tensor: height, width and 3 channels (red, green, blue). Each color pixel is three numbers.

img.shape       # (H, W, 3): height, width, 3 color channels
img = img / 255.0   # bring it from 0..255 to 0..1
02

Convolution: filters that slide over the image

A convolution filter is a small matrix (say 3x3) that you slide over the image. At each position, you multiply the filter with the patch of image under it and add up. The result is a new image that brings out a certain pattern: edges, blur, contrast.

It's worth writing a few classic filters by hand once. Blur averages the neighbors (smooths). Sharpen accentuates differences. Sobel detects edges, where intensity changes abruptly. Once you've written them, you understand exactly what a convolutional layer in a network does: the same operation, only the filters are learned, not written by you.

sobel_x = np.array([[-1, 0, 1],
                    [-2, 0, 2],
                    [-1, 0, 1]])
# you slide sobel_x over the image to pull out the vertical edges
03

Data augmentation

When you have few images, the model memorizes. Augmentation artificially grows the set by creating variants of the images: you flip them, rotate them, crop them, shift the colors a bit. The model sees the same label in slightly different forms and learns to generalize, not to memorize.

  • Horizontal flip: mirrors the image left to right.
  • Random rotation and crop: change the position and the framing.
  • Color jitter: slightly varies the brightness and color.
  • Cutout: covers a random patch, forcing the model not to rely on a single detail.
Practice problems
  • Write a Sobel filter by hand and apply it to an image to pull out the edges.
  • Test whether a set of augmentations improves the score on a small classification problem.
  • Take a list of augmentations and decide for each whether it keeps the label on a digit problem.
If you want more