deft/Deep-Learning.org

• Plan
  • Neural Network and Deep Learning
    • Week 1: Introduction
    • Week 2: Basic of Neural Network programming
    • Week 3: One hidden layer Neural Networks
    • Week 4: Deep Neural Network
  • Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization
  • Structuring your Machine Learning project
  • Convolutional Neural Networks
  • Natural Language Processing: Building sequence models
• Neural Network and Deep Learning
  • Introduction
    • What is a neural network?
    • Supervised Learning with Neural Networks
      • Convolutional NN good for images
      • Strutured data (db of data) vs Unstructured data
    • Why is Deep Learning taking off?
  • Geoffrey Hinton interview
  • Binary Classification
  • Logistic Regression
  • Logistic Regression Cost Function
  • Gradient Descent
  • Derivatives
  • More Derivative Examples
  • Computaion Graph
  • Computing Derivatives
  • Computing Derivatives for multiple examples
  • Vectorization
  • Vectorizing Logistic Regression
  • Vectorizing Logistic Regression's Gradient Computation
  • Broadcasting in Python
  • Quick Tour of Jupyter / ipython notebooks
  • Neural Network Basics
• Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization
  • Week 1: Setting up your Machine
    • Recipe
    • Regularization
      • Regularization: reduce variance
      • Dropout Regularization
      • Over regularization methods
    • Setting up your optimization problem
      • Normalizing Inputs
      • Gradient Checking
        • Don't use gard check in traingin, only in debug
        • If algorithm fail, grad check, look at component (is db? dW? dW on certain layer, etc…)
        • Remember regularization
        • Doesn't work with dropout, turn off drop out (put 1.0) then check
        • Run at random initialization; perhaps again after training
  • Week 2: Optimization Algorithms
    • Mini batch
    • Minibatch size
    • Exponentially weighted average
  • Week 3: Hyperparameter
    • Video 1: use random not a grid to search for hyperparameter best value
    • Video 2: choose appropriate scale to pick hyperparameter
    • Hyperparameter: Tuning in practice Panda vs caviar
    • Batch normalization
      • In a network
      • Fitting Batch norm into a deep network
      • Why Batch Normalizing?
      • Batch Norm at test time
    • Multi-class classification
      • Softmax Regression
      • Training a softmax classifier
    • Introduction to programming frameworks
      • Deep learning frameworks
• Structuring your Machine Learning project
  • Week 1
    • Introduction to ML Strategy
      • Why ML Strategy
      • Orthogonalization
        • Chain of assumptions in ML
    • Setting up your goal
      • Single number evaluation metric
        • First
        • Another example
      • Satisficing and Optimizing metric
        • Another cat classification example
      • Train/dev/test distribution
        • Cat classification dev/test sets
        • True story (detail changed)
        • Guideline
      • Size of dev and test sets
        • Old way of splitting
        • Size of test set
      • When to change dev/test sets and metrics?
      • Orthogonalization for cat pictures: anti-pron
      • Another example
    • Comparing to Humand-level performance
      • Why human-level performance
      • Avoidable bias
        • Cat classification example
      • Understanding Human-level performance
        • Human-level error as proxy for Bayes error
        • Error analysis example
        • Summary of bias/variance with human-level perf
      • Surpassing human-level performance
        • Surpassing human-level performance
        • Problems where ML significantly surpasses human-level performance
      • Improving your model performance
        • The two fundamental assumptions of supervised learning
        • Reducing (avoidable) bias and variance
  • Week 2
    • Error Analysis
      • Error Analysis
        • Carrying out error analysis
        • Evaluate multiple idea in parallel
      • Cleaning up incorrectly labeled dataset
        • Incorrectly labeled examples
        • Error analysis
        • Correctin incorrect dev/test set examples
      • Buid your first system quickly then iterate
        • Speech recognition example
    • Mismatched training and dev/test set
      • Training and testing on different distributions
        • Cat app example
        • Speech recognition example
      • Bias and Variance with mismatched data distribution
        • Cat classifier example
        • Bias/variance on mismatched trainig and dev/test sets
        • More general formulation
      • Addressing data mismatch
        • Addressing data mismatch
        • Artificial data synthesis
        • Artificial data synthesis (2)
    • Learning from multiple tasks
      • Transfer learning
        • Transfer learning
        • When transfer learning makes sense
      • Multi-task learning
        • Simplified autonomous driving example
        • Neural network architecture
        • When multi-task learning makes sens.
    • End-to-end deep learning
      • What is end-to-end deep learning?
        • What is end-to-end deep learning?
        • Face recognition
        • More examples
      • Whether to use end-to-end deep learning
        • Pros and cons of end-to-end learning
        • Applying end-to-end deep learning
• Convolutional Neural Networks
  • Week 1
    • Computer Vision
    • Edge Detection Example
    • More edge detection
    • Padding
      • first solution but a border around the image: Padding
    • Strided Convolutions
      • Summary of convolutions
      • Convolution in math textbook (flip vertical and horizontal the filter)
    • Convolution over volumes
      • Multiple filters
    • One layer of convolutional neural network
      • Summary of notation
    • A simple convolution neural network example
      • Type of layer in a CNN
    • Pooling Layer
      • Max pooling
      • Example
      • Average Pooling
      • Summarize
    • CNN Example
      • Sizes
    • Why convolutions?
  • Week 2
    • Case studies
      • Why look at case studies?
        • Outline
      • Classic Networks
        • LeNet - 5
        • AlexNet
        • VGG - 16
      • Residual Networks (ResNets)
        • Residual block
      • Why ResNets Work
        • Why do residual networks work so well?
        • ResNet example
      • Networks in Networks and 1x1 Convolutions
        • What does a 1x1 convolution do?
        • Using 1x1 conv
      • Inception Network Motivation
        • Motivation for inception network.
        • The problem of computational cost
        • Using 1x1 conv
      • Inception Network
        • Inception module
        • Fun fact
    • Practical advice for using ConvNet
      • Using Open-Source Implementation
      • Transfer Learning
        • Transfer Learning
      • Data Augmentation
        • Common augmentation
        • Color shifting
        • Implementing distortions during training
      • State of Computer Vision
        • Data vs hand-engineered
        • Tips for doing well on benchmarks/winning competitions
        • Use open source code

– #+TITLE: Deep Learning Coursera – #+AUTHOR: Yann Esposito

Plan

5 courses

Neural Network and Deep Learning

Week 1: Introduction

Week 2: Basic of Neural Network programming

Week 3: One hidden layer Neural Networks

Week 4: Deep Neural Network

Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization

Structuring your Machine Learning project

Convolutional Neural Networks

Natural Language Processing: Building sequence models

DONE Neural Network and Deep Learning

CLOSED: [2017-08-22 Tue 13:43]

Introduction

What is a neural network?

Supervised Learning with Neural Networks

• Lucrative application: ads, showing the add you're most likely to click on
• Photo tagging
• Speech recognition
• Machine translation
• Autonomous driving

Convolutional NN good for images

Strutured data (db of data) vs Unstructured data

• Structured data: Tables
• Unstructured data: Audio, image, text…

Computer are much better at interpreting unstructured data.

Why is Deep Learning taking off?

• Data (lot of data)
• Computation (faster learning loop)
• Algorithms (ex, use ReLU instead of sigma)

Geoffrey Hinton interview

Binary Classification

\[ (x,y) x\in \mathbb{R}^{n_x}, y \in {0,1} \]

$m$ training examples: $$ {(x^{(1)},y^{(1)}), ... (x^{(m)},y^{(m)})} $$

$$ m = m_{train} , m_{test} = #test examples $$

$$ X = [ X^{(1)} ... X^{(m)} ] is an n_x x m matrix $$ $$ X.shape (n_x,m) $$

$$ Y = [ y^{(1)} ... y^{(m)} ] $$ $$ Y.shape = (1,m) $$

Logistic Regression

Given $X \in \mathbb{R}^{n_x}$ you want $\hat{y} = P(y=1 | X)$

Paramters: $w \in \mathbb{R}^{n_x}, b\in \mathbb{R}$

Output: $\hat{y} = \sigma(w^Tx + b) = \sigma(z)$

$$\sigma(z)= \frac{1}{1 + e^{-z}}$$

If $z \rightarrow \infty => \sigma(z) \approx 1$ If $z \rightarrow - \infty => \sigma(z) \approx 0$

Alternative notation not used in this course:

$X_0=1, x\in\mathbb{R}^{n_x+1}$ $\hat{y} = \sigma(\Theta^Tx)$ …

Logistic Regression Cost Function

Search a convex loss function:

$L(\hat{y},y) = - (y\log(\hat{y}) + (1-y)\log(1-\hat{y}))$

If y = 1 : $L(\hat{y},y) = -\log\hat{y}$ <- want log\haty larg, want \hat{y} large If y = 0 : $L(\hat{y},y) = -\log\hat{y}$ <- want log (1-\hat{y}) large, want \hat{y} sall

Cost function: $$ J(w,b) = \frac{1}{m}\sum_{i=1}^mL(\hat{y^\{(i)}},y^{(i)}) = ... $$

Gradient Descent

Minize $J(w,b)$

1. initialize w,b (generaly uses zero)
2. Take a step in the steepest descent direction
3. repeat 2 until reaching global optimum

Repeat { $w := w - \alpha\frac{dJ(w)}{dw} = w - \alpha\mathtext{dw}$ }

Derivatives

More Derivative Examples

Computaion Graph

Computing Derivatives

Computing Derivatives for multiple examples

Vectorization

getting rid of explicit for loops in your code

Vectorizing Logistic Regression

Vectorizing Logistic Regression's Gradient Computation

Broadcasting in Python

Quick Tour of Jupyter / ipython notebooks

Neural Network Basics

J = a*b + a*c - (b+c) = a (b + c) - (b + c) = (a - 1) (b + c)

DONE Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization

CLOSED: [2017-09-01 Fri 09:52]

DONE Week 1: Setting up your Machine

CLOSED: [2017-08-22 Tue 13:43]

Recipe

If High bias? (bad training set performance?) Then try:

• Bigger network
• Training longer
• (NN architecture search)

Else if High variance? (bad dev set performance?) Then try:

• More data
• Regularization
• (NN architecture search)

Deep learning, not much bias/variance tradeoff if we have a big amount of computer power (bigger network) and lot of data.

Regularization

Regularization: reduce variance

• L2 regularization

λ / 2m || w ||_2 ^2

• L1 regularization: same with |w| instead of ||w||_2^2

λ is a regularization parameter (in code named lambd)

Cost = J(w^[1], b^[1], …, w^[L], b^[L]) = 1/m ∑ L(^y(i), y(i)) + λ/2m ∑_l=1^L || W^[l] ||^2

call the "Frobenius norm"

dW = from backprop + λ/m W^l

update W^l = W^l - αdW^l still works

Sometime L2 regularization called "weight decay".

Dropout Regularization

Eliminates nodes by layer randomly for each training example.

• implementing, (inverted dropout)

  • gen random boolean vector: d3 = np.random.rand(a3.shape[0], a3.shape[1]) < keep_prob # (for each iteration) a3 = np.mulitply(a3,d3) a3 /= keep_prob (for normalization to be certain the a3 output still the same, reduce testing problems)

Making prediction at test time: no drop out

Over regularization methods

• Data augmentation, (flipping images for example, random crops, random distortions, etc…)
• Early stopping, stop earlier iteration

Setting up your optimization problem

Normalizing Inputs

• μ = 1/m Sum X^(i)
• x := x - μ (centralize)
• σ = 1/m Sum X^(i)^2
• x /= σ^2

Gradient Checking

Don't use gard check in traingin, only in debug

If algorithm fail, grad check, look at component (is db? dW? dW on certain layer, etc…)

Remember regularization

Doesn't work with dropout, turn off drop out (put 1.0) then check

Run at random initialization; perhaps again after training

DONE Week 2: Optimization Algorithms

CLOSED: [2017-08-22 Tue 13:43]

Mini batch

X :: X^(1) … X^(m)

X,Y -> Xⁱ,Yⁱ where Xⁱ = X^(i*batch-size —> (i+1)*batch-size)

Minibatch size

• if mini batch size = m => Batch gradient descent (X¹,Y¹) = (X,Y)
• if mini match size = 1 => Stochastic gradient descent, every example is its own mini batch.

• in practice in between 1 and m, m –> too long, 1 loose speedup from vectorization.

  • vectorization ~1000

1. If small training set, use batch gradient descent (m <= 2000)
2. Typical mini-batch size: 64, 128, 256, 512, … 2^k to fits in CPU/GPU memory

Exponentially weighted average

v_t = βv_t-1 + (1-β)θ_t

DONE Week 3: Hyperparameter

CLOSED: [2017-09-01 Fri 09:52]

Video 1: use random not a grid to search for hyperparameter best value

Video 2: choose appropriate scale to pick hyperparameter

• uniformly random n^[l] (number of neuron for layer l) or L (number of layers)
• alpha: between 0.00001 to 1, then shouldn't use linear but instead use log-scale r = -4*np.random.rand() <- r in [-4,0] α = 10^r <- 10^-4 … 10^0
• β <- 0.9 … 0.999 (0.9 about avg on 10 values, 0.999 avg about 1000 values) 1-β = 0.1 …. 0.001 r <- [-3,-1] 1-β = 10^r

Hyperparameter: Tuning in practice Panda vs caviar

• Babysitting one model (panda) for few computer resources
• Training many models in parallel (caviar) for lot of computer resources

Batch normalization

In a network

Fitting Batch norm into a deep network

Why Batch Normalizing?

• don't use batch norm as a regularization even if sometime it could have this effect

Batch Norm at test time

μ = 1/m ∑ z^(i)

σ^2 = 1/m ∑ (z^(i) - μ)^2

z^(i)_norm = z^(i) - μ / sqrt( σ^2 + ε )

~z^(i) = γz^(i)_norm + β

Estimate μ and σ with exponentially weighted avg accross minibatches

Multi-class classification

Softmax Regression

notation: C = #classes (0,1,2…,C-1)

last hidden layer nb of neuron is equal to C: n^L = C

z^[L] = w[L]a^[L-1] + b[L] (C,1)

Activation function:

t = e^(Z[L]) a^[L] = e^(Z[L])/∑_i=0^C t_i

a^[L]_i = t_i / ∑_i=0^C t_i

Training a softmax classifier

Introduction to programming frameworks

Deep learning frameworks

Structuring your Machine Learning project

Week 1

Introduction to ML Strategy

Why ML Strategy

Try to find quick and effective way to choose a strategy

Ways of analyzing ML problems

Orthogonalization

Chain of assumptions in ML

• Fit training set well on cost function => bigger network, Adam, …
• Fit dev set well on cost function => Regularization, Bigger training set
• Fit test set well on cost function => Bigger dev set
• Perform well in real world => Change the devset or cost function

Try not to use early stoping as it simulanously affect cost on training and dev set.

Setting up your goal

Single number evaluation metric

First

Classifier	Precision	Recall
A	95%	90%
B	98%	85%

Rather than using two number, find a new evaluation metric

Classifier	Precision	Recall	F1 Score
A	95%	90%	92.4%
B	98%	85%	91.0

F1 score = 2 / (1/p) + (1/R) :: "Harmonic mean" of precision and recall.

So:

Having a good Dev set + single evaluation metric, really speed up iterating.

Another example

Algorithm	US	China	India	Other	Average
A	3%	7%	5%	9%
…
F	…	…

Try to improve the average.

Satisficing and Optimizing metric

It's not alway easy to select on metric to optimize.

Another cat classification example

Classifier	Accuracy	Running Time
A	90%	80ms
B	92%	95ms
C	95%	1500ms

cost = accuracy - 0.5x running time

maximize accuracy s.t. running time < 100ms

Accuracy <- Optimizing Running time <- Satisficing

If you have n metrics, pick one to optimizing, and all the other be satisficing.

Train/dev/test distribution

How you can setup these dataset to speed up your work.

Cat classification dev/test sets

Try to find a way that dev and test set come from the same distribution.

True story (detail changed)

Optimizing on dev set on load approvals for medium income zip codes.

(repay loan?)

Tested on low income zip codes.

Lost 3 months

Guideline

Choose a dev set and test set to reflect data you expect to get in the future and consider important to do well on.

Size of dev and test sets

Old way of splitting

70% train, 30% test 60% train, 20% dev, 20% test

For at max 10^4 examples

But in new era, 10^6 examples:

train: 98%, Dev 1%, Test 1%.

Size of test set

Set your test set to be big enough to give high confidence in the overall performance of your system. Can be far less than 30% of your data.

For some applications, you don't need test set and only dev set. For example if you have a very large dev set.

When to change dev/test sets and metrics?

Metric: classification error Algorithm A: 3% error → letting throught a lot of porn images Algorithm B: 5% error → doesn't let pass porn images

So your metric + evaluation prefer A, but you and your users prefer B.

When this happens, mispredict your algorithm B is better.

Error: 1/m_dev ∑_i=1^m I{y_pred^(i) /= y^(i)

They treat pron and non pron equaly but you don't want that.

We add a w(i) = 1 if non porn and 0 if porn in the formula

Orthogonalization for cat pictures: anti-pron

1. So far we've only discussed how to define a metric to evaluate classifier
2. Worry separately about how to do well on this metric
3. placing the target, and 2. is aiming the target.

Another example

Alg A: 3% err Alg B: 5% err

But B does better. You see that users are using blurier images. You dev/test are not using the same kind of images.

Change your metric and/or dev/test set.

Comparing to Humand-level performance

Why human-level performance

Human-level perf vs Bayes optimal error

Human are generally very close to bayes perf for lot of tasks.

• get lableld data from humans
• gain insight from manual error analysis (why did a person get this right?)
• better analysis of bias/variance

Avoidable bias

Cat classification example

Humans	1%	7.5%
Training error	8%	8%
Dev error	10%	10%
	focus on bias	focus on variance

Human level error as a proxy (estimate) for Bayes error.

Diff between Human err and Training err = available bias Diff between Train and Dev err = variance

Understanding Human-level performance

Human-level error as proxy for Bayes error

Medical image classification example: suppose (a) Typical human 3% err (b) Typical doctor 1% err (c) Experienced doctor 0.7% err (d) and team of experienced doctors 0.5% err

What is "human-level" error?

Bayes error is <= to 0.5% err So we use that to aim as saw before.

For a paper, (b) is good enough to talk about that.

Error analysis example

Human (proxy for bayes err)	1, 0.7, 0.5%	1, 0.7, 0.5	1, 0.7, 0.5
Train err	5%	1%	0.7%
Dev err	6%	5%	0.8%

Case 1: For this example it doesn't matter because avoidable bias (5 - 1%), is bigger than variance (6-5)

Case 2: focus on variance

Case 3, very important you use 0.5 as your "human-level" error. Because it show that you should focus on bias and not on variance.

This problem arose only when you're doing very good.

Summary of bias/variance with human-level perf

Human-level error (proxy for Bayes err)

^

"Avoidable bias"

v

Training error

^

"Variance"

v

Dev error

Surpassing human-level performance

Surpassing human-level performance

Team	0.5%	0.5%
One human	1%	1%
Training error	0.6%	0.3%
Dev error	0.8%	0.4%
Avoidable bias?	~0.5%	can't know

Problems where ML significantly surpasses human-level performance

• Online advertising
• Product recommendations
• Logistics (predicting transit time)
• Loan approvals

all thoses examples:

• come from structured data
• not natural perception problems
• Lots of data

Also, Speech recognition, Some image recognition, Medical, ECG, skin cancer, etc…

Improving your model performance

Set of guidelines

The two fundamental assumptions of supervised learning

1. You can fit the training set pretty well (~ avoidable bias)
2. The training set performance generalizes pretty well to the dev/test set

Reducing (avoidable) bias and variance

Human-level error (proxy for Bayes err)

^

train bigger model

"Avoidable bias" => train longer/better optimization algorithms (momentum, RMSprop, Adam)

NN architecture/hyperparameters search (RSS, CNN…)

v

Training error

^

More data

"variance" => Regulraization (L2, dropout, data augmentation)

NN architecture/hyperparameters search

v

Dev error

These concepts are easy to learn, hard to master. You'll be more systematics than most ML teams.

Week 2

Error Analysis

Error Analysis

Carrying out error analysis

• Imagine your cat algo doesn't work as good as expected.
• One of your colaborator think you should focus on working on dogs.
• Anaylize manually 100 mislabeled dev set examples
• Count up how many are dogs
• Supose 5% are dogs. So at most you could go from 10% err to 9.5% so not much useful.
• Supose taht 50% of them are dogs error, so you could go down from 10% to 5%, so you could be more confident.

Evaluate multiple idea in parallel

• fix pictures of dogs
• fix great cats (lion, panthers, …)
• improve performance of blurry images

Create spreadsheet:

Image	Dog	Great cats	Bluring
1	ok
2			ok
3		ok	ok
…
% of total	8%	43%	61%

You sometime notice other dimensions like instagram filters…

Could easily know where you should improve.

Cleaning up incorrectly labeled dataset

Incorrectly labeled examples

If you have incorrectly labeled data. First lets consider the training set.

So long as you don't have too much errors, DL is quite robust to random errors.

But this is a problem for systematic errors.

Error analysis

Image	Dog	Great cats	Bluring	Comments
…
98	ok			labeler missed cat in background
99			ok
100		ok	ok	drawing of a cat not a real cat
% of total	8%	43%	61%

1st case:

Overall dev set error: 10% Error due incorrect labels: 0.6% Errors due to other causes: 9.4%

2nd case:

Overall dev set error: 2% Error due incorrect labels: 0.6% Errors due to other causes: 1.4%

In 2nd case, take the time to fix mislabeled examples.

Correctin incorrect dev/test set examples

• Apply same process to your dev and test sets to make sure they continue to come from the same distribution.
• Consider examining examples your algorithm gor right as well as ones it got wrong.
• Train and dev/test data may now crom from slightly different distributions

Buid your first system quickly then iterate

Speech recognition example

• noisy background

  • café noise
  • car noise
• Accented speech
• Far from microphone
• young children's speech
• stuttering, uh, ah, um…

50 directions you could go, on which should you focus on?

1. Set up dev/test set and metric
2. Build initial system quickly
3. Use Bias/Variance analysis & Error Analysis to prioritize next steps

Guideline: Build your first system quickly then iterate

Do not otherthink, build something quick and dirty first.

Mismatched training and dev/test set

Training and testing on different distributions

Cat app example

Two sources of data:

• data from webpages
• data from mobile app

Let's say you don't have lot of users (~10k from mobile, 200k from web)

You care about doing well on mobile images. You don't want to use only the 10k, but the dilema is the 200k aren't from the same distribution.

Option 1: take the 210k images and split between train/dev/test (train 205k, 2.5k, 2.5k)

• avantage, same distribution
• disavantage, perform on web instead of web.
• only 119 other the 2.5k will be from mobile.

Option 1 not recommended

Option 2:

• train set have 200k images from the web and 5k from the mobile.
• dev and test all mobile app images.
• avantage you know aiming your target where you want it to be.
• disavantage, your training distribution is different

But other the long term it will get you better performance

Speech recognition example

• Speech artificial rearview mirror. (real product in China)

1. Training: take all the speech data you have; purshased data, smart speaker control, voice keyboard… (500k)
2. Dev/test: speech activated, rearview mirror (20k)

Set your training set to be 500k from 1. and Dev/Test from 2.

The training set could be 510k (500k from 1 and 10k from 2.) and Dev/Test set (5k+5k from the rest of 2.)

Much bigger training set.

Bias and Variance with mismatched data distribution

Cat classifier example

Assume humans get ~0% error.

Training error	1%
Dev error	10%

Maybe there isn't a variance pb as the distribution is different.

Training-dev set: Same distrib as training set but not used for training.

Train / dev / test ==> Train split in train-2 and train-dev

So now you learn only on train-2 and check on train-dev and dev and test.

Train err%	1%	1%
Train-dev err%	9%	1.5%
dev err%	10%	10%
	Var pb	data mismatch pb

Other examples:

Human err%	0%	0%
Train err%	10%	10%
Train-dev err%	11%	11%
dev err%	12%	20%
	Bias pb	Bias + data mismatch pb

Bias/variance on mismatched trainig and dev/test sets

Human level

4%

avoidable bias

Training set error

7%

variance

Training-dev set error

10%

data mismatch

Dev error

12%

degree of overfitting to dev set

Test error

12%

Example, training is much harder than dev/test set distribution:

Human level	4%
Training set error	7%
Training-dev set error	10%
Dev error	6%
Test error	6%

More general formulation

The numbers can be place onto a table:

	General Speech rec tasks	Rearview mirror speech data
Human lvl	"Human level err" (4%)	6%
err on trained on	"Training err" (7%)	6%
err not trained on	"Training-dev err" (10%)	"Dev/Test err" (6%)

Addressing data mismatch

There are not any systematic way to address that. But there are things you can try.

Addressing data mismatch

• Carry out manual error analysis to try to understand difference between training and dev/test sets. ex: you might find that a lot of dev set is noisy (car noise)
• Make training data more similar, or collect more data similar to dev/test sets. ex: simulate noisy in-car data.

Artificial data synthesis

• Clean + car noise = synthetized in-car audio

Create more data, and can be a reasonable process.

Let's say you have 10k hrs of sound and only 1hr of car noise.

There is a risk your algorithm will overfit your 1hr car noise.

Artificial data synthesis (2)

Car recognition

Using car generated by computer vs just photos. You might overfit generated cars. A video game might have only 20 cars, so overfit these 20 cars.

Learning from multiple tasks

Transfer learning

Learning recognize cats to help to read x-ray scans.

Transfer learning

Create new NN by changing just the last layer (the output).

(X,Y) now become (radiology images, diagnosis)

retrain the W^[Z], b^[Z].

You might want to train just the last layer, you all the layers.

The rule of thumb, just the last layer on few data. The rule of thumb, all the layer on lot of datas.

pre-training, and fine-tuning.

A lot of low-level features learning from a very large data set might help.

• Another example. Speech recognition system:

X (audio) y (speech recognition) (wakeword, trigger word (ok google, hey siri, etc…))

You could add several new layers, and retrain the new layers or even more layers.

It make sense to transfer make sense when you have a very different number of examples.

• 10^6 image recognition, but only 100 radiology data.
• 10k hrs sounds, but only 1h data for wake words…

Transfering from lot of data to small number of data.

It doesn't make sense to transfer the other way.

When transfer learning makes sense

Task from A to B

• Task A and B have the same input X
• You have a lot more data for Task A than Task B
• Low level features from A could be helpful for learning B

Multi-task learning

Simultaneously learn multiple tasks.

Simplified autonomous driving example

	y^(i)	(4,1)
pedestrians	0
cars	1
stop signs	1
traffic lights	0

Y = [ y^(1) y^(2) …. y^(m) ]

Neural network architecture

x -> [] -> [] …. -> ^y in R^4

Loss: y(i) -> 1/m ∑_i=1^m ∑_j=1^4 (L(y^(i)_j , y^(i)_j))

L is the usual loss function.

Unlike softmax regression, one image can have multiple labels.

• One NN doing 4 things is better than learning 4 different NN for each task.

Some examples might not be fully labelled. And you can train by summing only other 0/1 label and not on ? mark (un labeled values).

So you can use more informations.

When multi-task learning makes sens.

• Training on set of tasks taht could benefit from having shared lower-level features
• Usually: amount of data you have for each task is quite similar
• Can train a big enough neural network to do well on all the tasks

Multi-task learning used a lot more than transfer learning.

End-to-end deep learning

What is end-to-end deep learning?

What is end-to-end deep learning?

Speech recognition example

audio - MFCC -> features – ML –> phonemes -> words -> transcript

audio ————————————————> transcript

You might need a lot of data. 3k hrs of data, classical approach better. 10k to 100k hurs then end-to-end approach generally shines.

Face recognition

Multi state approach works better:

1. detect face, zoom-in and crop to center the face
2. then feed this croped image to find identity. Generally comparing to all employes.

Why?

• Have a lot of data for task 1
• Have a lot of data for task 2

If you were to try to learn everything at the same time you wouldn't have enough data.

More examples

Machine translation:

English -> text analysis -> … -> French English ————————-> French

Because we have lot of (x,y) examples.

Estimating child's age from scan of the hand:

Image -> bones -> age Image ———-> age (there is not enough data)

Whether to use end-to-end deep learning

Pros and cons of end-to-end learning

Pros:

• let the data speak (no human preconception)
• Less hand-designing of components needed

Cons:

• May need a large amount of data: input end —-> output end (x,y)
• Excludes potentially useful hand-designed components. Data, Hand-design

Applying end-to-end deep learning

Key question: do you have sufficient data to learn a function of the complexity needed to map x to y?

• choose X->Y mapping
• pure deep learning approch not appropriate if hard to find end-to-end exmaples.

Convolutional Neural Networks

Week 1

Computer Vision

size: 64x64x3 -> 12288 size: 1000x1000x3 -> 3 millions

[x_1 … x_3millions] and w^[1] -> [1000 x 3e6]

Edge Detection Example

Take the 6x6 image and "convolve it by the 3x3 matrix" filter: /yogsototh/deft/src/commit/75f6ba865dede2e4fe908010fed7c6a99259608b/1,1,1],[0,0,0],[-1,-1,-1

python: conv_forward tensorflow. tf.nn.conv2d keras: conv2D

More edge detection

Sobel filter: 1,2,1 , 0,0,0, -1,-2,-1 Scharr filter: 3,10,3 , 0,0,0, -3,-10,-3

Padding

Size of image shrink because of borders. If filter as size f and image size n -> final image after filter: n - f +1

first solution but a border around the image: Padding

"valid" : nxn * fxf -> n-f+1 x n-f+1 "same": Pad so the output size is the same as the input size n + 2p -f + 1 => p = (f-1)/2

3x3 -> p = 3-1/2 = 1 5x5 -> p = 5-1/2 = 2

• f is usually odd, easier for padding + the filter has a central position.
• 1x1, 3x3, 5x5, 7x7.

Strided Convolutions

Jump some columns/lines. Instead of sliding the filter on every columns/row, do it every n columns/ n rows.

nxn * fxf, padding: p, stride: s

floor ((n + 2p -f / s) + 1) x floor ((n + 2p -f / s) + 1)

Summary of convolutions

nxn image fxf filter

padding p stride s

output size: \[ \floor ((n + 2p -f / s) + 1) x \floor ((n + 2p -f / s) + 1) \]

Convolution in math textbook (flip vertical and horizontal the filter)

cross-correlation vs convolution

By convention we call cross-correlation, convolution operator.

The convolution op is cross-associative: (A * B) * C = A * (B * C)

Convolution over volumes

6x6x3 * 3x3x3 -> 4x4 height x width x #channels

By convention the nb of channels will be same in the image and in the filter.

Multiple filters

6x6x3 * 3x3x3 —\ 4x4

• 3x3x3 —/ 4x4 ====> 4x4x2, 2, n_c = #filters

One layer of convolutional neural network

10 filters that are 3x3x3 in one layer of NN, how many parameters?

3x3x3 = 27 + bias = 28 params 28 x 10 = 280 params

Summary of notation

If layer l is a convolutional layer: f^[l] = filter size p^[l] = padding s^[l] = stride nc^[l] = number of filters

each filter is f^[l] x f^[l] x n_c^[l-1] Activations: a^[l] -> n_H^[l] x n_W[l] x n_c^[l] A^[l] -> m x n_H^[l] x n_w^[l] x n_c^[l] Weights: f^[l] x f^[l] x n_c^[l-1] x n_c^[l] (n_c^[l]: #filters in layer l) Bias: n_c^[l] - (1,1,1,n_c^[l])

Input: n_H^[l-1] x n_W^[l-1] x n_c^[l-1] Output: n_H^[l] x n_W^[l] x n_c^[l]

n^[l] = floor ( n^[l-1] + 2p^[l] - f^[l] / s^[l]) +1

A simple convolution neural network example

_	/ —————————->

39x39x3 f[1]=3, s^[1]=1, p^[1]= 0 n_H^[0] = n_W^[0] = 39 10 filters n_c^[0]=3

_	/ —————————->

37x37x10 f[2]=5, s^[1]=2, p^[1]=0 20 filters

_	/ —————————->

17x17x20 f[3]=5, s^[1]=2, p^[1]=0 40 filters

_	/ —- 1960 params –> softmax y^hat

7x7x40

Type of layer in a CNN

• Convolution (CONV)
• Pooling (POOL)
• Fully connected (FC)

Pooling Layer

Max pooling

4x4 — max over 2x2 region –> 2x2

Hyperparameters: f=2, s=2 No parameters to learn!

In practice it works well.

Example

5x5 with f=3 s=1

1 3 2 1 3 2 9 1 1 5 9 9 5 1 3 2 3 2 ====> 9 9 5 8 3 5 1 0 8 6 9 5 6 1 2 9

Overs #channels, the output has the same number of channels. Max pooling over each channel independently

Average Pooling

Same as previous, but we take the average instead of the max

Summarize

Hyperparameters

• f: filter size
• s: stride
• Max or Average pooling
• p: padding (almost never used, p=0 in general)

nH x nW x nc —-> ((nH - f / s) + 1) x ((nW - f / s) + 1) + n_c

CNN Example

• Input Image: 32x32x3 (try to recognize a 7 letter) — conv f=5, s=1 —->
• Conv1: 28x28x6 —- max pool, f=2, s=2 —-> POOL1 14x14x6 } LAYER 1 — conv f=5 s=1 –>
• Conv2: 10x10x16 —- max pool, f=2, s=2 —-> POOL2 5x5x16 } LAYER 2

400 —-> FC3 120 —> FC4 84 —-> 0 softmax (10 outputs) W^[3] (120,400) b^[3] (120)

In general:

• n_H, n_W will decrease as we go deeper
• n_c will increase as we go deeper
• CONV - POOL - CONV - POOL - FC -FC -FC -SOFTMAX

Sizes

Activation size go down, # parameters, few for Conv and 0 for POOL, a lot in FC

Why convolutions?

Two advantages:

• parameter sharing and sparsity of connections.

If we had to make a FC between a 32x32x3 –> 28x28x6, would need millions of parameters. Conv as only 156 parameters.

• Parameter sharing: A feature detector that's useful in one part of the image is probably useful in another part of the image.
• Sparsity of connections: In each layer, each output value depends only on a small number of inputs.

Week 2

Case studies

Why look at case studies?

Outline

Classic networks:

• LeNet-5
• AlexNet
• VGG

ResNet (residual network), 152 deep network

Inception

Classic Networks

LeNet - 5

Recognize handwritten digits, 32x32x1 — conv 5x5 s = 1 —> 28x28x6 — avg pool, f=2, s=2 –> 14x14x6 — conv 5x5,s=1 –> 10x10x16 — avg pool, f=2 s=2 —> 5x5x16 (400) –> FC 120 –> FC 84 –> Softmax y^

Size was 60k parameters. Today hunder millions parameters

n_H, n_W decrease, n_C increase

Conv, pool, Conv, pool, fc, fc, output

LeCun et al. 1998, Gradient-based learning applied to document recognition.

AlexNet

Alex Krizhevsky et al. 2012, ImageNet classification with deep convolutional neural networks.

227x227x3 — conv 11x11, s=4 —> 55x55x96 — MAX Pool, 3x3, s=2 —-> 27x27x96 — 5x5 same ——> 27x27x256 — MAX POOL, 3x3, s=2 —-> 13x13x256 — 3x3, same —–> 13x13x384 — 3x3 —> 13x13x384 —3x3 —> 13x13x256 — MAX POOL, 3x3, s=2 —> 6x6x256 — FC 9216 –> FC 4096 –> FC 4096 –> Softmax

Similar to previous but MUCH bigger:

60 millions parameters

Also use ReLU

VGG - 16

CONV = 3x3 filter, s=1, same MAX-POOL = 2x2, s=2

224x224x3 — [CONV 64]x2 —> 225x224x64 — POOL —> 112x112x64 — [CONV 128]x2 —> 112x112x128 — POOL —> 56x56x128 — [CONV 256]x3 —> 56x56x256 — POOL —> 28x28x256 — [CONV 512]x3 —> 28x28x512 — POOL —> 14x14x512 — [CONV 512]x3 —> 14x14x512 — POOL —> 7x7x512 —> FC 4096 –> FC 4096 —> Sofmax 1000

Simonyan & Zisserman 2015. Very deep convolutional networks for large-scale image recognition.

about 138 millions parameters

Also VGG-19 is also another even bigger network, but VGG-16 perform as good as VGG-19

Residual Networks (ResNets)

Very very deep neural, over 100 layers.

Residual block

a^[l] —> a^[l+1] –> a^[l+2]

a^[l] –+–> linear –> ReLU –> a^[l+1] —–> linear —–> ReLU –> a^[l+2]

————————————————- shortcut (skip connection)

a^[l+2] = g (z^[l+2] + a^[l])

passes information deeper in the NN.

He et al, 2015, Deep residual networks for image recognition

x -> [] -> [] -> [] …. -> a^[l]

Help in vanishing gradient descent parameters.

Why ResNets Work

Why do residual networks work so well?

In practice, learning on too deep network make bad results on the training set. So that prevent using network that are too deep.

But it much less true when learning ResNets.

X —> Big NN —> a^[l]

X —-> Big NN —> a^[l] --> [] –> [] -->a ^[l+2] ReLU a >= 0 | | ————–

a^[l+2] = g (z^[l+2] + a^[l]) = g ( W^[l+2]a^[l+1] + b^[l+2] + a^[l] )

If we're using L2 regularization that would shrink the value of W^[l+2], also b^[l+2] If W^[l+2] =0 and also b^[l+2], then we'll have g(a^[l] which is equal to a^[l]

So the identity function is easy to learn for ReLU. So adding the shortcut don't hurt performance. But if its goes good, it's better but almost never worse.

Remark: We're assuming z^[l+2] and a^[l] is of the same dimension. In case they have different dimensions, add an extra matrix W_s so we have:

(W_s x a^[l])

ResNet example

Plain —-> ResNet

Networks in Networks and 1x1 Convolutions

Using a 1x1 convolution.

What does a 1x1 convolution do?

6x6x1 * 2 —> simple multiplication

6x6x32 x 1x1x32 —> 6x6x#filters

• Lin et al. 2013, Network in Network.

Using 1x1 conv

28x28x192 —- ReLU, CONV 1x1, 32 filters —> 28x28x32

Let shrink n_C as well

Effect non-linearity, we could keep the nb of layers and its fine.

1x1 conv does a non trivial operation.

Inception Network Motivation

We might have to pick, conv 3x3, pool, layer, etc…

Motivation for inception network.

Do all transformations at the same time:

28x28x192 — 1x1 —-> 28x28x64 \\\— 3x3,same —-> 28x28x128 \\— 5x5,same —-> 28x28x32 \— max pool, same, s=1 —-> 28x28x32

---

28x28x256

Szegedy et al. 2014, Going deeper with convolutions.

pb computational cost.

The problem of computational cost

28x28x192 — conv 5x5, same, 32 —-> 28x28x32

32 filters, filters are 5x5x192

Nb of multiplications: 28x28x32 x 5x5x192 = 120 millions (costly)

Using 1x1 conv

28x28x192 ——- conv 1x1, 16, 1x1x192 —> 28x28x16 ——– conv 5x5, 32, 5x5x16 —> 28x28x32 Bottleneck Layer

cost of 1st conv layer: 28x28x16 x 192 = 2.4 millions cost of 2nd conv layer: 28x28x32 x 5x5x16 = 10 millions total cost: 12.4 millions (about 10x less than before)

You can reduce substantially the size without hurting performace of the NN while improving performances.

Inception Network

Inception module

Previous Activation: 28x28x192

1x1 conv ——————————————–> 28x28x64 -\ 1x1 conv —————-> 3x3 conv —————-> 28x28x128 – Channel concat: 1x1 conv —————-> 5x5 conv —————-> 28x28x32 // 28x28x256 MAXPOOL,3x3,s=1, same –> 28x28x192 –> 1x1 CONV —> 28x28x32 /

Inception network, si the same pattern (block) connected in many layers.

Inception-block –> Inception block —> ….. —> Softmax layer output \-> softmax layer \-> softmax layer

GoogleNet

Fun fact

We need to go deeper (from the Inception meme)

Since the dev, there are newer versions of inception modules. Inception v1, v2, v3 ….

Practical advice for using ConvNet

Using Open-Source Implementation

Lot of these networks are difficult to reproduce, replicate the work.

Search look online implementation, instead of re-implementing from scratch.

Demo:

1. google
2. github repository
3. check license (MIT for example)
4. git clone …

Also advantage, pre-trained networks. Starting from open-source implementation is faster.

Transfer Learning

There are lot of data set on the Internet. You could often download have an initial pretrained NN.

Transfer Learning

Classification Problem: is it Tigger, Misty or Neither (recognize cats)

Trained network trained on ImageNet.

Get rid of the soft-max layer and create your own, and output, Tigger/Misty or Neither. And freeze the other parameters. Just learn the Softmax layer.

You might get pretty good results.

Depending on the framework:

• TrainableParameters = 0
• freeze = 1

Pre-compute and save the last layer using for activation. You don't need to recompute these activations.

If you have a larger dataset:

Freeze, fewer layers (the firsts) Or freese a few layers and create a new layers with your own architecture.

If you have a lot of data:

Use the hole thing and use it as initialization.

You should almost always do transer learning unless you have an exceptionally large dataset to train.

Data Augmentation

For the majority of Computer vision problem, having more data is almost always useful and help.

Common augmentation

• mirroring images
• Random cropping, not perfect, but work well in practice
• Also: Rotation, shearing, local warping… but not much used in practice

Color shifting

Add to RGB different distortions. Ex: +20,-20,+20 —> more mauve etc…

Advanced: PCA color augmentation (in AlexNet paper)

Implementing distortions during training

Harddisk —> CPU threadd —> distortions —-> training (CPU/GPU) \– load -> color minibatch

Meta parameters, so certainly use open-source implementation to data augmentation.

State of Computer Vision

Data vs hand-engineered

Most ML problem.

Little data <——————————————–> Lots of data

speach recognition: lot of data image recognition: OK data Object detection: less data

Lot of data: simpler algorithms, less hand-engineering. Few data: more hand-engineering, hacks

Two sources of knowledge:

• Labeled dataset (x,y)
• Hand engineering features/network architecture/other components

When very few data: Transfer learning.

Tips for doing well on benchmarks/winning competitions

• Ensembling

  • Train several networks independently and average their outputs (not their weights)
  • 3/15 networks (but almost never used in production, because it is costly for few benefits)

• Multi-crop at test time

  • run classifier on multiple versions of test images and average results
  • 10-crop: central + 4 corner + same on mirrored

Do not do this in production systems.

Use open source code

• Use architectures of networks published in the literature
• Use open source implementations if possible
• Use pretrained models and fine-tune on your dataset