1372 lines
40 KiB
Org Mode
1372 lines
40 KiB
Org Mode
-- #+TITLE: Deep Learning Coursera
|
||
-- #+AUTHOR: Yann Esposito
|
||
#+STARTUP: latexpreview
|
||
#+TODO: TODO IN-PROGRESS WAIT | DONE CANCELED
|
||
#+COLUMNS: %TODO %3PRIORITY %40ITEM(Task) %17EFFORT(Estimated Effort){:} %CLOCKSUM %8TAGS(TAG)
|
||
|
||
* Plan
|
||
|
||
5 courses
|
||
|
||
** Neural Network and Deep Learning
|
||
*** Week 1: Introduction
|
||
*** Week 2: Basic of Neural Network programming
|
||
*** Week 3: One hidden layer Neural Networks
|
||
*** Week 4: Deep Neural Network
|
||
** Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization
|
||
** Structuring your Machine Learning project
|
||
** Convolutional Neural Networks
|
||
** Natural Language Processing: Building sequence models
|
||
* DONE Neural Network and Deep Learning
|
||
CLOSED: [2017-08-22 Tue 13:43]
|
||
** Introduction
|
||
|
||
*** What is a neural network?
|
||
|
||
*** Supervised Learning with Neural Networks
|
||
|
||
- Lucrative application: ads, showing the add you're most likely to click on
|
||
- Photo tagging
|
||
- Speech recognition
|
||
- Machine translation
|
||
- Autonomous driving
|
||
|
||
|
||
***** Convolutional NN good for images
|
||
|
||
***** Strutured data (db of data) vs Unstructured data
|
||
|
||
- Structured data: Tables
|
||
- Unstructured data: Audio, image, text...
|
||
|
||
Computer are much better at interpreting unstructured data.
|
||
|
||
*** Why is Deep Learning taking off?
|
||
|
||
[[///Users/yaesposi/Library/Mobile%20Documents/com~apple~CloudDocs/deft/img/Scale%20drives%20deep%20learning%20progress.png]]
|
||
|
||
- Data (lot of data)
|
||
- Computation (faster learning loop)
|
||
- Algorithms (ex, use ReLU instead of sigma)
|
||
** Geoffrey Hinton interview
|
||
** Binary Classification
|
||
|
||
\[ (x,y) x\in \mathbb{R}^{n_x}, y \in {0,1} \]
|
||
|
||
$m$ training examples: $$ {(x^{(1)},y^{(1)}), ... (x^{(m)},y^{(m)})} $$
|
||
|
||
$$ m = m_{train} , m_{test} = #test examples $$
|
||
|
||
$$ X = [ X^{(1)} ... X^{(m)} ] is an n_x x m matrix $$
|
||
$$ X.shape (n_x,m) $$
|
||
|
||
$$ Y = [ y^{(1)} ... y^{(m)} ] $$
|
||
$$ Y.shape = (1,m) $$
|
||
|
||
** Logistic Regression
|
||
|
||
Given $X \in \mathbb{R}^{n_x}$ you want $\hat{y} = P(y=1 | X)$
|
||
|
||
Paramters: $w \in \mathbb{R}^{n_x}, b\in \mathbb{R}$
|
||
|
||
Output: $\hat{y} = \sigma(w^Tx + b) = \sigma(z)$
|
||
|
||
$$\sigma(z)= \frac{1}{1 + e^{-z}}$$
|
||
|
||
If $z \rightarrow \infty => \sigma(z) \approx 1$
|
||
If $z \rightarrow - \infty => \sigma(z) \approx 0$
|
||
|
||
|
||
Alternative notation not used in this course:
|
||
|
||
$X_0=1, x\in\mathbb{R}^{n_x+1}$
|
||
$\hat{y} = \sigma(\Theta^Tx)$
|
||
...
|
||
|
||
** Logistic Regression Cost Function
|
||
|
||
Search a convex loss function:
|
||
|
||
$L(\hat{y},y) = - (y\log(\hat{y}) + (1-y)\log(1-\hat{y}))$
|
||
|
||
If y = 1 : $L(\hat{y},y) = -\log\hat{y}$ <- want log\haty larg, want \hat{y} large
|
||
If y = 0 : $L(\hat{y},y) = -\log\hat{y}$ <- want log (1-\hat{y}) large, want \hat{y} sall
|
||
|
||
Cost function: $$ J(w,b) = \frac{1}{m}\sum_{i=1}^mL(\hat{y^\{(i)}},y^{(i)}) = ... $$
|
||
|
||
** Gradient Descent
|
||
|
||
Minize $J(w,b)$
|
||
|
||
1. initialize w,b (generaly uses zero)
|
||
2. Take a step in the steepest descent direction
|
||
3. repeat 2 until reaching global optimum
|
||
|
||
Repeat {
|
||
$w := w - \alpha\frac{dJ(w)}{dw} = w - \alpha\mathtext{dw}$
|
||
}
|
||
|
||
** Derivatives
|
||
** More Derivative Examples
|
||
** Computaion Graph
|
||
** Computing Derivatives
|
||
** Computing Derivatives for multiple examples
|
||
** Vectorization
|
||
getting rid of explicit for loops in your code
|
||
** Vectorizing Logistic Regression
|
||
** Vectorizing Logistic Regression's Gradient Computation
|
||
** Broadcasting in Python
|
||
** Quick Tour of Jupyter / ipython notebooks
|
||
** Neural Network Basics
|
||
J = a*b + a*c - (b+c) = a (b + c) - (b + c) = (a - 1) (b + c)
|
||
* DONE Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization
|
||
CLOSED: [2017-09-01 Fri 09:52]
|
||
** DONE Week 1: Setting up your Machine
|
||
CLOSED: [2017-08-22 Tue 13:43]
|
||
*** Recipe
|
||
|
||
If *High bias*? (bad training set performance?)
|
||
Then try:
|
||
- Bigger network
|
||
- Training longer
|
||
- (NN architecture search)
|
||
Else if *High variance*? (bad dev set performance?)
|
||
Then try:
|
||
- More data
|
||
- Regularization
|
||
- (NN architecture search)
|
||
|
||
Deep learning, not much bias/variance tradeoff if we have a big amount of
|
||
computer power (bigger network) and lot of data.
|
||
*** Regularization
|
||
**** Regularization: reduce variance
|
||
- L2 regularization
|
||
|
||
λ / 2m || w ||_2 ^2
|
||
|
||
- L1 regularization: same with |w| instead of ||w||_2^2
|
||
|
||
|
||
λ is a regularization parameter (in code named =lambd=)
|
||
|
||
Cost = J(w^[1], b^[1], ..., w^[L], b^[L]) = 1/m \sum L(^y(i), y(i)) + λ/2m \sum_l=1^L || W^[l] ||^2
|
||
|
||
|
||
call the "Frobenius norm"
|
||
|
||
dW = from backprop + λ/m W^l
|
||
|
||
update W^l = W^l - αdW^l still works
|
||
|
||
Sometime L2 regularization called "weight decay".
|
||
|
||
**** Dropout Regularization
|
||
|
||
Eliminates nodes by layer randomly for each training example.
|
||
|
||
- implementing, (inverted dropout)
|
||
- gen random boolean vector:
|
||
d3 = np.random.rand(a3.shape[0], a3.shape[1]) < keep_prob # (for each iteration)
|
||
a3 = np.mulitply(a3,d3)
|
||
a3 /= keep_prob (for normalization to be certain the a3 output still the same, reduce testing problems)
|
||
|
||
Making prediction at test time: no drop out
|
||
|
||
**** Over regularization methods
|
||
|
||
- Data augmentation, (flipping images for example, random crops, random distortions, etc...)
|
||
- Early stopping, stop earlier iteration
|
||
|
||
*** Setting up your optimization problem
|
||
|
||
**** Normalizing Inputs
|
||
|
||
- μ = 1/m Sum X^(i)
|
||
- x := x - μ (centralize)
|
||
- σ = 1/m Sum X^(i)^2
|
||
- x /= σ^2
|
||
|
||
|
||
**** Gradient Checking
|
||
***** Don't use gard check in traingin, only in debug
|
||
***** If algorithm fail, grad check, look at component (is db? dW? dW on certain layer, etc...)
|
||
***** Remember regularization
|
||
***** Doesn't work with dropout, turn off drop out (put 1.0) then check
|
||
***** Run at random initialization; perhaps again after training
|
||
|
||
** DONE Week 2: Optimization Algorithms
|
||
CLOSED: [2017-08-22 Tue 13:43]
|
||
*** Mini batch
|
||
|
||
X :: X^(1) ... X^(m)
|
||
|
||
X,Y -> X^{i},Y^{i} where X^{i} = X^(i*batch-size ---> (i+1)*batch-size)
|
||
|
||
*** Minibatch size
|
||
|
||
- if mini batch size = m => Batch gradient descent (X^{1},Y^{1}) = (X,Y)
|
||
- if mini match size = 1 => Stochastic gradient descent, every example is its own mini batch.
|
||
- in practice in between 1 and m, m --> too long, 1 loose speedup from vectorization.
|
||
+ vectorization ~1000
|
||
|
||
1. If small training set, use batch gradient descent (m <= 2000)
|
||
2. Typical mini-batch size: 64, 128, 256, 512, ... 2^k to fits in CPU/GPU memory
|
||
|
||
*** Exponentially weighted average
|
||
|
||
v_t = βv_{t-1} + (1-β)θ_t
|
||
** DONE Week 3: Hyperparameter
|
||
CLOSED: [2017-09-01 Fri 09:52]
|
||
*** Video 1: use random not a grid to search for hyperparameter best value
|
||
*** Video 2: choose appropriate scale to pick hyperparameter
|
||
- uniformly random n^[l] (number of neuron for layer l) or L (number of layers)
|
||
- alpha: between 0.00001 to 1, then shouldn't use linear but instead use log-scale
|
||
r = -4*np.random.rand() <- r in [-4,0]
|
||
α = 10^r <- 10^-4 ... 10^0
|
||
|
||
- β <- 0.9 ... 0.999 (0.9 about avg on 10 values, 0.999 avg about 1000 values)
|
||
1-β = 0.1 .... 0.001
|
||
r <- [-3,-1]
|
||
1-β = 10^r
|
||
*** Hyperparameter: Tuning in practice Panda vs caviar
|
||
- Babysitting one model (panda) for few computer resources
|
||
- Training many models in parallel (caviar) for lot of computer resources
|
||
*** Batch normalization
|
||
**** In a network
|
||
**** Fitting Batch norm into a deep network
|
||
**** Why Batch Normalizing?
|
||
- don't use batch norm as a regularization even if sometime it could have this
|
||
effect
|
||
**** Batch Norm at test time
|
||
μ = 1/m \sum z^(i)
|
||
|
||
σ^2 = 1/m \sum (z^(i) - μ)^2
|
||
|
||
z^(i)_norm = z^(i) - μ / sqrt( σ^2 + ε )
|
||
|
||
~z^(i) = γz^(i)_norm + β
|
||
|
||
Estimate μ and σ with exponentially weighted avg accross minibatches
|
||
*** Multi-class classification
|
||
**** Softmax Regression
|
||
notation: C = #classes (0,1,2...,C-1)
|
||
|
||
last hidden layer nb of neuron is equal to C: n^L = C
|
||
|
||
z^[L] = w[L]a^[L-1] + b[L] (C,1)
|
||
|
||
Activation function:
|
||
|
||
t = e^(Z[L])
|
||
a^[L] = e^(Z[L])/\sum_i=0^C t_i
|
||
|
||
a^[L]_i = t_i / \sum_i=0^C t_i
|
||
|
||
**** Training a softmax classifier
|
||
*** Introduction to programming frameworks
|
||
|
||
**** Deep learning frameworks
|
||
* Structuring your Machine Learning project
|
||
** Week 1
|
||
*** Introduction to ML Strategy
|
||
**** Why ML Strategy
|
||
Try to find quick and effective way to choose a strategy
|
||
|
||
Ways of analyzing ML problems
|
||
|
||
**** Orthogonalization
|
||
|
||
***** Chain of assumptions in ML
|
||
|
||
- Fit training set well on cost function => bigger network, Adam, ...
|
||
- Fit dev set well on cost function => Regularization, Bigger training set
|
||
- Fit test set well on cost function => Bigger dev set
|
||
- Perform well in real world => Change the devset or cost function
|
||
|
||
Try not to use early stoping as it simulanously affect cost on training and dev set.
|
||
|
||
*** Setting up your goal
|
||
|
||
**** Single number evaluation metric
|
||
|
||
***** First
|
||
|
||
| Classifier | Precision | Recall |
|
||
|------------+-----------+--------|
|
||
| A | 95% | 90% |
|
||
| B | 98% | 85% |
|
||
|
||
Rather than using two number, find a new evaluation metric
|
||
|
||
|
||
| Classifier | Precision | Recall | F1 Score |
|
||
|------------+-----------+--------+----------|
|
||
| A | 95% | 90% | 92.4% |
|
||
| B | 98% | 85% | 91.0 |
|
||
|
||
F1 score = 2 / (1/p) + (1/R) :: "Harmonic mean" of precision and recall.
|
||
|
||
So:
|
||
|
||
Having a good Dev set + single evaluation metric, really speed up iterating.
|
||
|
||
***** Another example
|
||
|
||
| Algorithm | US | China | India | Other | *Average* |
|
||
|-----------+-----+-------+-------+-------+-----------|
|
||
| A | 3% | 7% | 5% | 9% | |
|
||
| ... | | | | | |
|
||
| F | ... | ... | | | |
|
||
|
||
Try to improve the average.
|
||
|
||
**** Satisficing and Optimizing metric
|
||
|
||
It's not alway easy to select on metric to optimize.
|
||
|
||
***** Another cat classification example
|
||
|
||
| Classifier | Accuracy | Running Time |
|
||
|------------+----------+--------------|
|
||
| A | 90% | 80ms |
|
||
| B | 92% | 95ms |
|
||
| C | 95% | 1500ms |
|
||
|
||
cost = accuracy - 0.5x running time
|
||
|
||
maximize accuracy s.t. running time < 100ms
|
||
|
||
Accuracy <- Optimizing
|
||
Running time <- Satisficing
|
||
|
||
If you have n metrics, pick one to optimizing, and all the other be satisficing.
|
||
|
||
**** Train/dev/test distribution
|
||
|
||
How you can setup these dataset to speed up your work.
|
||
|
||
***** Cat classification dev/test sets
|
||
|
||
Try to find a way that dev and test set come from the same distribution.
|
||
|
||
***** True story (detail changed)
|
||
|
||
Optimizing on dev set on load approvals for medium income zip codes.
|
||
|
||
(repay loan?)
|
||
|
||
Tested on low income zip codes.
|
||
|
||
Lost 3 months
|
||
|
||
***** Guideline
|
||
|
||
Choose a dev set and test set to reflect data you expect to get in the future
|
||
and consider important to do well on.
|
||
|
||
**** Size of dev and test sets
|
||
|
||
***** Old way of splitting
|
||
70% train, 30% test
|
||
60% train, 20% dev, 20% test
|
||
|
||
For at max 10^4 examples
|
||
|
||
But in new era, 10^6 examples:
|
||
|
||
train: 98%, Dev 1%, Test 1%.
|
||
|
||
***** Size of test set
|
||
|
||
Set your test set to be big enough to give high confidence in the overall
|
||
performance of your system. Can be far less than 30% of your data.
|
||
|
||
For some applications, you don't need test set and only dev set.
|
||
For example if you have a very large dev set.
|
||
|
||
**** When to change dev/test sets and metrics?
|
||
|
||
Metric: classification error
|
||
Algorithm A: 3% error → letting throught a lot of porn images
|
||
Algorithm B: 5% error → doesn't let pass porn images
|
||
|
||
So your metric + evaluation prefer A, but you and your users prefer B.
|
||
|
||
When this happens, mispredict your algorithm B is better.
|
||
|
||
Error: 1/m_dev \sum_i=1^m I{y_pred^(i) /= y^(i)
|
||
|
||
They treat pron and non pron equaly but you don't want that.
|
||
|
||
We add a w(i) = 1 if non porn and 0 if porn in the formula
|
||
|
||
**** Orthogonalization for cat pictures: anti-pron
|
||
|
||
1. So far we've only discussed how to define a metric to evaluate classifier
|
||
2. Worry separately about how to do well on this metric
|
||
|
||
1. placing the target, and 2. is aiming the target.
|
||
|
||
**** Another example
|
||
Alg A: 3% err
|
||
Alg B: 5% err
|
||
|
||
But B does better. You see that users are using blurier images.
|
||
You dev/test are not using the same kind of images.
|
||
|
||
Change your metric and/or dev/test set.
|
||
|
||
*** Comparing to Humand-level performance
|
||
|
||
**** Why human-level performance
|
||
|
||
Human-level perf vs Bayes optimal error
|
||
|
||
Human are generally very close to bayes perf for lot of tasks.
|
||
|
||
- get lableld data from humans
|
||
- gain insight from manual error analysis (why did a person get this right?)
|
||
- better analysis of bias/variance
|
||
|
||
**** Avoidable bias
|
||
|
||
***** Cat classification example
|
||
|
||
| Humans | 1% | 7.5% |
|
||
| Training error | 8% | 8% |
|
||
| Dev error | 10% | 10% |
|
||
| | focus on bias | focus on variance |
|
||
|
||
Human level error as a proxy (estimate) for Bayes error.
|
||
|
||
*Diff between Human err and Training err = available bias*
|
||
*Diff between Train and Dev err = variance*
|
||
|
||
**** Understanding Human-level performance
|
||
|
||
***** Human-level error as proxy for Bayes error
|
||
Medical image classification example:
|
||
suppose
|
||
(a) Typical human 3% err
|
||
(b) Typical doctor 1% err
|
||
(c) Experienced doctor 0.7% err
|
||
(d) and team of experienced doctors 0.5% err
|
||
|
||
What is "human-level" error?
|
||
|
||
Bayes error is <= to 0.5% err
|
||
So we use that to aim as saw before.
|
||
|
||
For a paper, (b) is good enough to talk about that.
|
||
|
||
***** Error analysis example
|
||
|
||
| Human (proxy for bayes err) | 1, 0.7, 0.5% | 1, 0.7, 0.5 | 1, 0.7, 0.5 |
|
||
| Train err | 5% | 1% | 0.7% |
|
||
| Dev err | 6% | 5% | 0.8% |
|
||
| | | | |
|
||
|
||
Case 1:
|
||
For this example it doesn't matter because avoidable bias (5 - 1%), is bigger
|
||
than variance (6-5)
|
||
|
||
Case 2: focus on variance
|
||
|
||
Case 3, very important you use 0.5 as your "human-level" error. Because it show
|
||
that you should focus on bias and not on variance.
|
||
|
||
This problem arose only when you're doing very good.
|
||
|
||
***** Summary of bias/variance with human-level perf
|
||
|
||
Human-level error (proxy for Bayes err)
|
||
|
||
^
|
||
| "Avoidable bias"
|
||
v
|
||
|
||
Training error
|
||
|
||
^
|
||
| "Variance"
|
||
v
|
||
|
||
Dev error
|
||
|
||
**** Surpassing human-level performance
|
||
|
||
***** Surpassing human-level performance
|
||
|
||
| Team | 0.5% | 0.5% |
|
||
| One human | 1% | 1% |
|
||
| Training error | 0.6% | 0.3% |
|
||
| Dev error | 0.8% | 0.4% |
|
||
|-----------------+-------+------------|
|
||
| Avoidable bias? | ~0.5% | can't know |
|
||
|
||
***** Problems where ML significantly surpasses human-level performance
|
||
|
||
- Online advertising
|
||
- Product recommendations
|
||
- Logistics (predicting transit time)
|
||
- Loan approvals
|
||
|
||
all thoses examples:
|
||
+ come from structured data
|
||
+ not natural perception problems
|
||
+ Lots of data
|
||
|
||
|
||
Also, Speech recognition, Some image recognition, Medical, ECG, skin cancer,
|
||
etc...
|
||
|
||
**** Improving your model performance
|
||
|
||
Set of guidelines
|
||
|
||
***** The two fundamental assumptions of supervised learning
|
||
|
||
1. You can fit the training set pretty well (~ avoidable bias)
|
||
2. The training set performance generalizes pretty well to the dev/test set
|
||
|
||
***** Reducing (avoidable) bias and variance
|
||
|
||
Human-level error (proxy for Bayes err)
|
||
|
||
^
|
||
| train bigger model
|
||
| "Avoidable bias" => train longer/better optimization algorithms (momentum, RMSprop, Adam)
|
||
| NN architecture/hyperparameters search (RSS, CNN...)
|
||
v
|
||
|
||
Training error
|
||
|
||
^
|
||
| More data
|
||
| "variance" => Regulraization (L2, dropout, data augmentation)
|
||
| NN architecture/hyperparameters search
|
||
v
|
||
|
||
Dev error
|
||
|
||
These concepts are easy to learn, hard to master.
|
||
You'll be more systematics than most ML teams.
|
||
|
||
** Week 2
|
||
*** Error Analysis
|
||
**** Error Analysis
|
||
***** Carrying out error analysis
|
||
- Imagine your cat algo doesn't work as good as expected.
|
||
- One of your colaborator think you should focus on working on dogs.
|
||
- Anaylize manually 100 mislabeled dev set examples
|
||
- Count up how many are dogs
|
||
- Supose 5% are dogs. So at most you could go from 10% err to 9.5% so not much useful.
|
||
|
||
- Supose taht 50% of them are dogs error, so you could go down from 10% to 5%,
|
||
so you could be more confident.
|
||
***** Evaluate multiple idea in parallel
|
||
- fix pictures of dogs
|
||
- fix great cats (lion, panthers, ...)
|
||
- improve performance of blurry images
|
||
|
||
|
||
Create spreadsheet:
|
||
|
||
| Image | Dog | Great cats | Bluring |
|
||
| 1 | ok | | |
|
||
| 2 | | | ok |
|
||
| 3 | | ok | ok |
|
||
| ... | | | |
|
||
| % of total | 8% | 43% | 61% |
|
||
|
||
You sometime notice other dimensions like instagram filters...
|
||
|
||
Could easily know where you should improve.
|
||
**** Cleaning up incorrectly labeled dataset
|
||
***** Incorrectly labeled examples
|
||
If you have incorrectly labeled data.
|
||
First lets consider the training set.
|
||
|
||
So long as you don't have too much errors, DL is quite robust to random errors.
|
||
|
||
But this is a problem for systematic errors.
|
||
***** Error analysis
|
||
|
||
|
||
| Image | Dog | Great cats | Bluring | Comments |
|
||
| ... | | | | |
|
||
| 98 | ok | | | labeler missed cat in background |
|
||
| 99 | | | ok | |
|
||
| 100 | | ok | ok | drawing of a cat not a real cat |
|
||
| % of total | 8% | 43% | 61% | |
|
||
|
||
1st case:
|
||
|
||
Overall dev set error: 10%
|
||
Error due incorrect labels: 0.6%
|
||
Errors due to other causes: 9.4%
|
||
|
||
2nd case:
|
||
|
||
Overall dev set error: 2%
|
||
Error due incorrect labels: 0.6%
|
||
Errors due to other causes: 1.4%
|
||
|
||
In 2nd case, take the time to fix mislabeled examples.
|
||
***** Correctin incorrect dev/test set examples
|
||
|
||
- Apply same process to your dev and test sets to make sure they continue to
|
||
come from the same distribution.
|
||
- Consider examining examples your algorithm gor right as well as ones it got
|
||
wrong.
|
||
- Train and dev/test data may now crom from slightly different distributions
|
||
**** Buid your first system quickly then iterate
|
||
***** Speech recognition example
|
||
- noisy background
|
||
- café noise
|
||
- car noise
|
||
- Accented speech
|
||
- Far from microphone
|
||
- young children's speech
|
||
- stuttering, uh, ah, um...
|
||
|
||
50 directions you could go, on which should you focus on?
|
||
|
||
1. Set up dev/test set and metric
|
||
2. Build initial system quickly
|
||
3. Use Bias/Variance analysis & Error Analysis to prioritize next steps
|
||
|
||
Guideline: *Build your first system quickly then iterate*
|
||
|
||
Do not otherthink, build something quick and dirty first.
|
||
*** Mismatched training and dev/test set
|
||
**** Training and testing on different distributions
|
||
***** Cat app example
|
||
Two sources of data:
|
||
- data from webpages
|
||
- data from mobile app
|
||
|
||
Let's say you don't have lot of users (~10k from mobile, 200k from web)
|
||
|
||
You care about doing well on mobile images. You don't want to use only the 10k,
|
||
but the dilema is the 200k aren't from the same distribution.
|
||
|
||
Option 1: take the 210k images and split between train/dev/test (train 205k, 2.5k, 2.5k)
|
||
- avantage, same distribution
|
||
- disavantage, perform on web instead of web.
|
||
- only 119 other the 2.5k will be from mobile.
|
||
Option 1 not recommended
|
||
|
||
Option 2:
|
||
- train set have 200k images from the web and 5k from the mobile.
|
||
- dev and test all mobile app images.
|
||
- avantage you know aiming your target where you want it to be.
|
||
- disavantage, your training distribution is different
|
||
But other the long term it will get you better performance
|
||
***** Speech recognition example
|
||
- Speech artificial rearview mirror. (real product in China)
|
||
1. Training: take all the speech data you have; purshased data, smart speaker control, voice keyboard... (500k)
|
||
2. Dev/test: speech activated, rearview mirror (20k)
|
||
|
||
Set your training set to be 500k from 1. and Dev/Test from 2.
|
||
|
||
The training set could be 510k (500k from 1 and 10k from 2.) and Dev/Test set (5k+5k from the rest of 2.)
|
||
|
||
Much bigger training set.
|
||
**** Bias and Variance with mismatched data distribution
|
||
***** Cat classifier example
|
||
Assume humans get ~0% error.
|
||
|
||
| Training error | 1% |
|
||
| Dev error | 10% |
|
||
|
||
Maybe there isn't a variance pb as the distribution is different.
|
||
|
||
Training-dev set: Same distrib as training set but not used for training.
|
||
|
||
Train / dev / test ==> Train split in train-2 and train-dev
|
||
|
||
So now you learn only on train-2 and check on train-dev and dev and test.
|
||
|
||
| Train err% | 1% | 1% |
|
||
| Train-dev err% | 9% | 1.5% |
|
||
| dev err% | 10% | 10% |
|
||
| | Var pb | data mismatch pb |
|
||
|
||
Other examples:
|
||
|
||
| Human err% | 0% | 0% |
|
||
| Train err% | 10% | 10% |
|
||
| Train-dev err% | 11% | 11% |
|
||
| dev err% | 12% | 20% |
|
||
| | Bias pb | Bias + data mismatch pb |
|
||
***** Bias/variance on mismatched trainig and dev/test sets
|
||
|
||
| Human level | 4% |
|
||
avoidable bias
|
||
| Training set error | 7% |
|
||
variance
|
||
| Training-dev set error | 10% |
|
||
data mismatch
|
||
| Dev error | 12% |
|
||
degree of overfitting to dev set
|
||
| Test error | 12% |
|
||
|
||
Example, training is much harder than dev/test set distribution:
|
||
|
||
| Human level | 4% |
|
||
| Training set error | 7% |
|
||
| Training-dev set error | 10% |
|
||
| Dev error | 6% |
|
||
| Test error | 6% |
|
||
|
||
***** More general formulation
|
||
|
||
The numbers can be place onto a table:
|
||
|
||
| | General Speech rec tasks | Rearview mirror speech data |
|
||
|--------------------+--------------------------+-----------------------------|
|
||
| Human lvl | "Human level err" (4%) | 6% |
|
||
| err on trained on | "Training err" (7%) | 6% |
|
||
| err not trained on | "Training-dev err" (10%) | "Dev/Test err" (6%) |
|
||
|
||
**** Addressing data mismatch
|
||
|
||
There are not any systematic way to address that.
|
||
But there are things you can try.
|
||
|
||
***** Addressing data mismatch
|
||
- Carry out manual error analysis to try to understand difference between
|
||
training and dev/test sets.
|
||
ex: you might find that a lot of dev set is noisy (car noise)
|
||
- Make training data more similar, or collect more data similar to dev/test sets.
|
||
ex: simulate noisy in-car data.
|
||
|
||
***** Artificial data synthesis
|
||
|
||
- Clean + car noise = synthetized in-car audio
|
||
|
||
Create more data, and can be a reasonable process.
|
||
|
||
Let's say you have 10k hrs of sound and only 1hr of car noise.
|
||
|
||
There is a risk your algorithm will overfit your 1hr car noise.
|
||
|
||
***** Artificial data synthesis (2)
|
||
Car recognition
|
||
|
||
Using car generated by computer vs just photos. You might overfit generated
|
||
cars. A video game might have only 20 cars, so overfit these 20 cars.
|
||
|
||
*** Learning from multiple tasks
|
||
|
||
**** Transfer learning
|
||
|
||
Learning recognize cats to help to read x-ray scans.
|
||
|
||
***** Transfer learning
|
||
|
||
Create new NN by changing just the last layer (the output).
|
||
|
||
(X,Y) now become (radiology images, diagnosis)
|
||
|
||
retrain the W^[Z], b^[Z].
|
||
|
||
You might want to train just the last layer, you all the layers.
|
||
|
||
The rule of thumb, just the last layer on few data.
|
||
The rule of thumb, all the layer on lot of datas.
|
||
|
||
pre-training, and fine-tuning.
|
||
|
||
A lot of low-level features learning from a very large data set might help.
|
||
|
||
- Another example. Speech recognition system:
|
||
|
||
X (audio) y (speech recognition) (wakeword, trigger word (ok google, hey siri, etc...))
|
||
|
||
You could add several new layers, and retrain the new layers or even more layers.
|
||
|
||
|
||
It make sense to transfer make sense when you have a very different number of examples.
|
||
|
||
- 10^6 image recognition, but only 100 radiology data.
|
||
- 10k hrs sounds, but only 1h data for wake words...
|
||
|
||
Transfering from lot of data to small number of data.
|
||
|
||
It doesn't make sense to transfer the other way.
|
||
|
||
***** When transfer learning makes sense
|
||
|
||
Task from A to B
|
||
|
||
- Task A and B have the same input X
|
||
- You have a lot more data for Task A than Task B
|
||
- Low level features from A could be helpful for learning B
|
||
|
||
**** Multi-task learning
|
||
|
||
Simultaneously learn multiple tasks.
|
||
|
||
***** Simplified autonomous driving example
|
||
|
||
|
||
| | y^(i) | (4,1)
|
||
|----------------+-------|
|
||
| pedestrians | 0 |
|
||
| cars | 1 |
|
||
| stop signs | 1 |
|
||
| traffic lights | 0 |
|
||
|
||
Y = [ y^(1) y^(2) .... y^(m) ]
|
||
|
||
***** Neural network architecture
|
||
|
||
x -> [] -> [] .... -> ^y in R^4
|
||
|
||
Loss: y(i) -> 1/m \sum_i=1^m \sum_j=1^4 (L(y^(i)_j , y^(i)_j))
|
||
|
||
L is the usual loss function.
|
||
|
||
Unlike softmax regression, one image can have multiple labels.
|
||
|
||
- One NN doing 4 things is better than learning 4 different NN for each task.
|
||
|
||
Some examples might not be fully labelled.
|
||
And you can train by summing only other 0/1 label and not on ? mark (un labeled values).
|
||
|
||
So you can use more informations.
|
||
|
||
***** When multi-task learning makes sens.
|
||
|
||
- Training on set of tasks taht could benefit from having shared lower-level features
|
||
- Usually: amount of data you have for each task is quite similar
|
||
- Can train a big enough neural network to do well on all the tasks
|
||
|
||
Multi-task learning used a lot more than transfer learning.
|
||
|
||
*** End-to-end deep learning
|
||
**** What is end-to-end deep learning?
|
||
***** What is end-to-end deep learning?
|
||
Speech recognition example
|
||
|
||
|
||
audio - MFCC -> features -- ML --> phonemes -> words -> transcript
|
||
|
||
|
||
audio ------------------------------------------------> transcript
|
||
|
||
You might need a lot of data.
|
||
3k hrs of data, classical approach better.
|
||
10k to 100k hurs then end-to-end approach generally shines.
|
||
***** Face recognition
|
||
|
||
Multi state approach works better:
|
||
|
||
1. detect face, zoom-in and crop to center the face
|
||
2. then feed this croped image to find identity. Generally comparing to all
|
||
employes.
|
||
|
||
Why?
|
||
|
||
- Have a lot of data for task 1
|
||
- Have a lot of data for task 2
|
||
|
||
If you were to try to learn everything at the same time you wouldn't have enough
|
||
data.
|
||
***** More examples
|
||
Machine translation:
|
||
|
||
English -> text analysis -> ... -> French
|
||
English -------------------------> French
|
||
|
||
Because we have lot of (x,y) examples.
|
||
|
||
Estimating child's age from scan of the hand:
|
||
|
||
Image -> bones -> age
|
||
Image ----------> age (there is not enough data)
|
||
**** Whether to use end-to-end deep learning
|
||
***** Pros and cons of end-to-end learning
|
||
Pros:
|
||
- let the data speak (no human preconception)
|
||
- Less hand-designing of components needed
|
||
|
||
Cons:
|
||
- May need a large amount of data: input end ----> output end (x,y)
|
||
- Excludes potentially useful hand-designed components. Data, Hand-design
|
||
***** Applying end-to-end deep learning
|
||
Key question: do you have sufficient data to learn a function of the complexity
|
||
needed to map x to y?
|
||
|
||
- choose X->Y mapping
|
||
- pure deep learning approch not appropriate if hard to find end-to-end exmaples.
|
||
* Convolutional Neural Networks
|
||
** Week 1
|
||
*** Computer Vision
|
||
size: 64x64x3 -> 12288
|
||
size: 1000x1000x3 -> 3 millions
|
||
|
||
[x_1 ... x_{3millions}] and w^[1] -> [1000 x 3e6]
|
||
*** Edge Detection Example
|
||
|
||
Take the 6x6 image and "convolve it by the 3x3 matrix" filter: [[1,1,1],[0,0,0],[-1,-1,-1]]
|
||
|
||
python: conv_forward
|
||
tensorflow. tf.nn.conv2d
|
||
keras: conv2D
|
||
*** More edge detection
|
||
|
||
Sobel filter: 1,2,1 , 0,0,0, -1,-2,-1
|
||
Scharr filter: 3,10,3 , 0,0,0, -3,-10,-3
|
||
*** Padding
|
||
|
||
Size of image shrink because of borders.
|
||
If filter as size f and image size n -> final image after filter: n - f +1
|
||
**** first solution but a border around the image: Padding
|
||
|
||
"valid" : nxn * fxf -> n-f+1 x n-f+1
|
||
"same": Pad so the output size is the same as the input size
|
||
n + 2p -f + 1 => p = (f-1)/2
|
||
|
||
3x3 -> p = 3-1/2 = 1
|
||
5x5 -> p = 5-1/2 = 2
|
||
- f is usually odd, easier for padding + the filter has a central position.
|
||
- 1x1, 3x3, 5x5, 7x7.
|
||
*** Strided Convolutions
|
||
|
||
Jump some columns/lines. Instead of sliding the filter on every columns/row, do it every n columns/ n rows.
|
||
|
||
nxn * fxf, padding: p, stride: s
|
||
|
||
floor ((n + 2p -f / s) + 1) x floor ((n + 2p -f / s) + 1)
|
||
**** Summary of convolutions
|
||
|
||
nxn image
|
||
fxf filter
|
||
|
||
padding p
|
||
stride s
|
||
|
||
output size:
|
||
\[ \floor ((n + 2p -f / s) + 1) x \floor ((n + 2p -f / s) + 1) \]
|
||
**** Convolution in math textbook (flip vertical and horizontal the filter)
|
||
|
||
cross-correlation vs convolution
|
||
|
||
By convention we call cross-correlation, convolution operator.
|
||
|
||
The convolution op is cross-associative:
|
||
(A * B) * C = A * (B * C)
|
||
*** Convolution over volumes
|
||
|
||
6x6x3 * 3x3x3 -> 4x4
|
||
height x width x #channels
|
||
|
||
By convention the nb of channels will be same in the image and in the filter.
|
||
**** Multiple filters
|
||
|
||
6x6x3 * 3x3x3 ---\ 4x4
|
||
* 3x3x3 ---/ 4x4 ====> 4x4x2, 2, n_c = #filters
|
||
*** One layer of convolutional neural network
|
||
10 filters that are 3x3x3 in one layer of NN, how many parameters?
|
||
|
||
3x3x3 = 27 + bias = 28 params
|
||
28 x 10 = 280 params
|
||
**** Summary of notation
|
||
|
||
If layer l is a convolutional layer:
|
||
f^[l] = filter size
|
||
p^[l] = padding
|
||
s^[l] = stride
|
||
nc^[l] = number of filters
|
||
|
||
each filter is f^[l] x f^[l] x n_c^[l-1]
|
||
Activations: a^[l] -> n_H^[l] x n_W[l] x n_c^[l]
|
||
A^[l] -> m x n_H^[l] x n_w^[l] x n_c^[l]
|
||
Weights: f^[l] x f^[l] x n_c^[l-1] x n_c^[l] (n_c^[l]: #filters in layer l)
|
||
Bias: n_c^[l] - (1,1,1,n_c^[l])
|
||
|
||
Input: n_H^[l-1] x n_W^[l-1] x n_c^[l-1]
|
||
Output: n_H^[l] x n_W^[l] x n_c^[l]
|
||
|
||
n^[l] = floor ( n^[l-1] + 2p^[l] - f^[l] / s^[l]) +1
|
||
*** A simple convolution neural network example
|
||
|
||
|_|/ ---------------------------->
|
||
39x39x3 f[1]=3, s^[1]=1, p^[1]= 0
|
||
n_H^[0] = n_W^[0] = 39 10 filters
|
||
n_c^[0]=3
|
||
|
||
|
||
|_|/ ---------------------------->
|
||
37x37x10 f[2]=5, s^[1]=2, p^[1]=0
|
||
20 filters
|
||
|
||
|_|/ ---------------------------->
|
||
17x17x20 f[3]=5, s^[1]=2, p^[1]=0
|
||
40 filters
|
||
|
||
|_|/ ---- 1960 params --> softmax y^hat
|
||
7x7x40
|
||
|
||
**** Type of layer in a CNN
|
||
|
||
- Convolution (CONV)
|
||
- Pooling (POOL)
|
||
- Fully connected (FC)
|
||
|
||
*** Pooling Layer
|
||
|
||
**** Max pooling
|
||
|
||
4x4 --- max over 2x2 region --> 2x2
|
||
|
||
Hyperparameters: f=2, s=2
|
||
No parameters to learn!
|
||
|
||
In practice it works well.
|
||
|
||
**** Example
|
||
|
||
5x5 with f=3 s=1
|
||
|
||
1 3 2 1 3
|
||
2 9 1 1 5 9 9 5
|
||
1 3 2 3 2 ====> 9 9 5
|
||
8 3 5 1 0 8 6 9
|
||
5 6 1 2 9
|
||
|
||
|
||
Overs #channels, the output has the same number of channels.
|
||
Max pooling over each channel independently
|
||
|
||
**** Average Pooling
|
||
|
||
Same as previous, but we take the average instead of the max
|
||
|
||
**** Summarize
|
||
Hyperparameters
|
||
- f: filter size
|
||
- s: stride
|
||
- Max or Average pooling
|
||
- p: padding (almost never used, p=0 in general)
|
||
|
||
|
||
nH x nW x nc ----> ((nH - f / s) + 1) x ((nW - f / s) + 1) + n_c
|
||
|
||
*** CNN Example
|
||
|
||
- Input Image: 32x32x3 (try to recognize a 7 letter)
|
||
--- conv f=5, s=1 ---->
|
||
- Conv1: 28x28x6 ---- max pool, f=2, s=2 ----> POOL1 14x14x6 } LAYER 1
|
||
--- conv f=5 s=1 -->
|
||
- Conv2: 10x10x16 ---- max pool, f=2, s=2 ----> POOL2 5x5x16 } LAYER 2
|
||
|
||
400 ----> FC3 120 ---> FC4 84 ----> 0 softmax (10 outputs)
|
||
W^[3] (120,400)
|
||
b^[3] (120)
|
||
|
||
In general:
|
||
|
||
- n_H, n_W will decrease as we go deeper
|
||
- n_c will increase as we go deeper
|
||
- CONV - POOL - CONV - POOL - FC -FC -FC -SOFTMAX
|
||
|
||
**** Sizes
|
||
Activation size go down, # parameters, few for Conv and 0 for POOL, a lot in FC
|
||
|
||
*** Why convolutions?
|
||
|
||
Two advantages:
|
||
- parameter sharing and sparsity of connections.
|
||
|
||
|
||
If we had to make a FC between a 32x32x3 --> 28x28x6, would need millions of parameters.
|
||
Conv as only 156 parameters.
|
||
|
||
- *Parameter sharing*: A feature detector that's useful in one part of the image
|
||
is probably useful in another part of the image.
|
||
|
||
- *Sparsity of connections*: In each layer, each output value depends only on a
|
||
small number of inputs.
|
||
** Week 2
|
||
*** Case studies
|
||
**** Why look at case studies?
|
||
***** Outline
|
||
Classic networks:
|
||
- LeNet-5
|
||
- AlexNet
|
||
- VGG
|
||
|
||
ResNet (residual network), 152 deep network
|
||
|
||
Inception
|
||
**** Classic Networks
|
||
***** LeNet - 5
|
||
Recognize handwritten digits,
|
||
32x32x1 --- conv 5x5 s = 1 --->
|
||
28x28x6 --- avg pool, f=2, s=2 -->
|
||
14x14x6 --- conv 5x5,s=1 -->
|
||
10x10x16 --- avg pool, f=2 s=2 --->
|
||
5x5x16 (400) --> FC 120 --> FC 84 --> Softmax y^
|
||
|
||
Size was 60k parameters. Today hunder millions parameters
|
||
|
||
n_H, n_W decrease, n_C increase
|
||
|
||
Conv, pool, Conv, pool, fc, fc, output
|
||
|
||
LeCun et al. 1998, Gradient-based learning applied to document recognition.
|
||
***** AlexNet
|
||
Alex Krizhevsky et al. 2012, ImageNet classification with deep convolutional neural networks.
|
||
|
||
227x227x3 --- conv 11x11, s=4 --->
|
||
55x55x96 --- MAX Pool, 3x3, s=2 ---->
|
||
27x27x96 --- 5x5 same ------>
|
||
27x27x256 --- MAX POOL, 3x3, s=2 ---->
|
||
13x13x256 --- 3x3, same ----->
|
||
13x13x384 --- 3x3 ---> 13x13x384 ---3x3 ---> 13x13x256 --- MAX POOL, 3x3, s=2 --->
|
||
6x6x256 --- FC 9216 --> FC 4096 --> FC 4096 --> Softmax
|
||
|
||
Similar to previous but MUCH bigger:
|
||
|
||
60 millions parameters
|
||
|
||
Also use ReLU
|
||
***** VGG - 16
|
||
|
||
CONV = 3x3 filter, s=1, same
|
||
MAX-POOL = 2x2, s=2
|
||
|
||
224x224x3 --- [CONV 64]x2 ---> 225x224x64 --- POOL --->
|
||
112x112x64 --- [CONV 128]x2 ---> 112x112x128 --- POOL --->
|
||
56x56x128 --- [CONV 256]x3 ---> 56x56x256 --- POOL --->
|
||
28x28x256 --- [CONV 512]x3 ---> 28x28x512 --- POOL --->
|
||
14x14x512 --- [CONV 512]x3 ---> 14x14x512 --- POOL --->
|
||
7x7x512 ---> FC 4096 --> FC 4096 ---> Sofmax 1000
|
||
|
||
Simonyan & Zisserman 2015. Very deep convolutional networks for large-scale image recognition.
|
||
|
||
about 138 millions parameters
|
||
|
||
Also VGG-19 is also another even bigger network, but VGG-16 perform as good as VGG-19
|
||
**** Residual Networks (ResNets)
|
||
Very very deep neural, over 100 layers.
|
||
***** Residual block
|
||
a^[l] ---> a^[l+1] --> a^[l+2]
|
||
|
||
a^[l] --+--> linear --> ReLU --> a^[l+1] -----> linear -----> ReLU --> a^[l+2]
|
||
| |
|
||
+-------------------------------------------------+
|
||
shortcut (skip connection)
|
||
|
||
|
||
a^[l+2] = g (z^[l+2] + a^[l])
|
||
|
||
passes information deeper in the NN.
|
||
|
||
He et al, 2015, Deep residual networks for image recognition
|
||
|
||
x -> [] -> [] -> [] .... -> a^[l]
|
||
|
||
Help in vanishing gradient descent parameters.
|
||
**** Why ResNets Work
|
||
***** Why do residual networks work so well?
|
||
In practice, learning on too deep network make bad results on the training set.
|
||
So that prevent using network that are too deep.
|
||
|
||
But it much less true when learning ResNets.
|
||
|
||
X ---> Big NN ---> a^[l]
|
||
|
||
X ----> Big NN ---> a^[l] -+-> [] --> [] -+->a ^[l+2]
|
||
ReLU a >= 0 | |
|
||
+--------------+
|
||
|
||
a^[l+2] = g (z^[l+2] + a^[l])
|
||
= g ( W^[l+2]a^[l+1] + b^[l+2] + a^[l] )
|
||
|
||
If we're using L2 regularization that would shrink the value of W^[l+2], also b^[l+2]
|
||
If W^[l+2] =0 and also b^[l+2], then we'll have g(a^[l] which is equal to a^[l]
|
||
|
||
So the identity function is easy to learn for ReLU.
|
||
So adding the shortcut don't hurt performance.
|
||
But if its goes good, it's better but almost never worse.
|
||
|
||
Remark: We're assuming z^[l+2] and a^[l] is of the same dimension.
|
||
In case they have different dimensions, add an extra matrix W_s so we have:
|
||
|
||
(W_s x a^[l])
|
||
***** ResNet example
|
||
Plain ----> ResNet
|
||
**** Networks in Networks and 1x1 Convolutions
|
||
Using a 1x1 convolution.
|
||
***** What does a 1x1 convolution do?
|
||
|
||
6x6x1 * 2 ---> simple multiplication
|
||
|
||
6x6x32 x 1x1x32 ---> 6x6x#filters
|
||
|
||
- Lin et al. 2013, Network in Network.
|
||
***** Using 1x1 conv
|
||
|
||
28x28x192 ---- ReLU, CONV 1x1, 32 filters ---> 28x28x32
|
||
|
||
Let shrink n_C as well
|
||
|
||
Effect non-linearity, we could keep the nb of layers and its fine.
|
||
|
||
*1x1 conv does a non trivial operation.*
|
||
**** Inception Network Motivation
|
||
We might have to pick, conv 3x3, pool, layer, etc...
|
||
***** Motivation for inception network.
|
||
Do all transformations at the same time:
|
||
|
||
28x28x192 --- 1x1 ----> 28x28x64
|
||
\\\--- 3x3,same ----> 28x28x128
|
||
\\--- 5x5,same ----> 28x28x32
|
||
\--- max pool, same, s=1 ----> 28x28x32
|
||
------------------------------------------------------
|
||
28x28x256
|
||
|
||
Szegedy et al. 2014, Going deeper with convolutions.
|
||
|
||
pb computational cost.
|
||
***** The problem of computational cost
|
||
28x28x192 --- conv 5x5, same, 32 ----> 28x28x32
|
||
|
||
32 filters, filters are 5x5x192
|
||
|
||
Nb of multiplications: 28x28x32 x 5x5x192 = 120 millions (costly)
|
||
***** Using 1x1 conv
|
||
28x28x192 ------- conv 1x1, 16, 1x1x192 --->
|
||
28x28x16 -------- conv 5x5, 32, 5x5x16 ---> 28x28x32
|
||
Bottleneck Layer
|
||
|
||
cost of 1st conv layer: 28x28x16 x 192 = 2.4 millions
|
||
cost of 2nd conv layer: 28x28x32 x 5x5x16 = 10 millions
|
||
total cost: 12.4 millions (about 10x less than before)
|
||
|
||
You can reduce substantially the size without hurting performace of the NN while
|
||
improving performances.
|
||
**** Inception Network
|
||
***** Inception module
|
||
Previous Activation: 28x28x192
|
||
|
||
1x1 conv --------------------------------------------> 28x28x64 -\
|
||
1x1 conv ----------------> 3x3 conv ----------------> 28x28x128 -- Channel concat:
|
||
1x1 conv ----------------> 5x5 conv ----------------> 28x28x32 // 28x28x256
|
||
MAXPOOL,3x3,s=1, same --> 28x28x192 --> 1x1 CONV ---> 28x28x32 /
|
||
|
||
Inception network, si the same pattern (block) connected in many layers.
|
||
|
||
Inception-block --> Inception block ---> ..... ---> Softmax layer output
|
||
\-> softmax layer \-> softmax layer
|
||
|
||
GoogleNet
|
||
***** Fun fact
|
||
We need to go deeper (from the Inception meme)
|
||
|
||
Since the dev, there are newer versions of inception modules.
|
||
Inception v1, v2, v3 ....
|
||
*** Practical advice for using ConvNet
|
||
**** Using Open-Source Implementation
|
||
Lot of these networks are difficult to reproduce, replicate the work.
|
||
|
||
Search look online implementation, instead of re-implementing from scratch.
|
||
|
||
Demo:
|
||
1. google
|
||
2. github repository
|
||
3. check license (MIT for example)
|
||
4. git clone ...
|
||
|
||
|
||
Also advantage, pre-trained networks.
|
||
Starting from open-source implementation is faster.
|
||
**** Transfer Learning
|
||
There are lot of data set on the Internet.
|
||
You could often download have an initial pretrained NN.
|
||
***** Transfer Learning
|
||
Classification Problem: is it Tigger, Misty or Neither (recognize cats)
|
||
|
||
Trained network trained on ImageNet.
|
||
|
||
Get rid of the soft-max layer and create your own, and output, Tigger/Misty or Neither.
|
||
And freeze the other parameters.
|
||
Just learn the Softmax layer.
|
||
|
||
You might get pretty good results.
|
||
|
||
Depending on the framework:
|
||
- TrainableParameters = 0
|
||
- freeze = 1
|
||
|
||
Pre-compute and save the last layer using for activation.
|
||
You don't need to recompute these activations.
|
||
|
||
*If you have a larger dataset*:
|
||
|
||
Freeze, fewer layers (the firsts)
|
||
Or freese a few layers and create a new layers with your own architecture.
|
||
|
||
*If you have a lot of data*:
|
||
|
||
Use the hole thing and use it as initialization.
|
||
|
||
You should almost always do transer learning unless you have an exceptionally
|
||
large dataset to train.
|
||
**** Data Augmentation
|
||
For the majority of Computer vision problem, having more data is almost always
|
||
useful and help.
|
||
***** Common augmentation
|
||
- mirroring images
|
||
- Random cropping, not perfect, but work well in practice
|
||
- Also: Rotation, shearing, local warping... but not much used in practice
|
||
***** Color shifting
|
||
Add to RGB different distortions.
|
||
Ex: +20,-20,+20 ---> more mauve
|
||
etc...
|
||
|
||
Advanced: PCA color augmentation (in AlexNet paper)
|
||
***** Implementing distortions during training
|
||
Harddisk ---> CPU threadd ---> distortions ----> training (CPU/GPU)
|
||
\-- load -> color
|
||
minibatch
|
||
|
||
Meta parameters, so certainly use open-source implementation to data
|
||
augmentation.
|
||
**** State of Computer Vision
|
||
***** Data vs hand-engineered
|
||
Most ML problem.
|
||
|
||
Little data <--------------------------------------------> Lots of data
|
||
|
||
speach recognition: lot of data
|
||
image recognition: OK data
|
||
Object detection: less data
|
||
|
||
Lot of data: simpler algorithms, less hand-engineering.
|
||
Few data: more hand-engineering, hacks
|
||
|
||
Two sources of knowledge:
|
||
- Labeled dataset (x,y)
|
||
- Hand engineering features/network architecture/other components
|
||
|
||
|
||
When very few data: Transfer learning.
|
||
***** Tips for doing well on benchmarks/winning competitions
|
||
- Ensembling
|
||
- Train several networks independently and average their outputs (not their weights)
|
||
- 3/15 networks (but almost never used in production, because it is costly for few benefits)
|
||
- Multi-crop at test time
|
||
- run classifier on multiple versions of test images and average results
|
||
- 10-crop: central + 4 corner + same on mirrored
|
||
|
||
Do not do this in production systems.
|
||
***** Use open source code
|
||
- Use architectures of networks published in the literature
|
||
- Use open source implementations if possible
|
||
- Use pretrained models and fine-tune on your dataset
|