deft/Deep-Learning.org

-- #+TITLE: Deep Learning Coursera
-- #+AUTHOR: Yann Esposito
#+STARTUP: latexpreview
#+TODO: TODO IN-PROGRESS WAITING | DONE CANCELED
#+COLUMNS: %TODO %3PRIORITY %40ITEM(Task) %17EFFORT(Estimated Effort){:} %CLOCKSUM %8TAGS(TAG)

* Plan

5 courses

** Neural Network and Deep Learning
*** Week 1: Introduction
*** Week 2: Basic of Neural Network programming
*** Week 3: One hidden layer Neural Networks
*** Week 4: Deep Neural Network
** Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization
** Structuring your Machine Learning project
** Convolutional Neural Networks
** Natural Language Processing: Building sequence models
* DONE Neural Network and Deep Learning
  CLOSED: [2017-08-22 Tue 13:43]
** Introduction

*** What is a neural network?

*** Supervised Learning with Neural Networks

 - Lucrative application: ads, showing the add you're most likely to click on
 - Photo tagging
 - Speech recognition
 - Machine translation
 - Autonomous driving


***** Convolutional NN good for images

***** Strutured data (db of data) vs Unstructured data

 - Structured data: Tables
 - Unstructured data: Audio, image, text...

 Computer are much better at interpreting unstructured data.

*** Why is Deep Learning taking off?

 [[///Users/yaesposi/Library/Mobile%20Documents/com~apple~CloudDocs/deft/img/Scale%20drives%20deep%20learning%20progress.png]]

 - Data (lot of data)
 - Computation (faster learning loop)
 - Algorithms (ex, use ReLU instead of sigma)
** Geoffrey Hinton interview
** Binary Classification

\[ (x,y) x\in \mathbb{R}^{n_x}, y \in {0,1} \]

$m$ training examples: $$ {(x^{(1)},y^{(1)}), ... (x^{(m)},y^{(m)})} $$

$$ m = m_{train} , m_{test}  = #test examples $$

$$ X = [ X^{(1)} ... X^{(m)} ] is an n_x x m matrix $$
$$ X.shape (n_x,m) $$

$$ Y = [ y^{(1)} ... y^{(m)} ] $$
$$ Y.shape = (1,m) $$

** Logistic Regression

Given $X \in \mathbb{R}^{n_x}$ you want $\hat{y} = P(y=1 | X)$

Paramters: $w \in \mathbb{R}^{n_x}, b\in \mathbb{R}$

Output: $\hat{y} = \sigma(w^Tx + b) = \sigma(z)$

$$\sigma(z)= \frac{1}{1 + e^{-z}}$$

If $z \rightarrow \infty => \sigma(z) \approx 1$
If $z \rightarrow - \infty => \sigma(z) \approx 0$


Alternative notation not used in this course:

$X_0=1, x\in\mathbb{R}^{n_x+1}$
$\hat{y} = \sigma(\Theta^Tx)$
...

** Logistic Regression Cost Function

Search a convex loss function:

$L(\hat{y},y) = - (y\log(\hat{y}) + (1-y)\log(1-\hat{y}))$

If y = 1 : $L(\hat{y},y) = -\log\hat{y}$ <- want log\haty larg, want \hat{y} large
If y = 0 : $L(\hat{y},y) = -\log\hat{y}$ <- want log (1-\hat{y}) large, want \hat{y} sall

Cost function: $$ J(w,b) = \frac{1}{m}\sum_{i=1}^mL(\hat{y^\{(i)}},y^{(i)}) = ... $$

** Gradient Descent

Minize $J(w,b)$

1. initialize w,b (generaly uses zero)
2. Take a step in the steepest descent direction
3. repeat 2 until reaching global optimum

Repeat {
  $w := w - \alpha\frac{dJ(w)}{dw} = w - \alpha\mathtext{dw}$
}

** Derivatives
** More Derivative Examples
** Computaion Graph
** Computing Derivatives
** Computing Derivatives for multiple examples
** Vectorization
getting rid of explicit for loops in your code
** Vectorizing Logistic Regression
** Vectorizing Logistic Regression's Gradient Computation
** Broadcasting in Python
** Quick Tour of Jupyter / ipython notebooks
** Neural Network Basics
J = a*b + a*c - (b+c) = a (b + c) - (b + c) = (a - 1) (b + c)
* DONE Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization
  CLOSED: [2017-09-01 Fri 09:52]
** DONE Week 1: Setting up your Machine
   CLOSED: [2017-08-22 Tue 13:43]
*** Recipe

If *High bias*? (bad training set performance?)
   Then try:
     - Bigger network
     - Training longer
     - (NN architecture search)
   Else if *High variance*? (bad dev set performance?)
     Then try:
     - More data
     - Regularization
     - (NN architecture search)

Deep learning, not much bias/variance tradeoff if we have a big amount of
computer power (bigger network) and lot of data.
*** Regularization
**** Regularization: reduce variance
 - L2 regularization

 λ / 2m || w ||_2 ^2

 - L1 regularization: same with |w| instead of ||w||_2^2


 λ is a regularization parameter (in code named =lambd=)

 Cost = J(w^[1], b^[1], ..., w^[L], b^[L]) = 1/m \sum L(^y(i), y(i)) + λ/2m \sum_l=1^L || W^[l] ||^2


 call the "Frobenius norm"

 dW = from backprop + λ/m W^l

 update W^l = W^l - αdW^l still works

 Sometime L2 regularization called "weight decay".

**** Dropout Regularization

 Eliminates nodes by layer randomly for each training example.

 - implementing, (inverted dropout)
   - gen random boolean vector:
      d3 = np.random.rand(a3.shape[0], a3.shape[1]) < keep_prob # (for each iteration)
      a3 = np.mulitply(a3,d3)
      a3 /= keep_prob (for normalization to be certain the a3 output still the same, reduce testing problems)

 Making prediction at test time: no drop out

**** Over regularization methods

 - Data augmentation, (flipping images for example, random crops, random distortions, etc...)
 - Early stopping, stop earlier iteration

*** Setting up your optimization problem

**** Normalizing Inputs

 - μ = 1/m Sum X^(i)
 - x := x - μ (centralize)
 - σ = 1/m Sum X^(i)^2
 - x /= σ^2


**** Gradient Checking
***** Don't use gard check in traingin, only in debug
***** If algorithm fail, grad check, look at component (is db? dW? dW on certain layer, etc...)
***** Remember regularization
***** Doesn't work with dropout, turn off drop out (put 1.0) then check
***** Run at random initialization; perhaps again after training

** DONE Week 2: Optimization Algorithms
   CLOSED: [2017-08-22 Tue 13:43]
*** Mini batch

    X :: X^(1) ... X^(m)

    X,Y -> X^{i},Y^{i} where X^{i} = X^(i*batch-size ---> (i+1)*batch-size)

*** Minibatch size

- if mini batch size = m => Batch gradient descent (X^{1},Y^{1}) = (X,Y)
- if mini match size = 1 => Stochastic gradient descent, every example is its own mini batch.
- in practice in between 1 and m, m --> too  long, 1 loose speedup from vectorization.
  + vectorization ~1000

1. If small training set, use batch gradient descent (m <= 2000)
2. Typical mini-batch size: 64, 128, 256, 512, ... 2^k to fits in CPU/GPU memory

*** Exponentially weighted average

v_t = βv_{t-1} + (1-β)θ_t
** DONE Week 3: Hyperparameter
   CLOSED: [2017-09-01 Fri 09:52]
*** Video 1: use random not a grid to search for hyperparameter best value
*** Video 2: choose appropriate scale to pick hyperparameter
- uniformly random n^[l] (number of neuron for layer l) or L (number of layers)
- alpha: between 0.00001 to 1, then shouldn't use linear but instead use log-scale
    r = -4*np.random.rand() <- r in [-4,0]
    α = 10^r                <- 10^-4 ... 10^0

- β <- 0.9 ... 0.999 (0.9 about avg on 10 values, 0.999 avg about 1000 values)
    1-β = 0.1 .... 0.001
    r <- [-3,-1]
    1-β = 10^r
*** Hyperparameter: Tuning in practice Panda vs caviar
- Babysitting one model (panda) for few computer resources
- Training many models in parallel (caviar) for lot of computer resources
*** Batch normalization
**** In a network
**** Fitting Batch norm into a deep network
**** Why Batch Normalizing?
- don't use batch norm as a regularization even if sometime it could have this
  effect
**** Batch Norm at test time
μ = 1/m \sum z^(i)

σ^2 = 1/m \sum (z^(i) - μ)^2

z^(i)_norm = z^(i) - μ / sqrt( σ^2 + ε )

~z^(i) = γz^(i)_norm + β

Estimate μ and σ with exponentially weighted avg accross minibatches
*** Multi-class classification
**** Softmax Regression
notation: C = #classes (0,1,2...,C-1)

last hidden layer nb of neuron is equal to C: n^L = C

z^[L] = w[L]a^[L-1] + b[L] (C,1)

Activation function:

t = e^(Z[L])
a^[L] = e^(Z[L])/\sum_i=0^C t_i

a^[L]_i = t_i / \sum_i=0^C t_i

**** Training a softmax classifier
*** Introduction to programming frameworks

**** Deep learning frameworks
* Structuring your Machine Learning project
** Week 1
*** Introduction to ML Strategy
**** Why ML Strategy
Try to find quick and effective way to choose a strategy

Ways of analyzing ML problems

**** Orthogonalization

***** Chain of assumptions in ML

 - Fit training set well on cost function => bigger network, Adam, ...
 - Fit dev set well on cost function => Regularization, Bigger training set
 - Fit test set well on cost function => Bigger dev set
 - Perform well in real world => Change the devset or cost function

 Try not to use early stoping as it simulanously affect cost on training and dev set.

*** Setting up your goal

**** Single number evaluation metric

***** First

| Classifier | Precision | Recall |
|------------+-----------+--------|
| A          |       95% |    90% |
| B          |       98% |    85% |

Rather than using two number, find a new evaluation metric


| Classifier | Precision | Recall | F1 Score |
|------------+-----------+--------+----------|
| A          |       95% |    90% |    92.4% |
| B          |       98% |    85% |     91.0 |

F1 score = 2 / (1/p) + (1/R) :: "Harmonic mean" of precision and recall.

So:

Having a good Dev set + single evaluation metric, really speed up iterating.

***** Another example

| Algorithm | US  | China | India | Other | *Average* |
|-----------+-----+-------+-------+-------+-----------|
| A         | 3%  | 7%    |    5% |    9% |           |
| ...       |     |       |       |       |           |
| F         | ... | ...   |       |       |           |

Try to improve the average.

**** Satisficing and Optimizing metric

It's not alway easy to select on metric to optimize.

***** Another cat classification example

| Classifier | Accuracy | Running Time |
|------------+----------+--------------|
| A          |      90% | 80ms         |
| B          |      92% | 95ms         |
| C          |      95% | 1500ms       |

cost = accuracy - 0.5x running time

maximize accuracy s.t. running time < 100ms

Accuracy <- Optimizing
Running time <- Satisficing

If you have n metrics, pick one to optimizing, and all the other be satisficing.

**** Train/dev/test distribution

How you can setup these dataset to speed up your work.

***** Cat classification dev/test sets

Try to find a way that dev and test set come from the same distribution.

***** True story (detail changed)

Optimizing on dev set on load approvals for medium income zip codes.

(repay loan?)

Tested on low income zip codes.

Lost 3 months

*****  Guideline

Choose a dev set and test set to reflect data you expect to get in the future
and consider important to do well on.

**** Size of dev and test sets

*****  Old way of splitting
70% train, 30% test
60% train, 20% dev, 20% test

For at max 10^4 examples

But in new era, 10^6 examples:

train: 98%, Dev 1%, Test 1%.

***** Size of test set

Set your test set to be big enough to give high confidence in the overall
performance of your system. Can be far less than 30% of your data.

For some applications, you don't need test set and only dev set.
For example if you have a very large dev set.

**** When to change dev/test sets and metrics?

Metric: classification error
Algorithm A: 3% error → letting throught a lot of porn images
Algorithm B: 5% error → doesn't let pass porn images

So your metric + evaluation prefer A, but you and your users prefer B.

When this happens, mispredict your algorithm B is better.

Error: 1/m_dev \sum_i=1^m I{y_pred^(i) /= y^(i)

They treat pron and non pron equaly but you don't want that.

We add a w(i) = 1 if non porn and 0 if porn in the formula

**** Orthogonalization for cat pictures: anti-pron

1. So far we've only discussed how to define a metric to evaluate classifier
2. Worry separately about how to do well on this metric

1. placing the target, and 2. is aiming the target.

**** Another example
Alg A: 3% err
Alg B: 5% err

But B does better. You see that users are using blurier images.
You dev/test are not using the same kind of images.

Change your metric and/or dev/test set.

*** Comparing to Humand-level performance

**** Why human-level performance

 Human-level perf vs Bayes optimal error

 Human are generally very close to bayes perf for lot of tasks.

 - get lableld data from humans
 - gain insight from manual error analysis (why did a person get this right?)
 - better analysis of bias/variance

**** Avoidable bias

***** Cat classification example

 | Humans         |            1% |              7.5% |
 | Training error |            8% |                8% |
 | Dev error      |           10% |               10% |
 |                | focus on bias | focus on variance |

 Human level error as a proxy (estimate) for Bayes error.

 *Diff between Human err and Training err = available bias*
 *Diff between Train and Dev err = variance*

**** Understanding Human-level performance

***** Human-level error as proxy for Bayes error
 Medical image classification example:
 suppose
 (a) Typical human 3% err
 (b) Typical doctor 1% err
 (c) Experienced doctor 0.7% err
 (d) and team of experienced doctors 0.5% err

 What is "human-level" error?

 Bayes error is <= to 0.5% err
 So we use that to aim as saw before.

 For a paper, (b) is good enough to talk about that.

***** Error analysis example

 | Human (proxy for bayes err) | 1, 0.7, 0.5% | 1, 0.7, 0.5 | 1, 0.7, 0.5 |
 | Train err                   |           5% |          1% |        0.7% |
 | Dev err                     |           6% |          5% |        0.8% |
 |                             |              |             |             |

 Case 1:
 For this example it doesn't matter because avoidable bias (5 - 1%), is bigger
 than variance (6-5)

 Case 2: focus on variance

 Case 3, very important you use 0.5 as your "human-level" error. Because it show
 that you should focus on bias and not on variance.

 This problem arose only when you're doing very good.

***** Summary of bias/variance with human-level perf

 Human-level error (proxy for Bayes err)

     ^
     | "Avoidable bias"
     v

 Training error

     ^
     | "Variance"
     v

 Dev error

**** Surpassing human-level performance

***** Surpassing human-level performance

 | Team            |  0.5% |       0.5% |
 | One human       |    1% |         1% |
 | Training error  |  0.6% |       0.3% |
 | Dev error       |  0.8% |       0.4% |
 |-----------------+-------+------------|
 | Avoidable bias? | ~0.5% | can't know |

***** Problems where ML significantly surpasses human-level performance

 - Online advertising
 - Product recommendations
 - Logistics (predicting transit time)
 - Loan approvals

 all thoses examples:
 + come from structured data
 + not natural perception problems
 + Lots of data


 Also, Speech recognition, Some image recognition, Medical, ECG, skin cancer,
 etc...

**** Improving your model performance

 Set of guidelines

***** The two fundamental assumptions of supervised learning

 1. You can fit the training set pretty well (~ avoidable bias)
 2. The training set performance generalizes pretty well to the dev/test set

***** Reducing (avoidable) bias and variance

 Human-level error (proxy for Bayes err)

     ^
     |                       train bigger model
     | "Avoidable bias" =>   train longer/better optimization algorithms (momentum, RMSprop, Adam)
     |                       NN architecture/hyperparameters search (RSS, CNN...)
     v

 Training error

     ^
     |                 More data
     | "variance" =>   Regulraization (L2, dropout, data augmentation)
     |                 NN architecture/hyperparameters search
     v

 Dev error

 These concepts are easy to learn, hard to master.
 You'll be more systematics than most ML teams.

** Week 2
*** Error Analysis
**** Error Analysis
***** Carrying out error analysis
 - Imagine your cat algo doesn't work as good as expected.
 - One of your colaborator think you should focus on working on dogs.
 - Anaylize manually 100 mislabeled dev set examples
 - Count up how many are dogs
 - Supose 5% are dogs. So at most you could go from 10% err to 9.5% so not much useful.

 - Supose taht 50% of them are dogs error, so you could go down from 10% to 5%,
   so you could be more confident.
***** Evaluate multiple idea in parallel
 - fix pictures of dogs
 - fix great cats (lion, panthers, ...)
 - improve performance of blurry images


 Create spreadsheet:

 |      Image |  Dog |  Great cats |  Bluring |
 |          1 | ok   |             |          |
 |          2 |      |             | ok       |
 |          3 |      | ok          | ok       |
 |        ... |      |             |          |
 | % of total | 8%   | 43%         | 61%      |

 You sometime notice other dimensions like instagram filters...

 Could easily know where you should improve.
**** Cleaning up incorrectly labeled dataset
***** Incorrectly labeled examples
 If you have incorrectly labeled data.
 First lets consider the training set.

 So long as you don't have too much errors, DL is quite robust to random errors.

 But this is a problem for systematic errors.
***** Error analysis


 |      Image |  Dog |  Great cats |  Bluring | Comments                         |
 |        ... |      |             |          |                                  |
 |         98 | ok   |             |          | labeler missed cat in background |
 |         99 |      |             | ok       |                                  |
 |        100 |      | ok          | ok       | drawing of a cat not a real cat  |
 | % of total | 8%   | 43%         | 61%      |                                  |

 1st case:

 Overall dev set error: 10%
 Error due incorrect labels: 0.6%
 Errors due to other causes: 9.4%

 2nd case:

 Overall dev set error: 2%
 Error due incorrect labels: 0.6%
 Errors due to other causes: 1.4%

 In 2nd case, take the time to fix mislabeled examples.
***** Correctin incorrect dev/test set examples

 - Apply same process to your dev and test sets to make sure they continue to
   come from the same distribution.
 - Consider examining examples your algorithm gor right as well as ones it got
   wrong.
 - Train and dev/test data may now crom from slightly different distributions
**** Buid your first system quickly then iterate
***** Speech recognition example
 - noisy background
   - café noise
   - car noise
 - Accented speech
 - Far from microphone
 - young children's speech
 - stuttering, uh, ah, um...

 50 directions you could go, on which should you focus on?

 1. Set up dev/test set and metric
 2. Build initial system quickly
 3. Use Bias/Variance analysis & Error Analysis to prioritize next steps

 Guideline: *Build your first system quickly then iterate*

 Do not otherthink, build something quick and dirty first.
*** Mismatched training and dev/test set
**** Training and testing on different distributions
***** Cat app example
Two sources of data:
- data from webpages
- data from mobile app

Let's say you don't have lot of users (~10k from mobile, 200k from web)

You care about doing well on mobile images. You don't want to use only the 10k,
but the dilema is the 200k aren't from the same distribution.

Option 1: take the 210k images and split between train/dev/test (train 205k, 2.5k, 2.5k)
  - avantage, same distribution
  - disavantage, perform on web instead of web.
  - only 119 other the 2.5k will be from mobile.
Option 1 not recommended

Option 2:
 - train set have 200k images from the web and 5k from the mobile.
 - dev and test all mobile app images.
 - avantage you know aiming your target where you want it to be.
 - disavantage, your training distribution is different
 But other the long term it will get you better performance
***** Speech recognition example
- Speech artificial rearview mirror. (real product in China)
1. Training: take all the speech data you have; purshased data, smart speaker control, voice keyboard... (500k)
2. Dev/test: speech activated, rearview mirror (20k)

Set your training set to be 500k from 1. and Dev/Test from 2.

The training set could be 510k (500k from 1 and 10k from 2.) and Dev/Test set (5k+5k from the rest of 2.)

Much bigger training set.
****  Bias and Variance with mismatched data distribution
***** Cat classifier example
Assume humans get ~0% error.

| Training error |  1% |
| Dev error      | 10% |

Maybe there isn't a variance pb as the distribution is different.

Training-dev set: Same distrib as training set but not used for training.

Train / dev / test ==> Train split in train-2 and train-dev

So now you learn only on train-2 and check on train-dev and dev and test.

| Train err%     |     1% |               1% |
| Train-dev err% |     9% |             1.5% |
| dev err%       |    10% |              10% |
|                | Var pb | data mismatch pb |

Other examples:

| Human err%     |      0% |                      0% |
| Train err%     |     10% |                     10% |
| Train-dev err% |     11% |                     11% |
| dev err%       |     12% |                     20% |
|                | Bias pb | Bias + data mismatch pb |
***** Bias/variance on mismatched trainig and dev/test sets

| Human level            |  4% |
                                 avoidable bias
| Training set error     |  7% |
                                 variance
| Training-dev set error | 10% |
                                 data mismatch
| Dev error              | 12% |
                                 degree of overfitting to dev set
| Test error             | 12% |

Example, training is much harder than dev/test set distribution:

| Human level            |  4% |
| Training set error     |  7% |
| Training-dev set error | 10% |
| Dev error              |  6% |
| Test error             |  6% |

*****  More general formulation

The numbers can be place onto a table:

|                    | General Speech rec tasks | Rearview mirror speech data |
|--------------------+--------------------------+-----------------------------|
| Human lvl          | "Human level err" (4%)   |                          6% |
| err on trained on  | "Training err" (7%)      |                          6% |
| err not trained on | "Training-dev err" (10%) |         "Dev/Test err" (6%) |

**** Addressing data mismatch

There are not any systematic way to address that.
But there are things you can try.

***** Addressing data mismatch
- Carry out manual error analysis to try to understand difference between
  training and dev/test sets.
  ex: you might find that a lot of dev set is noisy (car noise)
- Make training data more similar, or collect more data similar to dev/test sets.
  ex: simulate noisy in-car data.

***** Artificial data synthesis

- Clean + car noise = synthetized in-car audio

Create more data, and can be a reasonable process.

Let's say you have 10k hrs of sound and only 1hr of car noise.

There is a risk your algorithm will overfit your 1hr car noise.

***** Artificial data synthesis (2)
Car recognition

Using car generated by computer vs just photos. You might overfit generated
cars. A video game might have only 20 cars, so overfit these 20 cars.

*** Learning from multiple tasks

**** Transfer learning

Learning recognize cats to help to read x-ray scans.

***** Transfer learning

Create new NN by changing just the last layer (the output).

(X,Y) now become (radiology images, diagnosis)

retrain the W^[Z], b^[Z].

You might want to train just the last layer, you all the layers.

The rule of thumb, just the last layer on few data.
The rule of thumb, all the layer on lot of datas.

pre-training, and fine-tuning.

A lot of low-level features learning from a very large data set might help.

- Another example. Speech recognition system:

X (audio) y (speech recognintion) (wakeword, trigger word (ok google, hey siri, etc...))

You could add several new layers, and retrain the new layers or even more layers.


It make sense to transfer make sense when you have a very different number of examples.

- 10^6 image recognintion, but only 100 radiology data.
- 10k hrs sounds, but only 1h data for wake words...

Transfering from lot of data to small number of data.

It doesn't make sense to transfer the other way.

***** When transfer learning makes sense

Task from A to B

- Task A and B have the same input X
- You have a lot more data for Task A than Task B
- Low level features from A could be helpful for learning B

**** Multi-task learning

Simultaneously learn multiple tasks.

***** Simplified autonomous driving example


|                | y^(i) | (4,1)
|----------------+-------|
| pedestrians    |   0   |
| cars           |   1   |
| stop signs     |   1   |
| traffic lights |   0   |

Y = [ y^(1) y^(2)  .... y^(m) ]

***** Neural network architecture

x -> [] -> [] .... -> ^y in R^4

Loss: y(i) -> 1/m \sum_i=1^m \sum_j=1^4 (L(y^(i)_j , y^(i)_j))

L is the usual loss function.

Unlike softmax regression, one image can have multiple labels.

- One NN doing 4 things is better than learning 4 different NN for each task.

Some examples might not be fully labelled.
And you can train by summing only other 0/1 label and not on ? mark (un labeled values).

So you can use more informations.

***** When multi-task learning makes sens.

- Training on set of tasks taht could benefit from having shared lower-level features
- Usually: amount of data you have for each task is quite similar
- Can train a big enough neural network to do well on all the tasks

Multi-task learning used a lot more than transfer learning.

*** End-to-end deep learning
**** What is end-to-end deep learning?
***** What is end-to-end deep learning?
Speech recognition example


audio - MFCC -> features -- ML --> phonemes -> words -> transcript


audio ------------------------------------------------> transcript

You might need a lot of data.
3k hrs of data, classical approach better.
10k to 100k hurs then end-to-end approach generally shines.
***** Face recognition

Multi state approach works better:

1. detect face, zoom-in and crop to center the face
2. then feed this croped image to find identity. Generally comparing to all
   employes.

Why?

- Have a lot of data for task 1
- Have a lot of data for task 2

If you were to try to learn everything at the same time you wouldn't have enough
data.
***** More examples
Machine translation:

English -> text analysis -> ... -> French
English -------------------------> French

Because we have lot of (x,y) examples.

Estimating child's age from scan of the hand:

Image -> bones -> age
Image ----------> age (there is not enough data)
**** Whether to use end-to-end deep learning
***** Pros and cons of end-to-end learning
Pros:
- let the data speak (no human preconception)
- Less hand-designing of components needed

Cons:
- May need a large amount of data: input end ----> output end (x,y)
- Excludes potentially useful hand-designed components. Data, Hand-design
***** Applying end-to-end deep learning
Key question: do you have sufficient data to learn a function of the complexity
needed to map x to y?

- choose X->Y mapping
- pure deep learning approch not appropriate if hard to find end-to-end exmaples.