Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
Uploaded byzukun
PDF, PPTX1,980 views

Icml2012 tutorial representation_learning

This document provides an outline and overview of Yoshua Bengio's 2012 tutorial on representation learning. The key points covered include: 1) The tutorial will cover motivations for representation learning, algorithms such as probabilistic models and auto-encoders, and analysis and practical issues. 2) Representation learning aims to automatically learn good representations of data rather than relying on handcrafted features. Learning representations can help address challenges like exploiting unlabeled data and the curse of dimensionality. 3) Deep learning algorithms attempt to learn multiple levels of increasingly complex representations, with the goal of developing more abstract, disentangled representations that generalize beyond local patterns in the data.

Download as PDF, PPTX
Representa)on  Learning       Yoshua  Bengio     ICML  2012  Tutorial   June  26th  2012,  Edinburgh,  Scotland        
Outline of the Tutorial 1.  Mo>va>ons  and  Scope   1.  Feature  /  Representa>on  learning   2.  Distributed  representa>ons   3.  Exploi>ng  unlabeled  data   4.  Deep  representa>ons   5.  Mul>-­‐task  /  Transfer  learning   6.  Invariance  vs  Disentangling   2.  Algorithms   1.  Probabilis>c  models  and  RBM  variants   2.  Auto-­‐encoder  variants  (sparse,  denoising,  contrac>ve)   3.  Explaining  away,  sparse  coding  and  Predic>ve  Sparse  Decomposi>on   4.  Deep  variants   3.  Analysis,  Issues  and  Prac>ce   1.  Tips,  tricks  and  hyper-­‐parameters   2.  Par>>on  func>on  gradient   3.  Inference   4.  Mixing  between  modes   5.  Geometry  and  probabilis>c  Interpreta>ons  of  auto-­‐encoders   6.  Open  ques>ons   See  (Bengio,  Courville  &  Vincent  2012)     “Unsupervised  Feature  Learning  and  Deep  Learning:  A  Review  and  New  Perspec>ves”   And  http://www.iro.umontreal.ca/~bengioy/talks/deep-learning-tutorial-2012.html  for  a   detailed  list  of  references.  
Ultimate Goals •  AI   •  Needs  knowledge   •  Needs  learning   •  Needs  generalizing  where  probability  mass   concentrates   •  Needs  ways  to  fight  the  curse  of  dimensionality   •  Needs  disentangling  the  underlying  explanatory  factors   (“making  sense  of  the  data”)   3  
Representing data •  In  prac>ce  ML  very  sensi>ve  to  choice  of  data  representa>on   à  feature  engineering  (where  most  effort  is  spent)   à (beber)  feature  learning  (this  talk):        automa>cally  learn  good  representa>ons     •  Probabilis>c  models:   •  Good  representa>on  =  captures  posterior  distribu,on  of   underlying  explanatory  factors  of  observed  input   •  Good  features  are  useful  to  explain  varia>ons   4  
Deep Representation Learning Deep  learning  algorithms  abempt  to  learn  mul>ple  levels  of  representa>on  of   increasing  complexity/abstrac>on     When  the  number  of  levels  can  be  data-­‐ selected,  this  is  a  deep  architecture       5  
A Good Old Deep Architecture   Op>onal  Output  layer   Here  predic>ng  a  supervised  target     Hidden  layers   These  learn  more  abstract   representa>ons  as  you  head  up     Input  layer   This  has  raw  sensory  inputs  (roughly)   6  
What We Are Fighting Against: The Curse ofDimensionality      To  generalize  locally,   need  representa>ve   examples  for  all   relevant  varia>ons!     Classical  solu>on:  hope   for  a  smooth  enough   target  func>on,  or   make  it  smooth  by   handcrafing  features  
Easy Learning * * = example (x,y) * y * true unknown function * * * * * * * * * * learned function: prediction = f(x) x
Local Smoothness Prior: Locally Capture the Variations * = training example y * true function: unknown prediction learnt = interpolated f(x) * * test point x x *
Real Data Are on Highly Curved Manifolds 10  
Not Dimensionality so much as Number of Variations (Bengio, Delalleau & Le Roux 2007) •  Theorem:  Gaussian  kernel  machines  need  at  least  k  examples   to  learn  a  func>on  that  has  2k  zero-­‐crossings  along  some  line             •  Theorem:  For  a  Gaussian  kernel  machine  to  learn  some   maximally  varying  func>ons    over  d  inputs  requires  O(2d)   examples    
Is there any hope to generalize non-locally? Yes! Need more priors! 12  
Part  1   Six Good Reasons to Explore Representation Learning 13  
#1 Learning features, not just handcrafting them Most  ML  systems  use  very  carefully  hand-­‐designed   features  and  representa>ons   Many  prac>>oners  are  very  experienced  –  and  good  –  at  such   feature  design  (or  kernel  design)   In  this  world,  “machine  learning”  reduces  mostly  to  linear   models  (including  CRFs)  and  nearest-­‐neighbor-­‐like  features/ models  (including  n-­‐grams,  kernel  SVMs,  etc.)     Hand-­‐cra7ing  features  is  )me-­‐consuming,  bri<le,  incomplete   14  
How can we automatically learn good features? Claim:  to  approach  AI,  need  to  move  scope  of  ML  beyond   hand-­‐crafed  features  and  simple  models   Humans  develop  representa>ons  and  abstrac>ons  to   enable  problem-­‐solving  and  reasoning;  our  computers   should  do  the    same   Handcrafed  features  can  be  combined  with  learned   features,  or  new  more  abstract  features  learned  on  top   of  handcrafed  features   15  
#2 The need for distributed representations •  Clustering,  Nearest-­‐ Clustering   Neighbors,  RBF  SVMs,  local   non-­‐parametric  density   es>ma>on  &  predic>on,   decision  trees,  etc.   •  Parameters  for  each   dis>nguishable  region   •  #  dis>nguishable  regions   linear  in  #  parameters   16  
#2 The need for distributed representations Mul>-­‐   Clustering   •  Factor  models,  PCA,  RBMs,   Neural  Nets,  Sparse  Coding,   Deep  Learning,  etc.   •  Each  parameter  influences   many  regions,  not  just  local   neighbors   •  #  dis>nguishable  regions   grows  almost  exponen>ally   C1   C2   C3   with  #  parameters   •  GENERALIZE  NON-­‐LOCALLY   TO  NEVER-­‐SEEN  REGIONS   input   17  
#2 The need for distributed representations Mul>-­‐   Clustering   Clustering   Learning  a  set  of  features  that  are  not  mutually  exclusive   can  be  exponen>ally  more  sta>s>cally  efficient  than   nearest-­‐neighbor-­‐like  or  clustering-­‐like  models   18  
#3 Unsupervised feature learning Today,  most  prac>cal  ML  applica>ons  require  (lots  of)   labeled  training  data   But  almost  all  data  is  unlabeled   The  brain  needs  to  learn  about  1014  synap>c  strengths   …  in  about  109  seconds   Labels  cannot  possibly  provide  enough  informa>on   Most  informa>on  acquired  in  an  unsupervised  fashion   19  
#3 How do humans generalize from very few examples? •  They  transfer  knowledge  from  previous  learning:   •  Representa>ons   •  Explanatory  factors   •  Previous  learning  from:  unlabeled  data                    +  labels  for  other  tasks   •  Prior:  shared  underlying  explanatory  factors,  in   par)cular  between  P(x)  and  P(Y|x)     20    
#3 Sharing Statistical Strength by Semi-Supervised Learning •  Hypothesis:  P(x)  shares  structure  with  P(y|x)   purely   semi-­‐   supervised   supervised   21  
#4 Learning multiple levels of representation There  is  theore>cal  and  empirical  evidence  in  favor  of   mul>ple  levels  of  representa>on    Exponen)al  gain  for  some  families  of  func)ons   Biologically  inspired  learning   Brain  has  a  deep  architecture   Cortex  seems  to  have  a     generic  learning  algorithm     Humans  first  learn  simpler     concepts  and  then  compose     them  to  more  complex  ones   22    
#4 Sharing Components in a Deep Architecture Polynomial  expressed  with  shared  components:  advantage  of   depth  may  grow  exponen>ally       Sum-­‐product   network  
#4 Learning multiple levels of representation (Lee,  Largman,  Pham  &  Ng,  NIPS  2009)   (Lee,  Grosse,  Ranganath  &  Ng,  ICML  2009)     Successive  model  layers  learn  deeper  intermediate  representa>ons     High-­‐level   Layer  3   linguis>c  representa>ons   Parts  combine   to  form  objects   Layer  2   Layer  1   24   Prior:  underlying  factors  &  concepts  compactly  expressed  w/  mul)ple  levels  of  abstrac)on    
#4 Handling the compositionality of human language and thought zt-­‐1   zt   zt+1   •  Human  languages,  ideas,  and   ar>facts  are  composed  from   simpler  components   xt-­‐1   xt   xt+1   •  Recursion:  the  same   operator  (same  parameters)   is  applied  repeatedly  on   different  states/components   of  the  computa>on   •  Result  afer  unfolding  =  deep   (Bobou  2011,  Socher  et  al  2011)   representa>ons   25  
#5 Multi-Task Learning task 1 task 2 task 3 •  Generalizing  beber  to  new   output y1 output y2 output y3 tasks  is  crucial  to  approach  AI   Task  A   Task  B   Task  C   •  Deep  architectures  learn  good   intermediate  representa>ons   that  can  be  shared  across  tasks   •  Good  representa>ons  that   disentangle  underlying  factors   of  varia>on  make  sense  for   raw input x many  tasks  because  each  task   concerns  a  subset  of  the  factors   26  
#5 Sharing Statistical Strength task 1 task 2 task 3 •  Mul>ple  levels  of  latent   output y1 output y2 output y3 variables  also  allow   Task  A   Task  B   Task  C   combinatorial  sharing  of   sta>s>cal  strength:   intermediate  levels  can  also   be  seen  as  sub-­‐tasks   •  E.g.  dic>onary,  with   intermediate  concepts  re-­‐ used  across  many  defini>ons   raw input x Prior:  some  shared  underlying  explanatory  factors  between  tasks       27  
#5 Combining Multiple Sources of Evidence with Shared Representations person   url   event   •  Tradi>onal  ML:  data  =  matrix   url   words   history   •  Rela>onal  learning:  mul>ple  sources,   different  tuples  of  variables   •  Share  representa>ons  of  same  types   across  data  sources   •  Shared  learned  representa>ons  help   event   url   person   propagate  informa>on  among  data   history   words   url   sources:  e.g.,  WordNet,  XWN,   Wikipedia,  FreeBase,  ImageNet… (Bordes  et  al  AISTATS  2012)   P(person,url,event)   P(url,words,history)   28  
#5 Different object types represented in same space Google:   S.  Bengio,  J.   Weston  &  N.   Usunier   (IJCAI  2011,   NIPS’2010,   JMLR  2010,   MLJ  2010)  
#6 Invariance and Disentangling •  Invariant  features   •  Which  invariances?   •  Alterna>ve:  learning  to  disentangle  factors   •  Good  disentangling  à      avoid  the  curse  of  dimensionality   30  
#6 Emergence of Disentangling •  (Goodfellow  et  al.  2009):  sparse  auto-­‐encoders  trained   on  images     •  some  higher-­‐level  features  more  invariant  to   geometric  factors  of  varia>on     •  (Glorot  et  al.  2011):  sparse  rec>fied  denoising  auto-­‐ encoders  trained  on  bags  of  words  for  sen>ment   analysis   •  different  features  specialize  on  different  aspects   (domain,  sen>ment)   31   WHY?  
#6 Sparse Representations •  Just  add  a  penalty  on  learned  representa>on   •  Informa>on  disentangling  (compare  to  dense  compression)   •  More  likely  to  be  linearly  separable  (high-­‐dimensional  space)   •  Locally  low-­‐dimensional  representa>on  =  local  chart   •  Hi-­‐dim.  sparse  =  efficient  variable  size  representa>on                  =  data  structure   Few  bits  of  informa>on                                                        Many  bits  of  informa>on   Prior:  only  few  concepts  and  a<ributes  relevant  per  example     32  
Bypassing the curse We  need  to  build  composi>onality  into  our  ML  models     Just  as  human  languages  exploit  composi>onality  to  give   representa>ons  and  meanings  to  complex  ideas   Exploi>ng  composi>onality  gives  an  exponen>al  gain  in   representa>onal  power   Distributed  representa>ons  /  embeddings:  feature  learning   Deep  architecture:  mul>ple  levels  of  feature  learning   Prior:  composi>onality  is  useful  to  describe  the   world  around  us  efficiently   33    
Bypassing the curse by sharing statistical strength •  Besides  very  fast  GPU-­‐enabled  predictors,  the  main  advantage   of  representa>on  learning  is  sta>s>cal:  poten>al  to  learn  from   less  labeled  examples  because  of  sharing  of  sta>s>cal  strength:   •  Unsupervised  pre-­‐training  and  semi-­‐supervised  training   •  Mul>-­‐task  learning   •  Mul>-­‐data  sharing,  learning  about  symbolic  objects  and  their   rela>ons   34  
Why now? Despite  prior  inves>ga>on  and  understanding  of  many  of  the   algorithmic  techniques  …   Before  2006  training  deep  architectures  was  unsuccessful   (except  for  convolu>onal  neural  nets  when  used  by  people  who  speak  French)   What  has  changed?   •  New  methods  for  unsupervised  pre-­‐training  have  been   developed  (variants  of  Restricted  Boltzmann  Machines  =   RBMs,  regularized  autoencoders,  sparse  coding,  etc.)   •  Beber  understanding  of  these  methods   •  Successful  real-­‐world  applica>ons,  winning  challenges  and   bea>ng  SOTAs  in  various  areas   35  
Major Breakthrough in 2006 •  Ability  to  train  deep  architectures  by   using  layer-­‐wise  unsupervised   learning,  whereas  previous  purely   supervised  abempts  had  failed   •  Unsupervised  feature  learners:   •  RBMs   •  Auto-­‐encoder  variants   Bengio Montréal •  Sparse  coding  variants   Toronto Hinton Le Cun New York 36  
Unsupervised and Transfer Learning Challenge + Transfer Learning Challenge: Deep Learning 1st Place NIPS’2011   Raw  data   Transfer   Learning   1  layer   2  layers   Challenge     Paper:   ICML’2012   ICML’2011   workshop  on   Unsup.  &   Transfer  Learning   3  layers   4  layers  
More Successful Applications •  Microsof  uses  DL  for  speech  rec.  service  (audio  video  indexing),  based  on   Hinton/Toronto’s  DBNs  (Mohamed  et  al  2011)   •  Google  uses  DL  in  its  Google  Goggles  service,  using  Ng/Stanford  DL  systems   •  NYT  today  talks  about  these:  http://www.nytimes.com/2012/06/26/technology/ in-a-big-network-of-computers-evidence-of-machine-learning.html?_r=1 •  Substan>ally  bea>ng  SOTA  in  language  modeling  (perplexity  from  140  to  102   on  Broadcast  News)  for  speech  recogni>on  (WSJ  WER  from  16.9%  to  14.4%)   (Mikolov  et  al  2011)  and  transla>on  (+1.8  BLEU)  (Schwenk  2012)   •  SENNA:  Unsup.  pre-­‐training  +  mul>-­‐task  DL  reaches  SOTA  on  POS,  NER,  SRL,   chunking,  parsing,  with  >10x  beber  speed  &  memory  (Collobert  et  al  2011)   •  Recursive  nets  surpass  SOTA  in  paraphrasing  (Socher  et  al  2011)   •  Denoising  AEs  substan>ally  beat  SOTA  in  sen>ment  analysis  (Glorot  et  al  2011)   •  Contrac>ve  AEs  SOTA  in  knowledge-­‐free  MNIST  (.8%  err)  (Rifai  et  al  NIPS  2011)   •  Le  Cun/NYU’s  stacked  PSDs  most  accurate  &  fastest  in  pedestrian  detec>on   and  DL  in  top  2  winning  entries  of  German  road  sign  recogni>on  compe>>on     38  
39  
Part  2   Representation Learning Algorithms 40  
A neural network = running several logistic regressions at the same time If  we  feed  a  vector  of  inputs  through  a  bunch  of  logis>c  regression   func>ons,  then  we  get  a  vector  of  outputs   But  we  don’t  have  to  decide   ahead  of  >me  what  variables   these  logis>c  regressions  are   trying  to  predict!   41  
A neural network = running several logistic regressions at the same time …  which  we  can  feed  into  another  logis>c  regression  func>on   and  it  is  the  training   criterion  that  will   decide  what  those   intermediate  binary   target  variables  should   be,  so  as  to  make  a   good  job  of  predic>ng   the  targets  for  the  next   layer,  etc.   42  
A neural network = running several logistic regressions at the same time •  Before  we  know  it,  we  have  a  mul>layer  neural  network….   How to do unsupervised training? 43  
PCA code= latent features h = Linear Manifold = Linear Auto-Encoder … … = Linear Gaussian Factors input reconstruction input  x,  0-­‐mean   Linear  manifold   features=code=h(x)=W  x   reconstruc>on(x)=WT  h(x)  =  WT  W  x   W  =  principal  eigen-­‐basis  of  Cov(X)   reconstruc>on(x)   reconstruc>on  error  vector   x   Probabilis>c  interpreta>ons:   1.  Gaussian  with  full   covariance  WT  W+λI   2.  Latent  marginally  iid   Gaussian  factors  h  with       x  =  WT  h  +  noise   44  
Directed Factor Models •  P(h)  factorizes  into  P(h1)  P(h2)…   h1 h2 h3 h4 h5 •  Different  priors:   •  PCA:  P(hi)  is  Gaussian   W3   W1   •  ICA:  P(hi)  is  non-­‐parametric   W5   •  Sparse  coding:  P(hi)  is  concentrated  near  0   •  Likelihood  is  typically  Gaussian  x  |  h     x1 x2          with  mean  given  by  WT  h   •  Inference  procedures  (predic>ng  h,  given  x)  differ   •  Sparse  h:  x  is  explained  by  the  weighted  addi>on  of  selected   filters  hi   x   W1   W3   W5   h1   h3   h5                               =  .9  x                        +  .8  x                      +  .7  x   45  
Stacking Single-Layer Learners •  PCA  is  great  but  can’t  be  stacked  into  deeper  more  abstract   representa>ons  (linear  x  linear  =  linear)   •  One  of  the  big  ideas  from  Hinton  et  al.  2006:  layer-­‐wise   unsupervised  feature  learning   Stacking Restricted Boltzmann Machines (RBM) à Deep Belief Network (DBN) 46  
Effective deep learning became possible through unsupervised pre-training [Erhan  et  al.,  JMLR  2010]   (with  RBMs  and  Denoising  Auto-­‐Encoders)   Purely  supervised  neural  net   With  unsupervised  pre-­‐training   47  
Layer-wise Unsupervised Learning input … 48  
Layer-Wise Unsupervised Pre-training features … input … 49  
Layer-Wise Unsupervised Pre-training ? reconstruction … input = … of input features … input … 50  
Layer-Wise Unsupervised Pre-training features … input … 51  
Layer-Wise Unsupervised Pre-training More abstract … features features … input … 52  
Layer-Wise Unsupervised Pre-training Layer-wise Unsupervised Learning ? reconstruction … … = of features More abstract … features features … input … 53  
Layer-Wise Unsupervised Pre-training More abstract … features features … input … 54  
Layer-wise Unsupervised Learning Even more abstract features … More abstract … features features … input … 55  
Supervised Fine-Tuning Output Target f(X) six ? = Y two! Even more abstract features … More abstract … features features … input … •  Addi>onal  hypothesis:  features  good  for  P(x)  good  for  P(y|x)   56  
Restricted Boltzmann Machines 57  
Undirected Models: the Restricted Boltzmann Machine [Hinton  et  al  2006]   •  Probabilis>c  model  of  the  joint  distribu>on  of   h1 h2 h3 the  observed  variables  (inputs  alone  or  inputs   and  targets)  x   •  Latent  (hidden)  variables  h  model  high-­‐order   dependencies   •  Inference  is  easy,  P(h|x)  factorizes   x1 x2 •  See  Bengio  (2009)  detailed  monograph/review:        “Learning  Deep  Architectures  for  AI”.   •  See  Hinton  (2010)            “A  prac,cal  guide  to  training  Restricted  Boltzmann  Machines”  
Boltzmann Machines & MRFs •  Boltzmann  machines:        (Hinton  84)     •  Markov  Random  Fields:     Sof  constraint  /  probabilis>c  statement           ¡  More    nteres>ng  with  latent  variables!   i                                                                                                                                                                                                                                                                                                                                                                            
Restricted Boltzmann Machine (RBM) •  A  popular  building   block  for  deep   architectures   hidden   •  Bipar)te  undirected   graphical  model   observed
Gibbs Sampling in RBMs h1 ~ P(h|x1) h2 ~ P(h|x2) h3 ~ P(h|x3) x1 x2 ~ P(x|h1) x3 ~ P(x|h2) ¡  Easy inference P(h|x)  and  P(x|h)  factorize   ¡  Efficient block Gibbs P(h|x)=  Π  P(hi|x)   sampling xàhàxàh… i  
Problems with Gibbs Sampling In  prac>ce,  Gibbs  sampling  does  not  always  mix  well…   RBM trained by CD on MNIST Chains from random state Chains from real digits (Desjardins  et  al  2010)  
RBM with (image, label) visible units hidden h U W image y 0 0 1 0 x label y (Larochelle  &  Bengio  2008)  
RBMs are Universal Approximators (Le Roux & Bengio 2008) •  Adding  one  hidden  unit  (with  proper  choice  of  parameters)   guarantees  increasing  likelihood     •  With  enough  hidden  units,  can  perfectly  model  any  discrete   distribu>on   •  RBMs  with  variable  #  of  hidden  units  =  non-­‐parametric  
RBM Conditionals Factorize
RBM Energy Gives Binomial Neurons
RBM Free Energy •  Free  Energy  =  equivalent  energy  when  marginalizing       •  Can  be  computed  exactly  and  efficiently  in  RBMs     •  Marginal  likelihood  P(x)  tractable  up  to  par>>on  func>on  Z  
Factorization of the Free Energy Let  the  energy  have  the  following  general  form:   Then  
Energy-Based Models Gradient
Boltzmann Machine Gradient •  Gradient  has  two  components:   positive phase negative phase ¡  In  RBMs,  easy  to  sample  or  sum  over  h|x   ¡  Difficult  part:  sampling  from  P(x),  typically  with  a  Markov  chain  
Positive & Negative Samples •  Observed (+) examples push the energy down •  Generated / dream / fantasy (-) samples / particles push the energy up X+ X- Equilibrium:  E[gradient]  =  0  
Training RBMs Contras>ve  Divergence:    start  nega>ve  Gibbs  chain  at  observed  x,  run  k   (CD-­‐k)   Gibbs  steps     SML/Persistent  CD:   run  nega>ve  Gibbs  chain  in  background  while   (PCD)    weights  slowly  change   Fast  PCD:   two  sets  of  weights,  one  with  a  large  learning  rate   only  used  for  nega>ve  phase,  quickly  exploring   modes   Herding:   Determinis>c  near-­‐chaos  dynamical  system  defines   both  learning  and  sampling   Tempered  MCMC:   use  higher  temperature  to  escape  modes  
Contrastive Divergence Contrastive Divergence (CD-k): start negative phase block Gibbs chain at observed x, run k Gibbs steps (Hinton 2002) h+ ~ P(h|x+) h-~ P(h|x-) Observed x+ k = 2 steps Sampled x- positive phase negative phase push down Free Energy x+ x- push up
Persistent CD (PCD) / Stochastic Max. Likelihood (SML) Run  nega>ve  Gibbs  chain  in  background  while  weights  slowly   change  (Younes  1999,  Tieleman  2008):   •    Guarantees  (Younes  1999;  Yuille  2005)     •  If  learning  rate  decreases  in  1/t,          chain  mixes  before  parameters  change  too  much,          chain  stays  converged  when  parameters  change   h+ ~ P(h|x+) previous x- Observed x+ new x- (positive phase)
PCD/SML + large learning rate Nega>ve  phase  samples  quickly  push  up  the  energy  of   wherever  they  are  and  quickly  move  to  another  mode   push FreeEnergy down x+ x- push up
Some RBM Variants •  Different  energy  func>ons  and  allowed                   values  for  the  hidden  and  visible  units:   •  Hinton  et  al  2006:  binary-­‐binary  RBMs   •  Welling  NIPS’2004:  exponen>al  family  units   •  Ranzato  &  Hinton  CVPR’2010:  Gaussian  RBM  weaknesses  (no   condi>onal  covariance),  propose  mcRBM   •  Ranzato  et  al  NIPS’2010:  mPoT,  similar  energy  func>on   •  Courville  et  al  ICML’2011:  spike-­‐and-­‐slab  RBM     76  
Convolutionally Trained Spike & Slab RBMs Samples
Training  examples   Generated  samples   ssRBM is not Cheating
Auto-Encoders & Variants 79  
Auto-Encoders  code=  latent  features   •  MLP  whose  target  output  =  input   •  Reconstruc>on=decoder(encoder(input)),            e  ncoder                      decoder   e.g.    input   …   …      reconstruc>on   •  Probable  inputs  have  small  reconstruc>on  error   because  training  criterion  digs  holes  at  examples   •  With  bobleneck,  code  =  new  coordinate  system   •  Encoder  and  decoder  can  have  1  or  more  layers   •  Training  deep  auto-­‐encoders  notoriously  difficult   80    
Stacking Auto-Encoders Auto-­‐encoders  can  be  stacked  successfully  (Bengio  et  al  NIPS’2006)  to  form   highly  non-­‐linear  representa>ons,  which  with  fine-­‐tuning  overperformed   purely  supervised  MLPs     81  
Auto-Encoder Variants •  Discrete  inputs:  cross-­‐entropy  or  log-­‐likelihood  reconstruc>on   criterion  (similar  to  used  for  discrete  targets  for  MLPs)   •  Regularized  to  avoid  learning  the  iden>ty  everywhere:   •  Undercomplete  (eg  PCA):    bobleneck  code  smaller  than  input   •  Sparsity:  encourage  hidden  units  to  be  at  or  near  0          [Goodfellow  et  al  2009]   •  Denoising:  predict  true  input  from  corrupted  input          [Vincent  et  al  2008]   •  Contrac>ve:  force  encoder  to  have  small  deriva>ves          [Rifai  et  al  2011]   82  
Manifold Learning •  Addi>onal  prior:  examples  concentrate  near  a  lower   dimensional  “manifold”  (region  of  high  density  with  only  few   opera>ons  allowed  which  allow  small  changes  while  staying  on   the  manifold)   83  
Denoising Auto-Encoder (Vincent  et  al  2008)   •  Corrupt  the  input   •  Reconstruct  the  uncorrupted  input   Hidden code (representation) KL(reconstruction | raw input) Corrupted input Raw input reconstruction •  Encoder  &  decoder:  any  parametriza>on   •  As  good  or  beber  than  RBMs  for  unsupervised  pre-­‐training  
Denoising Auto-Encoder •  Learns  a  vector  field  towards  higher   probability  regions   •  Some  DAEs  correspond  to  a  kind  of   Gaussian  RBM  with  regularized  Score   Corrupted input Matching  (Vincent  2011)   •  But  with  no  par>>on  func>on,  can  measure   training  criterion   Corrupted input
Stacked Denoising Auto-Encoders Infinite MNIST
Auto-Encoders Learn Salient Variations, like a non-linear PCA •  Minimizing  reconstruc>on  error  forces  to   keep  varia>ons  along  manifold.   •  Regularizer  wants  to  throw  away  all   varia>ons.   •  With  both:  keep  ONLY  sensi>vity  to   varia>ons  ON  the  manifold.   87  
Contractive Auto-Encoders (Rifai,  Vincent,  Muller,  Glorot,  Bengio  ICML  2011;  Rifai,  Mesnil,   Vincent,  Bengio,  Dauphin,  Glorot  ECML  2011;  Rifai,  Dauphin,   Vincent,  Bengio,  Muller  NIPS  2011)   Most  hidden  units  saturate:   few  ac>ve  units  represent  the   ac>ve  subspace  (local  chart)   Training  ccontrac>on  in  all   wants   riterion:   cannot  afford  contrac>on  in   direc>ons   manifold  direc>ons    
Jacobian’s  spectrum  is  peaked  =   local  low-­‐dimensional   representa>on  /  relevant  factors   89  
Contractive Auto-Encoders
Input  Point   Tangents   MNIST   91  
Input  Point   Tangents   MNIST  Tangents   92  
Distributed vs Local (CIFAR-10 unsupervised) Input  Point   Tangents   Local  PCA   Contrac>ve  Auto-­‐Encoder   93  
Learned Tangent Prop: the Manifold Tangent Classifier 3  hypotheses:   1.  Semi-­‐supervised  hypothesis  (P(x)  related  to  P(y|x))     2.  Unsupervised  manifold  hypothesis  (data  concentrates  near   low-­‐dim.  manifolds)   3.  Manifold  hypothesis  for  classifica>on  (low  density  between   class  manifolds)   Algorithm:   1.  Es>mate  local  principal  direc>ons  of  varia>on  U(x)  by  CAE   (principal  singular  vectors  of  dh(x)/dx)   2.  Penalize  f(x)=P(y|x)  predictor  by  ||  df/dx  U(x)  ||  
Manifold Tangent Classifier Results •  Leading  singular  vectors  on  MNIST,  CIFAR-­‐10,  RCV1:   •  Knowledge-­‐free  MNIST:  0.81%  error     •  Semi-­‐sup.       •  Forest  (500k  examples)    
Inference and Explaining Away •  Easy  inference  in  RBMs  and  regularized  Auto-­‐Encoders   •  But  no  explaining  away  (compe>>on  between  causes)   •  (Coates  et  al  2011):  even  when  training  filters  as  RBMs  it  helps   to  perform  addi>onal  explaining  away  (e.g.  plug  them  into  a   Sparse  Coding  inference),  to  obtain  beber-­‐classifying  features   •  RBMs  would  need  lateral  connec>ons  to  achieve  similar  effect   •  Auto-­‐Encoders  would  need  to  have  lateral  recurrent   connec>ons   96  
Sparse Coding (Olshausen  et  al  97)   •  Directed  graphical  model:     •  One  of  the  first  unsupervised  feature  learning  algorithms  with   non-­‐linear  feature  extrac>on  (but  linear  decoder)         MAP  inference  recovers  sparse  h  although  P(h|x)  not  concentrated  at  0     •  Linear  decoder,  non-­‐parametric  encoder   •  Sparse  Coding  inference,  convex  opt.  but  expensive   97  
Predictive Sparse Decomposition •  Approximate  the  inference  of  sparse  coding  by   an  encoder:   Predic>ve  Sparse  Decomposi>on  (Kavukcuoglu  et  al  2008)   •  Very  successful  applica>ons  in  machine  vision   with  convolu>onal  architectures   98  
Predictive Sparse Decomposition •  Stacked  to  form  deep  architectures   •  Alterna>ng  convolu>on,  rec>fica>on,  pooling   •  Tiling:  no  sharing  across  overlapping  filters   •  Group  sparsity  penalty  yields  topographic   maps   99  
Deep Variants 100